I have a two node Hyper-V Failover Cluster running Windows Server 2008 R2 SP1 Enterprise. I get slow network performance between Virtual machines and Physical machines.
Environment Information:
2x Dell PowerEdge R720, Dual Intel Xeon E5-2620 6 Core Processors, 48GB RAM Each, 2x 300x10K xRAID1 Operating System Volume, 6x Broadcom BCM5720 GB NICs (1xQP, 1xDP)
- 1st NIC - Dedicated to Host - 10.1.30.x (Everything Bound) (Connected to LAN Switch)
- 2nd NIC - Dedicated to iSCSI - 10.2.30.x (Only TCP v4 bound, No GW or DNS, Jumbo Frames Enabled)
- 3rd NIC - Dedicated Virtual Switch - No IP (Only MS Virtual Switch Bound, Connected to LAN Switch)
- 4th NIC - Dedicated for Live Migration - 10.4.30.x (No GW or DNS, Everything but MS Virtual Switch Bound, Cross Over Cable between Nodes)
- 5th NIC - Dedicated for CSV - 10.3.30.x (No GW or DNS, Everything but MS Virtual Switch Bound, Cross Over Cable between Nodes)
- 6th NIC - Dedicated for iSCSi - 10.2.30.x (Only TCP v4 bound, No GW or DNS, Jumbo Frames Enabled)
1 EqualLogic 7.2K - 12TB RAW SAN - 4x GB NICs spread across 2 Controllers. (MPIO Enabled)
2 x Cisco SMB SG300 20 Port GB Layer 3 Switches = iSCSI Switches (2 Ports configured between each for LACP)
3 x Cisco SMB SG300 52 Port GB Layer 3 Switches = LAN Switches (2 Ports configured between each for LACP)
No Virtual LANs or Uplink from iSCSI to LAN switches and vice versa.
I have performed dozens of tests to help isolated the issue.
When I copy/move files through the network from a VM to a Physical Machine or vice versa it is running very slow. Averaging 200KB/second transfers. Even when browsing folders on network shares you can see lag when the folder lists all the contents.
A 67MB Folder with 120 files takes about 3-5 minutes to copy.
When I copy/move files through the network from a Physical Machine to another Physical Machine it runs normal. Averaging 100MB/second transfers. Browsing of folders list all the contents very rapidly. The physical machines are either the
HyperV hosts or other servers/NAS appliances I have on the LAN. A 67MB folder took 2-3 seconds to copy. a 4GB file took 45 seconds to copy.
When I copy/move files through the network from a VM to a VM on the same Host it runs normal. Averaging 75-100MB/Second transfers. Browsing of folders list all contents very rapidly. A 67MB folder took 5 seconds to copy.
When I copy/move files through the network from a VM to a VM that are on different HyperV Hosts it runs slow. Averaging 25KB/Second transfers. Browsing of folders list all contents very slow. a 67MB Folder with 120 files takes about 35-45 Minutes
to copy.
When I copy/move files locally to/from the Local Drive of the VM it runs normal. Averaging very fast transfers as expected. I copied 1GB worth of files in less than 15 seconds.
I have played around with every combination of disabling/enabling NIC Properties Advance Settings. TCP Offload, Large Send Offload, RSS, Flow Control, CheckSum Offloads, Virtual Machine Queues. I have enabled/disabled these consistently at each
network devices/switch that were involved in these tests. I also enabled/disabled these using both the registry commands and/or the NETSH commands. I performed numerous tests with various combinations. All items are back to default settings.
Only a few combinations would improve the network performance but only very slightly. Increasing approx 5-15% but never going higher than 455KB /second or averaging more than 200KB/second.
I have applied the Sandy Bridge Processor/HyperV Microsoft HotFix - http://support.microsoft.com/kb/2517329 on both HyperV servers, (rebooted) and it increased through put by about 5%. VM's are still having slow performance.
There are No AV programs running on any of these machines (physical or virtual), yet. No software firewalls or any other malware/threat/IPS scans running along the network path of these machines. On the VM's when I did Windows Updates, they downloaded
in a respectable time frame considering how many updates were needed when I first built the machines. When I do speed tests using speedtest.net I get around .6MBps download and .8MBPs upload and we have a 10MBPs download/Upload Direct Internet Connection.
(Fiber over Ethernet Copper ). I made sure all machines have valid working certified CAT5e cables in place.
Any other ideas?