Hi all
I am troubleshooting slow LiveMigrations between Windows Server 2012 Datacenter (Standalone) Hyper-V hosts. All of them are HP ProLiant 3xx servers from 2009/2011 with DAS. When installing W2K12, we teamed all three GbE-NICs via Windows (New-NetbfoTeam, LoadBalancingAlgo
HyperVPort and TeamingMode LACP). The servers are attached to a HP ProCurve switch, where the server ports are configured as members i an LACP-trunk.
Connectivity worked fine, but LiveMigration felt very slow. ("Slow" is currently a fuzzy term - basically based upon speed difference between copying a 40 GB VHD via SMB between the servers, and letting LiveMigration move a 40 GB server. With SMB, we achive
about 100 MB/s - also here less than expected since we thought that multi-channeling would kick in...)
To narrow down possible error sources (basically Windows Teaming), I did the following on all three servers:
1. Removed one GbE NIC (NIC3) from the Team and moved the cable to a non-LACP port on the switch
2. Created a new vSwitch on this network adapter:
New-VMSwitch -name "vAllManagementTraffic" -SwitchType "External" -BandwidhReservationMode "Weight" -NetAdapterinterfaceDescription "NIC3-Management" - AllowManagement $False
3. Created vNICs visible to the management OS on top, with correct VLAN for LiveMigration and some QoS weight rules (LiveMigration should get 80% of the bw), and the correct IPs (not shown):
Add-VMNetworkAdapter -ManagementOS -Name "vManagement" -SwitchName "vAllManagementTraffic" Add-VMNetworkAdapter -ManagementOS -Name "vLiveMigration" -SwitchName "vAllManagementTraffic" Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "vLiveMigration" -Access -VlanId 11 Set-VMNetworkAdapter -ManagementOS -Name "vLiveMigration" -MinimumBandwidthWeight 80 Set-VMNetworkAdapter -ManagementOS -Name "vManagement" -MinimumBandwidthWeight 20
Update: For the LiveMigration adapter, I added the *NdisDeviceType (value: 1) registry setting in order to turn off Windows Firewall.
4. Under Hyper-V settings I then restricted the IP-addreses that could be used for LiveMigration to the IP-adress given to the vLiveMigration vNIC.
The result: LiveMigrations are as slow as before.
Have I overlooked something important?
Do I have wrong LiveMigration speed expectations?
How can I monitor/measure current LiveMigration thruput for getting some exact numbers?
Can somebody point me in the correct direction on how to troubleshoot further?
Update: I forgot to add some links to posts that deal with similar problems - threat one and thread two were the most useful. The hints basically consist of the following:
- Disable TCP Offload / RSS in NIC and OS (http://support.microsoft.com/kb/951037) - Not tested yet
- Dedicated NIC - OK
- Check binding order - OK
- QoS Management/iSCSI/LiveMigration - OK
- Power Settings in BIOS and Power Plans in Windows - OK
- Check in Resource Monitor if traffic is going thru the correct NIC - Not verified yet
Thanks for reading!
/Maurice