Cannot live migrate 2 of 12 VM's in FCM, when trying to live migrate these 2 VM's they get to around 48% and then the live migration stops. We've tried many things to resolve this, I'm just wondering if we've missed something.
Troubleshooting steps taken:
1. Settings > Management > Name, check for Notes in the properties
Notes existed, but after removing, did not make a difference. Cannot live migrate.
2. Files or lock folder in C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines
No folder exists with .lock files in it
3. VM migration timeout using HKEY_LOCAL_MACHINE\Software\Microsoft\Microsoft System Center Virtual Machine Manager Server\Settings\LiveMigrationTimeoutSecs
Reg path does not exist, stops at HKLM\SOFTWARE\Microsoft\Microsoft System Center Virtual Machine Manager Agent
4. In the screenshot, it says "the operation was canceled". Is there something else that pops up before it is canceled, or is it being canceled by another user?
No. "The operation fails at source." is another error, however no user is actually cancelling the live migration.
5. Do the cluster nodes have identical hardware?
Yes
6. Is the error happening when trying to migrate to a different node?
Yes
7. Have you tried running the Cluster Validation Wizard within the FCM to test the cluster?
"Ran Cluster Validation test (excluding Storage). There was only 1 warning about an erroneous network resource that has never been connected. Error is below; however, I removed that resource form the cluster
and then tried another Live Migration with no luck."
8. Does the underlying issue appear to be not with the cluster itself, but the VM?
Yes
9. Have you guys been able to live migrate VM's to all 4 of the clusters?
Yes
10. Uninstalling Veeam
No change, cannot live migrate.
11. Attempting to live migrate VM with PowerShell
Did not work
12. gwmi win32_bios
BIOS revision is same on all 4 nodes
13. Jumbo frame sizes?
Jumbo packets are disabled on all NICs.
14. VHD being written to file or a folder? If folder - are folder permissions correct?
Folder, Read/Write/Modify for System account. Permissions are OK.
16. Migrate node to node other VM's.
Other VM’s will live migrate OK
17. Migrating attached storage by itself
Storage only will live migrate OK
18. Moved MRT & RCD files and attempted to migrate again.
"Shut down VM, moved the RCT & MRT files, re-started VM - live migration did not work.
Shut down VM1 & VM2, started a quick migration - quick migration took about 10 minutes and VMs went into 'saved' state for quite a while before finally moving to other nodes.
Ran Cluster Validation test (excluding Storage). There was only 1 warning about an erroneous network resource that has never been connected. Error is below; however, I removed that resource form the cluster and
then tried another Live Migration with no luck.
Re-ran a quick migration while VMs were still off. This moved the VMs fairly quickly.
Started VMs and attempted live migration of VM2; live migration did not work
Attempted live migration of VM1 while Wireshark ran and generated a capture - live migration did not work. I have capture file we can investigate. Please let me know how you want to review this capture."
19. Processor compatibility mode not enabled
Enabled, no change. 2 VM's still do not live migrate
20. Check patch level
Windows and BIOS patch level is identical between nodes
21. Check for XML file cache at C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines Cache
Copying (if it did not already exist) or removing (if it did already exist) made no change either way. On one VM, it was not there; on another VM, it was there already.
22. Edit Registry for NetworkBufferCount at HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization\Migration
Adjusting this made no change
23. Disabling Hyperthreading
Client cannot disable Hyperthreading
24. Do VM's use shared VHD Sets?
No, they do not.
25. vCPU and vRAM differences?
No difference among the VM's. 8 vCPU, 32768 MB vRAM for each.
26. Processor > NUMA, does "Maximum amount of memory (MB)" match the same setting under Memory > RAM or Memory > Maximum RAM?
Memory settings are identical