Hi,
About 2 hours ago this evening, we lost communication to all 20 VM's on a 8 Node Cluster and they took 25 minutes to recover! Fibre Channel shared storage is being used throughout the cluster through 2 redundant routes from each node. The host OS was still working fine. I appreciate this points to the storage going down but I really don't think it had, there are no errors on either FC switch and no error on either storage device. We are using Server 2012 on HP DL360's.
The Get-ClusterLog returns this error at the time they all went down:
00000370.000010d8::2013/04/14-20:12:25.115 ERR [RHS] RhsCall::DeadlockMonitor: Call TERMINATERESOURCE timed out by 6 milliseconds for resource 'SCVMM VM1'.00000370.000010d8::2013/04/14-20:12:25.115 INFO [RHS] Enabling RHS termination watchdog with timeout 1200000 and recovery action 3.
00000370.000010d8::2013/04/14-20:12:25.115 ERR [RHS] Resource SCVMM VM1 handling deadlock. Cleaning current operation and terminating RHS process.
000006dc.000011b4::2013/04/14-20:12:25.115 WARN [RCM] HandleMonitorReply: FAILURENOTIFICATION for 'SCVMM VM1', gen(3) result 4/0.
000006dc.000011b4::2013/04/14-20:12:25.115 INFO [RCM] rcm::RcmResource::HandleMonitorReply: Resource 'SCVMM VM1' consecutive failure count 1.
00000370.000010d8::2013/04/14-20:12:25.115 ERR [RHS] About to send WER report.
Any help would be appreciated.