Hello everyone,
I'm new to analyzing crash dumps and so far have come up with that it is likely I had some sort of memory problem or VHD issue with vhdmp.sys. When it crashed there were many servers re synchronizing with their replicas which is hosted on
a SMB 3.0 share on a server 2012 standard edition file server. I know there likely was a lot of disk I/O and there are 12 VMs on it, 3 of them are rather big ones server about 2,000 people for video streaming and delivery.
System Info:
Mobo: SuperMicro X9DRI-LN4F+ Intel Xeon DualProc
Proc: 2x Intel Xeon-SandyBridge
E5-2690-OctoCore
Memory: 8x Samsung
1600 16GB DDR3 2Rx4 [16GB]
Raid Controller: Adaptec 5405 Z (1b4611d360f)
Battery : Adaptec ZMM-100CC
OS SSD: Intel x25-E Extreme (SSDSA2SH064G1)
Raid 50 Array for Hyper-V:
10X Seagate ConstellationES.2
ST33000650NS [3000GB]
10 gbps fiber nic:
SuperMicro AOC-STGN-i2S SuperMicro 10 Gigabit Port
Power Supply: 2x SuperMicro PWS-920P-1R 920W (one is a redundant power supply )
Here is my !analyze -v output from the debugger:
---------------------------------------------------------------------------
Microsoft (R) Windows Debugger Version 6.2.9200.20512 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [C:\Users\MatthewMobile\Desktop\Kernel_0_0_cab_108c3987\120412-14414-01.dmp]
Mini Kernel Dump File: Only registers and stack trace are available
Symbol search path is: SRV*C:\Symbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
Windows 8 Kernel Version 9200 MP (32 procs) Free x64
Product: Server, suite: TerminalServer DataCenter SingleUserTS
Built by: 9200.16424.amd64fre.win8_gdr.120926-1855
Machine Name:
Kernel base = 0xfffff801`a528b000 PsLoadedModuleList = 0xfffff801`a5555a60
Debug session time: Tue Dec 4 18:42:51.756 2012 (UTC - 8:00)
System Uptime: 19 days 8:49:44.110
Loading Kernel Symbols
...............................................................
................................................................
..........................
Loading User Symbols
Loading unloaded module list
.................
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************
Use !analyze -v to get detailed debugging information.
BugCheck C2, {7, 11c1, 4200016, fffffa80a33cf750}
GetPointerFromAddress: unable to read from fffff801a55e1168
GetUlongFromAddress: unable to read from fffff801a55e11f8
Probably caused by : vhdmp.sys ( vhdmp!VhdmpiFreeVirtualDiskSurface+6f )
Followup: MachineOwner
---------
12: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************
BAD_POOL_CALLER (c2)
The current thread is making a bad pool request. Typically this is at a bad IRQL level or double freeing the same allocation, etc.
Arguments:
Arg1: 0000000000000007, Attempt to free pool which was already freed
Arg2: 00000000000011c1, (reserved)
Arg3: 0000000004200016, Memory contents of the pool block
Arg4: fffffa80a33cf750, Address of the block of pool being deallocated
Debugging Details:
------------------
POOL_ADDRESS: fffffa80a33cf750 Nonpaged pool
FREED_POOL_TAG: VHDu
BUGCHECK_STR: 0xc2_7_VHDu
CUSTOMER_CRASH_COUNT: 1
DEFAULT_BUCKET_ID: WIN8_DRIVER_FAULT_SERVER
PROCESS_NAME: System
CURRENT_IRQL: 0
LAST_CONTROL_TRANSFER: from fffff801a54fba5a to fffff801a5305d40
STACK_TEXT:
fffff880`061469c8 fffff801`a54fba5a : 00000000`000000c2 00000000`00000007 00000000`000011c1 00000000`04200016 : nt!KeBugCheckEx
fffff880`061469d0 fffff880`09c4a563 : fffffa80`a33cf750 00000000`00000000 fffffa80`a15b5040 fffff880`09c7c110 : nt!ExDeferredFreePool+0xe86
fffff880`06146ab0 fffff880`09c4a7c5 : fffffa80`a33cf750 00000000`00000000 00000000`00000000 fffff801`a530f4c4 : vhdmp!VhdmpiFreeVirtualDiskSurface+0x6f
fffff880`06146ae0 fffff880`09c4ad8b : fffffa80`a33cf750 fffff880`09c7c110 fffffa80`a33cf918 fffff801`a530f4c4 : vhdmp!VhdmpiDecoupleVirtualDiskSurface+0x16d
fffff880`06146b10 fffff880`09c3ad49 : fffffa80`a33cf8f8 fffff880`09c889a0 fffff801`a550f110 fffffffe`9a5f4400 : vhdmp!VhdmpiHaltActiveSurface+0x123
fffff880`06146b50 fffff801`a533e591 : fffff801`a550f110 fffffa80`977c9b00 fffff801`a52f1300 fffff801`a530f400 : vhdmp!VhdmpiSynchronousWorkerRoutineWrapper+0x19
fffff880`06146b80 fffff801`a52ad535 : fffffa80`64a04660 00000000`00000080 fffff801`a533e450 fffffa80`977c9b00 : nt!ExpWorkerThread+0x142
fffff880`06146c10 fffff801`a52ebe16 : fffff880`03177180 fffffa80`977c9b00 fffff880`03183540 fffffa80`61d0f340 : nt!PspSystemThreadStartup+0x59
fffff880`06146c60 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiStartSystemThread+0x16
STACK_COMMAND: kb
FOLLOWUP_IP:
vhdmp!VhdmpiFreeVirtualDiskSurface+6f
fffff880`09c4a563 488b0da61b0300 mov rcx,qword ptr [vhdmp!WPP_GLOBAL_Control (fffff880`09c7c110)]
SYMBOL_STACK_INDEX: 2
SYMBOL_NAME: vhdmp!VhdmpiFreeVirtualDiskSurface+6f
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: vhdmp
IMAGE_NAME: vhdmp.sys
DEBUG_FLR_IMAGE_TIMESTAMP: 5010aaa8
BUCKET_ID_FUNC_OFFSET: 6f
FAILURE_BUCKET_ID: 0xc2_7_VHDu_vhdmp!VhdmpiFreeVirtualDiskSurface
BUCKET_ID: 0xc2_7_VHDu_vhdmp!VhdmpiFreeVirtualDiskSurface
Followup: MachineOwner
---------
---------------------------------------------------------------------------
Thank you all so much for any help, advice or direction!!