I have several virtual (hyper-v) Windows 2012 R2 servers that crashes at the same time at night with the same error.
Bagcheck code is 0x00000019 (0x0000000000000020, 0xffffe0001bf77000, 0xffffe0001bf772c0, 0x000000000c2c0000)
At this time Sophos AV begins scheduled scan job. Analysis of dump file shows that
Probably caused by : spaceport.sys ( spaceport!SpSpaceDeviceControl+193 )
Excluding spaceport.sys file from scan does not resolve issue.
Here are MEMORY.DMP AND minidump files code:
Minidump
************* Symbol Path validation summary **************
Response Time (ms) Location
Deferred srv*
************* Symbol Path validation summary **************
Response Time (ms) Location
Deferred srv*
Microsoft (R) Windows Debugger Version 6.3.9600.17336 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [C:\Users\V158447\Documents\tmp\071715-12453-01_fs02.dmp]
Mini Kernel Dump File: Only registers and stack trace are available
************* Symbol Path validation summary **************
Response Time (ms) Location
Deferred srv*
************* Symbol Path validation summary **************
Response Time (ms) Location
Deferred srv*
Symbol search path is: srv*
Executable search path is: srv*
Windows 8 Kernel Version 9600 MP (2 procs) Free x64
Product: Server, suite: TerminalServer SingleUserTS
Built by: 9600.17736.amd64fre.winblue_r9.150322-1500
Machine Name:
Kernel base = 0xfffff800`bba18000 PsLoadedModuleList = 0xfffff800`bbcf1850
Debug session time: Fri Jul 17 01:37:47.145 2015 (UTC - 5:00)
System Uptime: 0 days 23:59:11.892
Loading Kernel Symbols
...............................................................
................................................................
......
Loading User Symbols
Loading unloaded module list
.....
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************
Use !analyze -v to get detailed debugging information.
BugCheck 19, {20, ffffe0001bf77000, ffffe0001bf772c0, c2c0000}
Probably caused by : spaceport.sys ( spaceport!SpSpaceDeviceControl+193 )
Followup: MachineOwner
---------
1: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************
BAD_POOL_HEADER (19)
The pool is already corrupt at the time of the current request.
This may or may not be due to the caller.
The internal pool links must be walked to figure out a possible cause of
the problem, and then special pool applied to the suspect tags or the driver
verifier to a suspect driver.
Arguments:
Arg1: 0000000000000020, a pool block header size is corrupt.
Arg2: ffffe0001bf77000, The pool entry we were looking for within the page.
Arg3: ffffe0001bf772c0, The next pool entry.
Arg4: 000000000c2c0000, (reserved)
Debugging Details:
------------------
BUGCHECK_STR: 0x19_20
POOL_ADDRESS: GetPointerFromAddress: unable to read from fffff800bbd7b138
unable to get nt!MmNonPagedPoolStart
unable to get nt!MmSizeOfNonPagedPoolInBytes
ffffe0001bf77000
DEFAULT_BUCKET_ID: WIN8_DRIVER_FAULT_SERVER
PROCESS_NAME: SavService.exe
CURRENT_IRQL: 1
ANALYSIS_VERSION: 6.3.9600.17336 (debuggers(dbg).150226-1500) amd64fre
IRP_ADDRESS: ffffe0001bc8f9b8
LAST_CONTROL_TRANSFER: from fffff800bbcbd0f4 to fffff800bbb68ca0
STACK_TEXT:
ffffd000`22cd71e8 fffff800`bbcbd0f4 : 00000000`00000019 00000000`00000020 ffffe000`1bf77000 ffffe000`1bf772c0 : nt!KeBugCheckEx
ffffd000`22cd71f0 fffff800`bbacb7f4 : 00000000`00000002 00000000`00000000 00000000`00000000 00000000`0000002c : nt!ExAllocatePoolWithTag+0x1204
ffffd000`22cd72e0 fffff800`bbacffe1 : ffffe000`1bc8fa30 ffffd000`22cd7540 00000000`00000001 ffffe000`1bc8fcb3 : nt!IopCompleteRequest+0xc4
ffffd000`22cd7440 fffff800`338069f3 : 00000000`00000000 ffffe000`195ad601 00000000`00800004 ffffc000`bee2f924 : nt!IopfCompleteRequest+0x291
ffffd000`22cd7580 fffff800`3423f808 : 00000000`00000000 ffffd000`22cd76c0 ffffe000`1bc8fcb0 00000000`0004d004 : spaceport!SpSpaceDeviceControl+0x193
ffffd000`22cd75c0 fffff800`3420d84c : ffffe000`1bbde650 ffffd000`22cd7870 00000000`0004d004 fffff800`bbe50806 : CLASSPNP!ClassDeviceControl+0x13b68
ffffd000`22cd7730 fffff800`3422c64c : ffffe000`1bc7f1b0 00000000`00000001 ffffe000`1bc8fa30 ffffe000`195a9190 : disk!DiskDeviceControl+0x8c
ffffd000`22cd77c0 fffff800`33a2805d : ffffe000`1bc8fa30 ffffe000`1bc8fa30 ffffbd68`5f306b00 ffffc000`00000000 : CLASSPNP!ClassDeviceControlDispatch+0x2c
ffffd000`22cd77f0 fffff800`33a1d813 : ffffe000`1bc8fa00 00000000`00000000 00000000`00000000 ffffe000`1bc8fa30 : partmgr!PmIoctlRedirect+0x4d
ffffd000`22cd7860 fffff800`bbe8d9fc : 00000000`00000000 ffffd000`22cd7cc0 ffffe000`1bc7f1b0 fffff800`bbdb7ba3 : partmgr!PmFilterDeviceControl+0xd3
ffffd000`22cd78b0 fffff800`bbe8d6c6 : 00000000`00000000 ffffd000`22cd7cc0 ffffe000`1bc8fcf8 ffffe000`1bc8fa30 : nt!RawReadWriteDeviceControl+0xe4
ffffd000`22cd78f0 fffff800`33c07101 : ffffe000`1be19060 ffffe000`1be19060 ffffe000`1bc8fa30 00000000`00000000 : nt!RawDispatch+0xb6
ffffd000`22cd7960 fffff800`bbe3d77f : 00000000`00000001 ffffd000`22cd7cc0 ffffe000`1bc8fa30 00000000`00000001 : fltmgr!FltpDispatch+0xf1
ffffd000`22cd79c0 fffff800`bbe3cd22 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!IopXxxControlFile+0xa4f
ffffd000`22cd7b60 fffff800`bbb744b3 : ffffd000`22cd7cc0 fffff800`bbe9f99f ffffd000`00000001 00000000`02a6e238 : nt!NtDeviceIoControlFile+0x56
ffffd000`22cd7bd0 00000000`770a2352 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13
00000000`02a6ead8 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x770a2352
STACK_COMMAND: kb
FOLLOWUP_IP:
spaceport!SpSpaceDeviceControl+193
fffff800`338069f3 85db test ebx,ebx
SYMBOL_STACK_INDEX: 4
SYMBOL_NAME: spaceport!SpSpaceDeviceControl+193
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: spaceport
IMAGE_NAME: spaceport.sys
DEBUG_FLR_IMAGE_TIMESTAMP: 54505527
IMAGE_VERSION: 6.3.9600.17415
BUCKET_ID_FUNC_OFFSET: 193
FAILURE_BUCKET_ID: 0x19_20_spaceport!SpSpaceDeviceControl
BUCKET_ID: 0x19_20_spaceport!SpSpaceDeviceControl
ANALYSIS_SOURCE: KM
FAILURE_ID_HASH_STRING: km:0x19_20_spaceport!spspacedevicecontrol
FAILURE_ID_HASH: {856d5b8f-6bfc-cd5c-1543-fe1594ace9a5}
Followup: MachineOwner
---------
1: kd> lmvm spaceport
start end module name
fffff800`33800000 fffff800`33869000 spaceport (pdb symbols) C:\ProgramData\dbg\sym\spaceport.pdb\514293C6261C4E3D8C3584C9494E72F11\spaceport.pdb
Loaded symbol image file: spaceport.sys
Mapped memory image file: C:\ProgramData\dbg\sym\spaceport.sys\5450552769000\spaceport.sys
Image path: \SystemRoot\System32\drivers\spaceport.sys
Image name: spaceport.sys
Timestamp: Tue Oct 28 21:47:03 2014 (54505527)
CheckSum: 00072C4C
ImageSize: 00069000
File version: 6.3.9600.17415
Product version: 6.3.9600.17415
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 3.7 Driver
File date: 00000000.00000000
Translations: 0000.04b0
CompanyName: Microsoft Corporation
ProductName: Microsoft® Windows® Operating System
InternalName: spaceport.sys
OriginalFilename: spaceport.sys
ProductVersion: 6.3.9600.17415
FileVersion: 6.3.9600.17415 (winblue_r4.141028-1500)
FileDescription: Storage Spaces Driver
LegalCopyright: © Microsoft Corporation. All rights reserved.MEMORY.DMP
************* Symbol Path validation summary **************
Response Time (ms) Location
Deferred srv*
************* Symbol Path validation summary **************
Response Time (ms) Location
Deferred srv*
Microsoft (R) Windows Debugger Version 6.3.9600.17336 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [C:\Users\V158447\Documents\tmp\MEMORY_fs02.DMP]
Kernel Bitmap Dump File: Only kernel address space is available
************* Symbol Path validation summary **************
Response Time (ms) Location
Deferred srv*
************* Symbol Path validation summary **************
Response Time (ms) Location
Deferred srv*
Symbol search path is: srv*
Executable search path is: srv*
Windows 8 Kernel Version 9600 MP (2 procs) Free x64
Product: Server, suite: TerminalServer SingleUserTS
Built by: 9600.17736.amd64fre.winblue_r9.150322-1500
Machine Name:
Kernel base = 0xfffff800`bba18000 PsLoadedModuleList = 0xfffff800`bbcf1850
Debug session time: Fri Jul 17 01:37:47.145 2015 (UTC - 5:00)
System Uptime: 0 days 23:59:11.892
Loading Kernel Symbols
...............................................................
................................................................
......
Loading User Symbols
PEB is paged out (Peb.Ldr = 00000000`7f5e6018). Type ".hh dbgerr001" for details
Loading unloaded module list
.....
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************
Use !analyze -v to get detailed debugging information.
BugCheck 19, {20, ffffe0001bf77000, ffffe0001bf772c0, c2c0000}
Probably caused by : spaceport.sys ( spaceport!SpSpaceDeviceControl+193 )
Followup: MachineOwner
---------
1: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************
BAD_POOL_HEADER (19)
The pool is already corrupt at the time of the current request.
This may or may not be due to the caller.
The internal pool links must be walked to figure out a possible cause of
the problem, and then special pool applied to the suspect tags or the driver
verifier to a suspect driver.
Arguments:
Arg1: 0000000000000020, a pool block header size is corrupt.
Arg2: ffffe0001bf77000, The pool entry we were looking for within the page.
Arg3: ffffe0001bf772c0, The next pool entry.
Arg4: 000000000c2c0000, (reserved)
Debugging Details:
------------------
BUGCHECK_STR: 0x19_20
POOL_ADDRESS: unable to get nt!MmNonPagedPoolStart
unable to get nt!MmSizeOfNonPagedPoolInBytes
ffffe0001bf77000
DEFAULT_BUCKET_ID: WIN8_DRIVER_FAULT
PROCESS_NAME: SavService.exe
CURRENT_IRQL: 1
ANALYSIS_VERSION: 6.3.9600.17336 (debuggers(dbg).150226-1500) amd64fre
IRP_ADDRESS: ffffe0001bc8f9b8
LAST_CONTROL_TRANSFER: from fffff800bbcbd0f4 to fffff800bbb68ca0
STACK_TEXT:
ffffd000`22cd71e8 fffff800`bbcbd0f4 : 00000000`00000019 00000000`00000020 ffffe000`1bf77000 ffffe000`1bf772c0 : nt!KeBugCheckEx
ffffd000`22cd71f0 fffff800`bbacb7f4 : 00000000`00000002 00000000`00000000 00000000`00000000 00000000`0000002c : nt!ExAllocatePoolWithTag+0x1204
ffffd000`22cd72e0 fffff800`bbacffe1 : ffffe000`1bc8fa30 ffffd000`22cd7540 00000000`00000001 ffffe000`1bc8fcb3 : nt!IopCompleteRequest+0xc4
ffffd000`22cd7440 fffff800`338069f3 : 00000000`00000000 ffffe000`195ad601 00000000`00800004 ffffc000`bee2f924 : nt!IopfCompleteRequest+0x291
ffffd000`22cd7580 fffff800`3423f808 : 00000000`00000000 ffffd000`22cd76c0 ffffe000`1bc8fcb0 00000000`0004d004 : spaceport!SpSpaceDeviceControl+0x193
ffffd000`22cd75c0 fffff800`3420d84c : ffffe000`1bbde650 ffffd000`22cd7870 00000000`0004d004 fffff800`bbe50806 : CLASSPNP!ClassDeviceControl+0x13b68
ffffd000`22cd7730 fffff800`3422c64c : ffffe000`1bc7f1b0 00000000`00000001 ffffe000`1bc8fa30 ffffe000`195a9190 : disk!DiskDeviceControl+0x8c
ffffd000`22cd77c0 fffff800`33a2805d : ffffe000`1bc8fa30 ffffe000`1bc8fa30 ffffbd68`5f306b00 ffffc000`00000000 : CLASSPNP!ClassDeviceControlDispatch+0x2c
ffffd000`22cd77f0 fffff800`33a1d813 : ffffe000`1bc8fa00 00000000`00000000 00000000`00000000 ffffe000`1bc8fa30 : partmgr!PmIoctlRedirect+0x4d
ffffd000`22cd7860 fffff800`bbe8d9fc : 00000000`00000000 ffffd000`22cd7cc0 ffffe000`1bc7f1b0 fffff800`bbdb7ba3 : partmgr!PmFilterDeviceControl+0xd3
ffffd000`22cd78b0 fffff800`bbe8d6c6 : 00000000`00000000 ffffd000`22cd7cc0 ffffe000`1bc8fcf8 ffffe000`1bc8fa30 : nt!RawReadWriteDeviceControl+0xe4
ffffd000`22cd78f0 fffff800`33c07101 : ffffe000`1be19060 ffffe000`1be19060 ffffe000`1bc8fa30 00000000`00000000 : nt!RawDispatch+0xb6
ffffd000`22cd7960 fffff800`bbe3d77f : 00000000`00000001 ffffd000`22cd7cc0 ffffe000`1bc8fa30 00000000`00000001 : fltmgr!FltpDispatch+0xf1
ffffd000`22cd79c0 fffff800`bbe3cd22 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!IopXxxControlFile+0xa4f
ffffd000`22cd7b60 fffff800`bbb744b3 : ffffd000`22cd7cc0 fffff800`bbe9f99f ffffd000`00000001 00000000`02a6e238 : nt!NtDeviceIoControlFile+0x56
ffffd000`22cd7bd0 00000000`770a2352 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13
00000000`02a6ead8 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x770a2352
STACK_COMMAND: kb
FOLLOWUP_IP:
spaceport!SpSpaceDeviceControl+193
fffff800`338069f3 85db test ebx,ebx
SYMBOL_STACK_INDEX: 4
SYMBOL_NAME: spaceport!SpSpaceDeviceControl+193
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: spaceport
IMAGE_NAME: spaceport.sys
DEBUG_FLR_IMAGE_TIMESTAMP: 54505527
BUCKET_ID_FUNC_OFFSET: 193
FAILURE_BUCKET_ID: 0x19_20_spaceport!SpSpaceDeviceControl
BUCKET_ID: 0x19_20_spaceport!SpSpaceDeviceControl
ANALYSIS_SOURCE: KM
FAILURE_ID_HASH_STRING: km:0x19_20_spaceport!spspacedevicecontrol
FAILURE_ID_HASH: {856d5b8f-6bfc-cd5c-1543-fe1594ace9a5}
Followup: MachineOwner
---------
1: kd> lmvm spaceport
start end module name
fffff800`33800000 fffff800`33869000 spaceport (pdb symbols) C:\ProgramData\dbg\sym\spaceport.pdb\514293C6261C4E3D8C3584C9494E72F11\spaceport.pdb
Loaded symbol image file: spaceport.sys
Image path: \SystemRoot\System32\drivers\spaceport.sys
Image name: spaceport.sys
Timestamp: Tue Oct 28 21:47:03 2014 (54505527)
CheckSum: 00072C4C
ImageSize: 00069000
Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4I also noticed that BSOD appears when Sophos AV tries to scan MBR.
We use Sophos AV inside the VMs.We have 4 of 9 problem systems that were deployed form one template.
When I disable MBR scan in AV - scan completes successfully. But I think that it's not a secure way to resolve the issue.
Could anyone help me to resolve this issue?