Hello all,
I have quite a lot performance issues on a Hyper-V 2012 cluster. I have enabled Dynic Memory for my VM's, but I have the idea that it is not working, or maybe even contra-productive.
What I see is the following: When copying a large file from the network to a "local" disk of the VM (disk is running on an iSCSI volume attached to the hosts), the first couple of GB's are going fine. Then at once, the memory is filled up (the memory
graph in the taskmgr increases liniar to almost 100% usage. The VM becomes very unresponsive and the copy slows down from 115 MB/s to some 100 KB/s.
This is where dynamic memory comes in: The amount of memory that I see in the host is the startup memory (in this case 6 GB). It does not increase, all though there is enough mem free on the host and the max memory is set to, well let's show all values:
PS C:\Users\superadmin> get-vm sp0002-003 | refresh-vm | ft Name,VMAddition,Memory,MemoryAssignedMB,DynamicMemoryBu
fferPercentage,DynamicMemoryDemandMB,DynamicMemoryEnabled,DynamicMemoryMaximumMB,DynamicMemoryMinimumMB,DynamicMemorySta
tus
Name VMAddition Memory MemoryAssig DynamicMemo DynamicMemo DynamicMemo DynamicMemo DynamicMemo DynamicMemo
nedMB ryBufferPer ryDemandMB ryEnabled ryMaximumMB ryMinimumMB ryStatus
centage
---- ---------- ------ ----------- ----------- ----------- ----------- ----------- ----------- -----------
sp0002-0... 6.2.9200... 6144 3238 200 1133 True 40960 2048 OK
So what I see is that the DynamicMemoryDemand is 1133. But in the VM, the memory usage is 98% of 6GB!
Why is this not matching? Please help me troubleshoot this.
Regards,
Stephan van der Plas
Host:
PS C:\Users\superadmin> Get-VMHost iict-srv008
RunAsAccount : superadmin
MostRecentTaskID : 4248ffc7-f65a-4308-95b1-00b44a25afeb
MostRecentTaskUIState : Completed
MostRecentTask : Refresh virtual machine properties
OverallStateString : OK
OverallState : OK
CommunicationStateString : Responding
CommunicationState : Responding
Name : iict-srv008.contoso.local
FullyQualifiedDomainName : iict-srv008.contoso.local
ComputerName : iict-srv008
DomainName : contoso.local
Description :
RemoteUserName :
OverrideHostGroupReserves : False
CPUPercentageReserve : 10
NetworkPercentageReserve : 0
DiskSpaceReserveMB : 10240
MaxDiskIOReservation : 10000
MemoryReserveMB : 2048
VMPaths : {\\?\Volume{f614820c-f7f5-11e1-b1cb-90e6ba0d09c5}\, C:\ClusterStorage\Volume1,
C:\ClusterStorage\Volume2, C:\ClusterStorage\Volume3}
PROEnabled : False
MaintenanceHost : False
AvailableForPlacement : True
IsEmbedded : False
CredentialsNeeded : False
LogicalProcessorCount : 8
PhysicalCPUCount : 1
CoresPerCPU : 8
L2CacheSize : 0
L3CacheSize : 0
BusSpeed : 133
ProcessorSpeed : 2533
ProcessorModel : Xeon
ProcessorManufacturer : Intel
ProcessorArchitecture : 9
ProcessorFamily : 179
CpuUtilization : 7
TotalMemory : 51530297344
AvailableMemory : 20069
OperatingSystem : Microsoft Hyper-V Server 2012
OperatingSystemVersion : 6.2.9200
DVDDrives :
DVDDriveList : {}
VirtualizationPlatformString : Microsoft Hyper-V
VirtualizationPlatform : HyperV
VirtualizationPlatformDetail : Microsoft Hyper-V
IsVirtualizationSoftwareDetailUnknown : False
IsViridianHost : True
SupportsLiveMigration : True
FloppyDrives :
FloppyDriveList : {}
VMHostGroup : All Hosts\IICT\DDC
HostCluster : IICT-HYP-001.contoso.local
RemoteConnectEnabled : True
RemoteConnectPort : 2179
SecureRemoteConnectEnabled : False
VMRCCertificateAvailable : False
EnableLiveMigration : True
LiveMigrationMaximum : 2
LiveStorageMigrationMaximum : 2
UseAnyMigrationSubnet : False
MigrationSubnet : {fe80::34c1:597c:f7ec:5ff/128, 10.100.0.201/32, 10.127.0.23/32, 10.128.0.18/32.
..}
MigrationAuthProtocol : CredSSP
SslTcpPort : 5985
SslCertificateHash :
SshTcpPort : 0
SshPublicKeyHash :
SshPublicKey :
IsRemoteFXRoleInstalled : False
IsCPUSLAT : True
IsNumaSpanningEnabled : True
GPUMemoryTotalMB :
GPUMemoryAvailableMB :
ClusterNodeStatus : Running
TimeZone : 60
HyperVState : Running
HyperVStateString : Running
HyperVVersion : 6.2.9200.16384
HyperVVersionState : UpToDate
PerimeterNetworkHost : False
NonTrustedDomainHost : False
MaximumMemoryPerVM : 1048576
MinimumMemoryPerVM : 8
SuggestedMaximumMemoryPerVM : 512
ModifiedTime : 27-2-2013 11:01:09
Agent : iict-srv008.contoso.local
ManagedComputer : iict-srv008.contoso.local
VMs : {iict-srvp00-005, iict-srvp00-010, IICT-SRVP00-015, IICT-SP0002-005...}
Disks : {\\.\PHYSICALDRIVE2, \\.\PHYSICALDRIVE0, \\.\PHYSICALDRIVE1, \\.\PHYSICALDRIVE4
...}
GPUs : {}
InstalledVirtualSwitchExtensions : {Microsoft VMM DHCPv4 Server Switch Extension, Microsoft Windows Filtering Plat
form, Microsoft Virtual Ethernet Switch Native Extension, Microsoft NDIS Captur
e}
DiskVolumes : {C:\ClusterStorage\Volume1, \\?\Volume{7a6a8259-07d2-11e2-b1c9-806e6f6e6963}\,
C:\ClusterStorage\Volume3, C:\...}
RegisteredStorageFileShares : {}
FibreChannelHbas : {}
SASHbas : {}
InternetSCSIHbas : {ROOT\ISCSIPRT\0000_0}
VMwareResourcePool :
IsConfiguredForOutOfBandManagement : False
PhysicalMachine : f43ba4b0-d804-11dd-937e-53d2c25da7bf
HealthMonitors : {Network, Hyper-V Role Overall, WMI Performance Counter, Overall...}
RemoteStorageTotalCapacity : 33223197026304
RemoteStorageAvailableCapacity : 31444639604736
LocalStorageTotalCapacity : 0
LocalStorageAvailableCapacity : 0
TotalStorageCapacity : 33223197026304
AvailableStorageCapacity : 31444639604736
UsedStorageCapacity : 1778557421568
Custom1 :
Custom2 :
Custom3 :
Custom4 :
Custom5 :
Custom6 :
Custom7 :
Custom8 :
Custom9 :
Custom10 :
CustomProperty : {}
FibreChannelSANStatus : DeployTargetDoesNotHaveHBA (1204)
ISCSISANStatus : DeployHostDoesNotHaveMPIO (1253)
NPIVFibreChannelSANStatus : DeployHostIsNotNPIVCapable (1252)
CertificateRequest :
ComputerState : Responding
VirtualizationManager :
ServerConnection : Microsoft.SystemCenter.VirtualMachineManager.Remoting.ServerConnection
ID : fc26e436-bd3d-444d-a412-5f91636f8e23
IsViewOnly : False
ObjectType : VMHost
MarkedForDeletion : False
IsFullyCached : True
MostRecentTaskIfLocal : Refresh virtual machine properties
FQDN : iict-srv008.contoso.local
LogicalCPUCount : 8
CPUSpeed : 2533
CPUModel : Xeon
CPUManufacturer : Intel
CPUArchitecture : 9
CPUFamily : 179
VMRCEnabled : True
VMRCPort : 2179
SecureVMRCEnabled : False
VirtualServerState : Running
VirtualServerStateString : Running
VirtualServerVersion : 6.2.9200.16384
VirtualServerVersionState : UpToDate
ServicingWindows :
VM:
PS C:\Users\superadmin> Get-VM iict-sp0002-003
VMCPath : C:\ClusterStorage\volume1\sp0002-003\sp0002-003\Virtual Machines\4DBFF8
DA-0B44-40BA-993A-E4B5317181D0.xml
MarkedAsTemplate : False
OwnerIdentifier :
VMId : 4DBFF8DA-0B44-40BA-993A-E4B5317181D0
VMResourceGroup : sp0002-003
VMConfigResource : Virtual Machine Configuration sp0002-003
VMConfigResourceStatus : ClusterResourceOnline
VMResource : Virtual Machine sp0002-003
VMResourceStatus : ClusterResourceOnline
DiskResources : {}
UnsupportedReason : Success (0)
RefresherErrors : {}
VirtualMachineState : Running
HostGroupPath : All Hosts\IICT\DDC\sp0002-003
TotalSize : 222316801024
MemoryAssignedMB : 3044
MemoryAvailablePercentage : 66
DynamicMemoryDemandMB : 1034
DynamicMemoryStatus : OK
AllocatedGPU :
HasPassthroughDisk : False
Status : Running
IsOrphaned : False
HasSavedState : False
StatusString : Running
StartAction : NeverAutoTurnOnVM
StopAction : SaveVM
RunGuestAccount :
DelayStart : 0
CPUUtilization : 0
PerfCPUUtilization : 0
PerfMemory : 3044
PerfDiskBytesRead : 376305
PerfDiskBytesWrite : 4362892
PerfNetworkBytesRead : 749261
PerfNetworkBytesWrite : 17945
VirtualizationPlatform : HyperV
ComputerNameString : SP0002-003.child.contoso.local
CreationSource : Unknown source object
IsUndergoingLiveMigration : False
SourceObjectType : Unknown source object
OperatingSystemShutdownEnabled : True
TimeSynchronizationEnabled : True
DataExchangeEnabled : True
HeartbeatEnabled : True
BackupEnabled : True
ExcludeFromPRO : False
FailedJobID :
CheckpointLocation : C:\ClusterStorage\volume1\sp0002-003\sp0002-003
SelfServiceUserRole :
PassThroughDisks : {}
ComputerTier :
UpgradeDomain :
SCApplications : {}
LastRestoredCheckpointID : 4DBFF8DA-0B44-40BA-993A-E4B5317181D0
LastRestoredVMCheckpoint :
ComplianceStatus :
IsFaultTolerant : False
DeploymentErrorInfo :
ServiceDeploymentErrorMessage :
DeploymentState : Undeployed
TieredPerfData : 14a76033-38ea-4c56-938e-9224db0a6854
ReplicationSetting :
ReplicationStatus :
IsRecoveryVM : False
IsPrimaryVM : False
IsTestReplicaVM : False
IsDREnabled : False
DRState : Disabled
DRErrors : {}
HasDRError : False
ClusterNonPossibleOwner : {}
ClusterPreferredOwner : {}
AvailabilitySetNames :
MostRecentTaskID : b944b719-8d49-40c2-abc9-2d9a7412bcee
MostRecentTaskUIState : Completed
MostRecentTask : Refresh virtual machine
Location : C:\ClusterStorage\volume1\sp0002-003\sp0002-003
CreationTime : 25-2-2013 17:22:07
OperatingSystem : 64-bit edition of Windows Server 2008 R2 Standard
HasVMAdditions : True
VMAddition : Detected
NumLockEnabled : False
CPUCount : 4
IsHighlyAvailable : True
HAVMPriority : 2000
IsDRProtectionRequired : False
LimitCPUFunctionality : False
LimitCPUForMigration : True
Memory : 6144
DynamicMemoryEnabled : True
DynamicMemoryMaximumMB : 40960
DynamicMemoryBufferPercentage : 200
MemoryWeight : 5000
VirtualVideoAdapterEnabled : False
MonitorMaximumCount :
MonitorResolutionMaximum :
BootOrder : {CD, IdeHardDrive, PxeBoot, Floppy}
ComputerName : SP0002-003.child.contoso.local
UseHardwareAssistedVirtualization : True
SANStatus : {DeployVMOnClusterSharedVolume (1291)}
CostCenter :
QuotaPoint : 1
IsTagEmpty : False
Tag : (none)
CustomProperties : {, , , ...}
CustomProperty : {}
UndoDisksEnabled : False
CPUType : 3.60 GHz Xeon (2 MB L2 cache)
ExpectedCPUUtilization : 20
DiskIO : 0
NetworkUtilization : 0
RelativeWeight : 100
CPUReserve : 0
CPUMax : 100
CPUPerVirtualNumaNodeMaximum : 8
MemoryPerVirtualNumaNodeMaximumMB : 45550
VirtualNumaNodesPerSocketMaximum : 1
DynamicMemoryMinimumMB : 2048
NumaIsolationRequired : False
VirtualDVDDrives : {sp0002-003}
VirtualHardDisks : {2008 r2 standard sp1_disk_1, sp0002-003_disk_1}
VirtualDiskDrives : {sp0002-003, sp0002-003}
ShareSCSIBus : False
VirtualNetworkAdapters : {sp0002-003}
VirtualFloppyDrive : sp0002-003
VirtualCOMPorts : {COM1, COM2}
VirtualSCSIAdapters : {sp0002-003}
CapabilityProfile :
CapabilityProfileCompatibilityState : Compatible
VMCheckpoints : {}
HostId : fc26e436-bd3d-444d-a412-5f91636f8e23
HostType : VMHost
HostName : srv008.contoso.local
VMHost : srv008.contoso.local
LibraryServer :
CloudId :
Cloud :
LibraryGroup :
GrantedToList : {}
UserRoleID : 00000000-0000-0000-0000-000000000000
UserRole :
Owner :
ObjectType : VM
Accessibility : Public
Name : sp0002-003
IsViewOnly : False
Description :
AddedTime : 25-2-2013 17:22:07
ModifiedTime : 27-2-2013 10:52:39
Enabled : True
ServerConnection : Microsoft.SystemCenter.VirtualMachineManager.Remoting.ServerConnection
ID : 14a76066-38ea-4c56-938e-9224db0a6854
MarkedForDeletion : False
IsFullyCached : False
MostRecentTaskIfLocal : Refresh virtual machine
ServicingWindows :
You know you're an engineer when you have no life and can prove it mathematically