I am converting a small office from a physical server infrastructure to a virtual infrastructure using Hyper-V and iSCSI storage. I am using Hyper-V Server 2012 R2 (no GUI) on a Dell PE R430 with a 5-bay Synology. Using 2 x 1Gbps NICs for iSCSI with MPIO. I appear to have good iSCSI sessions established, a VM that is running on the LUN that can transfer files quickly from the host OS drives to the VMDK on the LUN, BUT my issues is the disk latency on the VM is horrendous. Within the VM the disk time is always at 100% and the response time is on average 600ms to 3000ms. I feel lost without my GUI right now, but any help would be greatly appreciated.
Host: Dell PowerEdge R430 (1 x Xeon 6-core CPU, 32GB RAM, 6 x 1Gbps NICS, and RAID1 with 2 x 120GB Intel SSDs)
Storage: Synology DS1515+ with SHR (5 x 2TB WD Red) and 2 NICs for LAN and 2 NICs for iSCSI
To get them started I setup a new Dell PowerEdge R430 and installed Microsoft Hyper-V Server 2012 R2.Typically I have installed Microsoft Windows Server 2012 R2 with Hyper-V installed so I can have the GUI as needed. However, this time I thought I would go with the free core version to save on software costs and I didn't have any trouble configuring this in labs from time to time. I have a Synology DS1515+ with a 5-drive SHR and a 7TB LUN for VM files and a 1GB LUN for a Witness disk as I plan to make this into a Failover Cluster with a CSV.
On the host I used the New-NetLbfoTeam cmdlet to setup a 2 NIC Team for Mgmt LAN, a 2 NIC Team for Hyper-V LAN vSwitch, a 2 separate NICs for iSCSI traffic. I use netsh commands and Get-NetAdapter cmdlets to set the IP addresses and made sure there were not any MAC address conflicts.
I used Keith Mayer's guide (http://blogs.technet.com/b/keithmayer/archive/2013/03/12/speaking-iscsi-with-windows-server-2012-and-hyper-v.aspx) to setup the iSCSI connection and sessions to my iSCSI Target on my Synology. I also setup MPIO. I created a new VM running Windows 8.1 for management.
I noticed that the VM wasn't performing well. I found the disk time was nearly always at 100% and the Disk response times were extremely bad ranging from 60ms to 50000ms. I would say the consistent good was around 600ms and the spikes were on average in the 3000ms range. Absolutely awful. I have more background in VMware and so I don't know really where to begin. It can transfer large files from the host local disk to the VM disk around 80 MB/s, but the experience shows the latency.
Two more examples. If I start the VM and open Task Manager and go to the Performance tab and select Disk it will show the response time (Also look in Resource Manager) and it will look okay <20ms, but open Chrome or Resource Monitor and spikes to >3000ms. Also, the System and Explorer processes show an average of 300-500ms response time at essentially idle. I think you get the idea.
I have installed the latest DSM on the Synology and I have installed missing updates to the OS on the host. I tried removing the iSCSI target connections and adding them back on the host, but that didn't help either. Restarted the host, storage, and VM multiple times in the process of troubleshooting. About to phone Microsoft.
Including some random outputs from PowerShell on the iSCSI and network stuff.
PS Info:PS C:\Users\ben> get-iscsiconnection
ConnectionIdentifier : ffffe00167f5a020-0
InitiatorAddress : 0.0.0.0
InitiatorPortNumber : 1984
TargetAddress : 10.1.1.51
TargetPortNumber : 3260
PSComputerName :
PS C:\Users\ben> Get-IscsiSession
AuthenticationType : NONE
InitiatorInstanceName : ROOT\ISCSIPRT\0000_0
InitiatorNodeAddress : iqn.1991-05.com.microsoft:client-hv01
InitiatorPortalAddress : 0.0.0.0
InitiatorSideIdentifier : 400001370000
IsConnected : True
IsDataDigest : False
IsDiscovered : False
IsHeaderDigest : False
IsPersistent : True
NumberOfConnections : 1
SessionIdentifier : ffffe00167f5a020-4000013700000001
TargetNodeAddress : iqn.2000-01.com.synology:client-ds01.target-1.644bfbc96d
TargetSideIdentifier : 0200
PSComputerName :
PS C:\Users\ben> Get-IscsiTargetPortal
InitiatorInstanceName : ROOT\ISCSIPRT\0000_0
InitiatorPortalAddress : 10.1.1.11
IsDataDigest : False
IsHeaderDigest : False
TargetPortalAddress : 10.1.1.51
TargetPortalPortNumber : 3260
PSComputerName :
InitiatorInstanceName : ROOT\ISCSIPRT\0000_0
InitiatorPortalAddress : 10.1.1.12
IsDataDigest : False
IsHeaderDigest : False
TargetPortalAddress : 10.1.1.51
TargetPortalPortNumber : 3260
PSComputerName :
PS C:\Users\ben> Get-Disk | Where-Object BusType -eq "iSCSI"
Number Friendly Name OperationalStatus Total Size Partition Style
------ ------------- ----------------- ---------- ---------------
1 SYNOLOGY iSCSI Storage Online 4 TB GPT
2 SYNOLOGY iSCSI Storage Online 1 GB GPT
PS C:\Users\ben> get-partition
Disk Number: 0
PartitionNumber DriveLetter Offset Size Type
--------------- ----------- ------ ---- ----
1 1048576 350 MB IFS
2 C 368050176 110.91 GB IFS
Disk Number: 1
PartitionNumber DriveLetter Offset Size Type
--------------- ----------- ------ ---- ----
1 17408 128 MB Reserved
2 E 135266304 4 TB Basic
Disk Number: 2
PartitionNumber DriveLetter Offset Size Type
--------------- ----------- ------ ---- ----
1 17408 32 MB Reserved
2 F 33619968 991.88 MB Basic
PS C:\Users\ben> ping 10.1.1.51
Pinging 10.1.1.51 with 32 bytes of data:
Reply from 10.1.1.51: bytes=32 time=1ms TTL=64
Reply from 10.1.1.51: bytes=32 time<1ms TTL=64
Reply from 10.1.1.51: bytes=32 time<1ms TTL=64
Reply from 10.1.1.51: bytes=32 time<1ms TTL=64
Ping statistics for 10.1.1.51:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 1ms, Average = 0ms
PS C:\Users\ben>
PS C:\Users\ben> get-netadapter
Name InterfaceDescription ifIndex Status MacAddress LinkSpeed
---- -------------------- ------- ------ ---------- ---------
StorageNIC4 Broadcom NetXtreme Gigabit Ethernet #4 17 Up 44-A8-42-39-3C-B1 1 Gbps
StorageNIC3 Broadcom NetXtreme Gigabit Ethernet #3 16 Up 44-A8-42-39-3C-B0 1 Gbps
Ethernet 3 Realtek PCIe GBE Family Controller #2 18 Up 00-13-3B-10-49-87 1 Gbps
NIC2 Broadcom NetXtreme Gigabit Ethernet #2 15 Up 44-A8-42-39-3C-AF 1 Gbps
NIC1 Broadcom NetXtreme Gigabit Ethernet 13 Up 44-A8-42-39-3C-AE 1 Gbps
vEthernet (Mgmt LAN) Hyper-V Virtual Ethernet Adapter #4 31 Up 44-A8-42-39-3C-04 10 Gbps
vEthernet (Hyper-V LAN) Hyper-V Virtual Ethernet Adapter #2 29 Up 00-13-3B-10-49-02 10 Gbps
LAN Microsoft Network Adapter Multiple...#2 25 Up 44-A8-42-39-3C-02 2 Gbps
Hyper-V Microsoft Network Adapter Multiple...#3 27 Up 00-13-3B-10-49-03 2 Gbps
Ethernet 2 Realtek PCIe GBE Family Controller 14 Up 00-13-3B-10-49-88 1 Gbps
PS C:\Users\ben> netsh interface show interface
Admin State State Type Interface Name
-------------------------------------------------------------------------
Enabled Connected Dedicated Ethernet 2
Enabled Connected Dedicated Hyper-V
Enabled Connected Dedicated LAN
Enabled Connected Dedicated vEthernet (Hyper-V LAN)
Enabled Connected Dedicated vEthernet (Mgmt LAN)
Enabled Connected Dedicated NIC1
Enabled Connected Dedicated NIC2
Enabled Connected Dedicated Ethernet 3
Enabled Connected Dedicated StorageNIC3
Enabled Connected Dedicated StorageNIC4
PS C:\Users\ben>