Greetings everyone,
I'm having an odd situation where the low queue depth random reads and writes on an Intel 905P (Optane, 960GB) PCIe card are cut drastically when running within a Hyper-V VM. This
happens on a Dell R740XD server with dual Xeon Gold CPU's as well as an AMD 1950X workstation, both times using a fully updated full GUI version of Windows 2016.
For testing, I'm using CrystalDiskMark 5.2.0, default settings.
On bare metal, queue depth 1 random reads are around 63500 IOPS and random writes are around 60000 IOPS.
However, if I install the Hyper-V role on that same computer, install a brand new VM that is running a fully updated version of Windows 2016, and then run the benchmark, the random reads
and writes are down by around 2/3 to 16700 read IOPS and 16000 write IOPS.
Example Data:
Intel 905P on bare metal (Threadripper 1950x) - HyperV role not installed
CrystalDiskMark 5.2.0 x64 (C) 2007-2016 hiyohiyo
Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes
Sequential Read (Q= 32,T= 1) : 2709.755 MB/s
Sequential Write (Q= 32,T= 1) : 2378.800 MB/s
Random Read 4KiB (Q= 32,T= 1) : 503.756 MB/s [122987.3 IOPS]
Random Write 4KiB (Q= 32,T= 1) : 473.558 MB/s [115614.7 IOPS]
Sequential Read (T= 1) : 2461.759 MB/s
Sequential Write (T= 1) : 2204.791 MB/s
Random Read 4KiB (Q= 1,T= 1) : 259.845 MB/s [ 63438.7 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 245.729 MB/s [ 59992.4 IOPS]
Test : 4096 MiB [D: 0.0% (0.2/894.3 GiB)] (x5) [Interval=5 sec]
Date : 2018/10/30 14:22:12
OS : Windows Server 2016 Server Standard (full installation) [10.0 Build 14393] (x64)
Intel 905P inside a VM on the same hardware
CrystalDiskMark 5.2.0 x64 (C) 2007-2016 hiyohiyo
Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes
Sequential Read (Q= 32,T= 1) : 2702.230 MB/s
Sequential Write (Q= 32,T= 1) : 2367.909 MB/s
Random Read 4KiB (Q= 32,T= 1) : 516.844 MB/s [126182.6 IOPS]
Random Write 4KiB (Q= 32,T= 1) : 494.393 MB/s [120701.4 IOPS]
Sequential Read (T= 1) : 2184.590 MB/s
Sequential Write (T= 1) : 1975.321 MB/s
Random Read 4KiB (Q= 1,T= 1) : 68.513 MB/s [ 16726.8 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 64.880 MB/s [ 15839.8 IOPS]
Test : 4096 MiB [E: 0.1% (0.1/126.9 GiB)] (x5) [Interval=5 sec]
Date : 2018/10/31 6:13:05
OS : Windows Server 2016 Server Standard (full installation) [10.0 Build 14393] (x64)
The only files on the 905P are a second disk in the VM that I'm running the benchmark on. The OS disk is on another separate drive. I've tried both a fixed size VHDX, a dynamically
assigned VHDX, as well as a physical hard disk attached, all with the same basic result of the QD1 numbers being much smaller.
As this is happening on a Dell R740XD server and an AMD workstation (non-dell), and I've tried other 905P drives just to rule that out, I'm left with Hyper-V. What I think it might
be is some storage IO default throttling in place, but I do not know where to look.
For reference, for a project at work we installed dual Intel P4800X (enterprise version of the 905P) for a 2012 SQL server for TempDB as it helped increase performance. The server
is a Dell R740XD, and I would like to virtualize the instance and take advantage of the SA benefit of unlimited virtualization since we have all the physical cores licensed (Enterprise). Upon initial set up of the physical server before HyperV I ran
benchmarks on the 4800X and all looked great. Once the Hyper-V role was installed and the VM was running I saw the very reduced performance.
I was able to duplicate my results using the 905P, so that is where my testing is happening now.
Any thoughts on where to look?
Appreciate it everyone,
Mark