We have a problem with one of our deployments of Windows Server 2012 Hyper-V with a 2 node cluster connected to a iSCSI SAN.

Our setup:

Hosts - Both run Windows Server 2012 Standard and are clustered.

• HP ProLiant G7, 24 GB RAM. This is the primary host and normaly all VMs run on this host.
• HP ProLiant G5, 20 GB RAM. This is the secondary host that and is intended to be used in case of failure of the primary host.
• We have no antivirus on the hosts and the scheduled ShadowCopy (previous version of files) is switched off.

iSCSI SAN:

• QNAP NAS TS-869 Pro, 8 INTEL SSDSA2CW160G3 160 GB i a RAID 5 with a Host Spare. 2 Teamed NIC.

Switch:

• DLINK DGS-1210-16 - Both the network cards of the Hosts that are dedicated to the Storage and the Storage itself are connected to the same switch and nothing else is connected to this switch.

Virtual Machines:

• 3 Windows Server 2012 Standard - 1 DC, 1 FileServer, 1 Application Server.
• 1 Windows Server 2008 Standard Exchange Server.
• All VMs are using dynamic disks (as recommended by Microsoft).

Updates

• We have applied the most resent updates to the Hosts, VMs and iSCSI SAN about 3 weeks ago with no change in our problem and we continually update the setup.

Normal operation:

• Normally this setup works just fine and we see no real difference in speed in startup, file copy and processing speed in LoB applications of this setup compared to a single host with two 10000 RPM Disks. Normal network speed is 10-200 Mbit, but occasionally we see speeds up to 400 Mbit/s of combined read/write for instance during file repair.

Our Problem:

• Our problem is that for some reason a random VHDX gets copied to System Volume Information by "System" of the Clusterd Shared Storage (i.e. C:\ClusterStorage\Volume1\System Volume Information).
• All VMs stops responding or responds very slowly during this copy process and you can for instance not send CTRL-ALT-DEL to a VM in the Hyper-V console, or for instance start task manager when already logged in.
• This happens at random and not every day and different VHDX files from different VMs gets copied each time. Some time it happens during daytime wich causes a lot of problems, especially when a 200 GB file gets copied (which take a lot of time).

What it is not:

• We thought that this was connected to the backup, but the backup had finished 3 hours before the last time this happended and the backup never uses any of the files in System Volume Information so it is not the backup.

An observation:

• When this happend today I switched on ShadowCopy (previous files) and set it to only to use 320 MB of storage and then the Copy Process stopped and the virtual Machines started responding again. This could be unrelated since there is no way to see how much of the VHDX that is left to be copied, so it might have been finished at the same time as I enabled  ShadowCopy (previos files).

Our question:

• Why is a VHDX copied to System Volume Information when scheduled ShadowCopy (previous version of files) is switched off? As far as I know, nothing should be copied to this folder when this functionis switched off?

List of VSS Writers:

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2012 Microsoft Corp.

Writer name: 'Task Scheduler Writer'
   Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
   Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
   State: [1] Stable
   Last error: No error

Writer name: 'VSS Metadata Store Writer'
   Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
   Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
   State: [1] Stable
   Last error: No error

Writer name: 'Performance Counters Writer'
   Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
   Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
   State: [1] Stable
   Last error: No error

Writer name: 'System Writer'
   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Instance Id: {7848396d-00b1-47cd-8ba9-769b7ce402d2}
   State: [1] Stable
   Last error: No error

Writer name: 'Microsoft Hyper-V VSS Writer'
   Writer Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}
   Writer Instance Id: {8b6c534a-18dd-4fff-b14e-1d4aebd1db74}
   State: [5] Waiting for completion
   Last error: No error

Writer name: 'Cluster Shared Volume VSS Writer'
   Writer Id: {1072ae1c-e5a7-4ea1-9e4a-6f7964656570}
   Writer Instance Id: {d46c6a69-8b4a-4307-afcf-ca3611c7f680}
   State: [1] Stable
   Last error: No error

Writer name: 'ASR Writer'
   Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
   Writer Instance Id: {fc530484-71db-48c3-af5f-ef398070373e}
   State: [1] Stable
   Last error: No error

Writer name: 'WMI Writer'
   Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
   Writer Instance Id: {3792e26e-c0d0-4901-b799-2e8d9ffe2085}
   State: [1] Stable
   Last error: No error

Writer name: 'Registry Writer'
   Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
   Writer Instance Id: {6ea65f92-e3fd-4a23-9e5f-b23de43bc756}
   State: [1] Stable
   Last error: No error

Writer name: 'BITS Writer'
   Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}
   Writer Instance Id: {71dc7876-2089-472c-8fed-4b8862037528}
   State: [1] Stable
   Last error: No error

Writer name: 'Shadow Copy Optimization Writer'
   Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
   Writer Instance Id: {cb0c7fd8-1f5c-41bb-b2cc-82fabbdc466e}
   State: [1] Stable
   Last error: No error

Writer name: 'Cluster Database'
   Writer Id: {41e12264-35d8-479b-8e5c-9b23d1dad37e}
   Writer Instance Id: {23320f7e-f165-409d-8456-5d7d8fbaefed}
   State: [1] Stable
   Last error: No error

Writer name: 'COM+ REGDB Writer'
   Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
   Writer Instance Id: {f23d0208-e569-48b0-ad30-1addb1a044af}
   State: [1] Stable
   Last error: No error

Please note:

• Please only answer our question and do not offer any general optimization tips that do not directly adress the issue! We want the problem to go away, not to finish a bit faster!

Hello!

We have Hyper-V Server 2012 installed on 4 machines with identical hardware configuration. All of them have identical BIOS settings (triple-checked that).

NUMA is enabled in BIOS. Each machine has 2 sockets with 6-core 12-thread (SMT) processors in each and 96GB RAM in total, hence it has two NUMA nodes with 12 cores and 48GB RAM.

On all of these machines there are virtual machines with identical configuration, which are configured with 24 processors and 12GB of RAM. No virtual machines have configured reservations, neither do have dynamic memory. All machines configured using "Use Hardware Topology" in the Processors->NUMA configuration page.

Three of VMs start fine. The other one appears to start normally, however the event log of the hypervisor contains the following warning:

Event ID: 3056

Event Source: Microsoft-Windows-Hyper-V-Worker

Message: 'vm-name' is configured to have a different non-uniform memory architecture (NUMA) topology to that of the physical machine. Some virtual processors are accessing memory on a remote NUMA node. This configuration is not recommended because it will reduce performance. (Virtual machine ID {...})

Unfortunately, this event appears to be undocumented (I couldn't find any relevant information neither on TechNet, nor using web search).

I tried creating another VM from scratch and it exhibits the same problem.

Do you have any ideas I should check, or maybe I'm missing something here?

P.S. The guest OS is a Windows Server 2012, but I don't think it matters, since the warning is logged even for empty machine that attempts to span more than 1 NUMA nodes (i.e. specifying more than 12 processors).

Hello,

I've got two fresh installations of a 4 node and a 2 node Hyper-V 2012 Cluster and two SCVMM 2012SP1.

When I add the cluster to VMM it installs the agent and dhcp switch extension. This causes my cluster and all running VMs to crash.

All Servers are up to date with the latest updates from Microsoft Update.

Language of all systems is German (de-de)

I notived this in two different isolated lab scenarios.

LAB1:

4 Hyper-V Hosts 6 1GBE NICs each

• two NICs for iSCSI with a vSwitch to provide iSCSI for virtual machines or virtual clusters
• iSCSI uses MPIO and 2 Switches
• DELL Equallogic Host Integration Tools (4.5) are installed and configured for Hyper-V cluster use on the Hosts
• 4 NICs setup as a "LACP" Team using "Hyper-V Port"
• 3 virtual interfaces for the Host (Host-Management, CSV and LM)
• The Host Team is connect to a single switch - LACP Status on host and switch is fine
• besides the DELL HIT, no additional software is installed
• VMM is running as a VM in the cluster
• the cluster hosts about 70 LAB-VMs

LAB2:

2 Hyper-V with 10 NICs each

• 2 10GBE NICs for iSCSI using MPIO again a vSwitch is used to provide VM access to the SAN
• 8 1GBE NICs as a Team using LACP and Hyper-V Port
• virtual NICs for Host Management, CSV and LM are added to the Host OS
• All NICs are connected to a 2 switch stack
• LACP status on the hosts and switches is ok
• No additional software is installed
• VMM is running as a VM in the cluster
• at the moment no additional VMs are running in this cluster

As soon as I start to add the cluster to VMM and the agent installation begins, all NICs on the cluster nodes got disconnect.

The cluster fails and all VMs including VMM are crashing.

I haved used exactly the same setup with SCVMM 2012SP1 CTP without any problems. The only difference seems to be the dhcp switch extension with is rolled out by default now. I have not used the extension in my previous labs, because there was no need for it.

Can anyone give me a hint a or a solution to avoid this problem.

Hi,

How does one provision iscsi disks in a core installation of Hyper-v. I have used the iSCSI initiator gui but cannot find a way of provisioning with disk
management.<o:p></o:p>

Any help much appreciated.<o:p></o:p>

Paul<o:p></o:p>

Paul Edwards

Hi All,

Can we use Hyper V replication for Exchange VM. Will it work smoothly.

Thanks

There were already a few discussions about the bandwidth requirement for Hyper-V replication on this forum but I could not find a good estimate that was useful for my situation. I realize that it is hard to be very precise about bandwidth requirements because everything depends on how much data will be changed. I will try to be as precise as possible in the description and I hope that you can give me a green light or if it is totally not possible what I want a red light.

We will have two racks in the Amsterdam area connected with a 100 Mbit/s WAN connection. This is not a shared Internet connection, but is a dedicated non shared (resilient) connection between the two racks with the main purpose of replication. There will be 7 physical servers at each location witch 96 GB RAM. On the Hyper-V servers will run a total of about 35 VM’s. Probably all VM’s will have 2 VHDX disks.

There will be a few supporting servers like domain controllers and update servers (WSUS), which will not be replicated.

The rest of the servers will be:

11 webservers

14 application servers

5 database servers.

The total server park is designed to run a search service for our customers. Data is indexed during the night and then you will see the biggest changes on de virtual harddisks. The data and indexing is BTW not done on the database servers. The database servers are mainly for the look and feel of the webservers. During the day customers query the indexes, these are read queries. The total amount of harddisk space is 7.3 TB. The total amount of index data (which changes the most) is 600 GB.

My idea is to replicate often during the day (5 minutes) but stop replication during the indexing period. I assume that I can change the replication interval with powershell.

Although it is hard to get an answer here about SAN technology because Microsoft does not make SAN’s, my equallogic SAN’s can also replicate. I tend to prefer replication with Hyper-V because than I am independent of future hardware decisions but, if SAN replication is more reliable or has other advantages than I can also use the SAN’s to replicate. I think that I read or heard somewhere on the Microsoft technet website that Hyper-V is not supported for enterprise environments. Just 30 VM’s is not an enterprise environment so I hope what I want is fully supported by Microsoft. If it is however better to do this on the SAN’s then I will do that. What would you advise?

Hello,

I'm relatively new to Server 2012 and Hyper-V but I just finished setting up our system and I need some consulting on how to monitor and optimize the CPU usage.   I know I could spend days and weeks researching, reading, tweaking and testing (and I should), but I dove right into a live production setup and I'm hoping to find an expert to say "do this and do that".

Our situation is as follows:

• - Hardware node runs Windows Server 2012 Standard with Hyper-V  (Intel Core i7 3770 3.4ghz with 32GB RAM and 240GB SSD.)
  
• - All Guest VM's run Windows Server 2003 Standard x86
  
• - We run the Metatrader 4 software (15-20 instances per VM) which is a 32-bit, single-thread application whose primary resource usage is CPU and experiences the highest usage during active Forex markets (3am-11am Eastern Time).  MT4 is old (designed for Windows XP) and seems to run best on XP/2003.

My wish list of things I would like to learn are:

• How to monitor the CPU usage, with a visual graph, over a 1-3 day period so we can see the spikes and dips (our software uses more resources between 3am-11am EST). 
  
• How to pinpoint which VM's are using more CPU resources than the others (using a similar 1-3 day visual graph) so we can lighten the load on certain servers or increase the load on others that are being too burdened.
  
• Whether or not we should use the CPU limiting options (Virtual Machine Reserve, Virtual Machine Limit, etc.)  (We will have 4-6 VMs per node, all owned by us, no third-party users so I'm not worried about abuse.)
  
• Whether or not we should use multiple Virtual Processors in each VM or just 1.  (Our software is all single-threaded but we run 15-20 instances per server.  I currently have all the VMs set up with 4 vCPUs).
  
• What other performance "tuning" should be done on the hardware nodes to optimize the resources for our situation?
  

I sincerely appreciate any advice that can be offered.

Scott

Hi Everyone, 

I know this is a stuip and lame questions, but I have looked at a lot of Hyper-V stuff over the weekend and I was planing on installing it on my spare computer to play with it .. 

But the one thing I cant find is , does it need Windows Server installed on the computer too ?? or can you just install it like you would any normal OS and just boot to CD .. 

Please let me know and thanks for looking at such a stuip question ... 

Hi,

I (as a proffesor) use Virtual PC in classroom simulating Windows XP connecting to domain on Windows Server 2003. I have created organizational unit in active directory and applied group policy with for exampe two restrictions: Control Panel disabled and not allowing Windows Movie Maker to run (in computer section). On older machines (P4~) group policy mainly have not been appied! Last school year (2011/12) I have woked in newer classroom- dual core processors... and everything has worked perfectly- "old" problem occured again when I have tried same thing with Windows Server 2008. Than I have "concluded" that "problem" is due to weak processor performance. This year "problem" occured in the "new" classroom with dual core processors- on almost all (except 2) computers! Can someone help me with this issue?. /in some similar problems as a solution was suggested changing network settings which I have tried but with no result.../

Thank You in advance, sincerely Drago Percic

Most customer deployments require communication from the network virtualized environment to the non-network virtualized environment. Therefore Hyper-V Network Virtualization gateways are required to bridge the two environments. The gateway is designed for when the VMs within the virtualized network need to communicate with physical devices outside the Virtualized Network. (source Technet)

Question 1 : Can I use a forefront TMG to act as a virtualization gateway ?

Example :   INTERNET (briged to physical network card)  <<====>> FOREFRONT TMG VM <<====>>  LAN (Virtualised Network) <<==>> client VM in Virtualised Network

I have already tried to give internet access to a VM machine in a virtualized network throught Forefront TMG but i was no able to do it.The machine was able to communicate with TMG but not with internet. (It was working with a non virtualised netwwork.)

My configuration example : (Source : Keithmayer blog on Technet)

My Forefront TMG IP adress is 10.1.1.11 and my client machine is 10.1.1.12

1. Enable the Windows Network Virtualization binding on the physical NIC of each Hyper-V Host ( Host 1 and Host 2 )

Enable-NetAdapterBinding Ethernet -ComponentID ms_netwnv

2. Configure Blue Subnet Locator and Route records on each Hyper-V Host ( Host 1 and Host 2 )

New-NetVirtualizationLookupRecord -CustomerAddress "10.1.1.11" -ProviderAddress "192.168.1.10" -VirtualSubnetID "6001" -MACAddress "101010101105" -Rule "TranslationMethodEncap" 
New-NetVirtualizationLookupRecord -CustomerAddress "10.1.1.12" -ProviderAddress "192.168.1.20" -VirtualSubnetID "6001" -MACAddress "101010101107" -Rule "TranslationMethodEncap" 
New-NetVirtualizationCustomerRoute -RoutingDomainID "{11111111-2222-3333-4444-000000000000}" -VirtualSubnetID "6001" -DestinationPrefix "10.1.1.0/24" -NextHop "0.0.0.0" -Metric 255 

3. Configure the Provider Address and Route records on Hyper-V Host 1 ( Host 1 Only )

$NIC = Get-NetAdapter Ethernet
New-NetVirtualizationProviderAddress -InterfaceIndex $NIC.InterfaceIndex -ProviderAddress "192.168.1.10" -PrefixLength 24 
New-NetVirtualizationProviderRoute -InterfaceIndex $NIC.InterfaceIndex -DestinationPrefix "0.0.0.0/0" -NextHop "192.168.1.1" 

4. Configure the Provider Address and Route records on Hyper-V Host 2 ( Host 2 Only )

$NIC = Get-NetAdapter Ethernet
New-NetVirtualizationProviderAddress -InterfaceIndex $NIC.InterfaceIndex -ProviderAddress "192.168.1.20" -PrefixLength 24 
New-NetVirtualizationProviderRoute -InterfaceIndex $NIC.InterfaceIndex -DestinationPrefix "0.0.0.0/0" -NextHop "192.168.1.1" 

5. Configure the Virtual Subnet ID on the Hyper-V Network Switch Ports for each "Blue" Virtual Machine on each Hyper-V Host ( Host 1 and Host 2 )

Get-VMNetworkAdapter -VMName Forefront | where {$_.MacAddress -eq "101010101107"} | Set-VMNetworkAdapter -VirtualSubnetID 6001Get-VMNetworkAdapter -VMName Client | where {$_.MacAddress -eq "101010101105"} | Set-VMNetworkAdapter -VirtualSubnetID 6001 

Question 2 : Why myy packets are not routed Thourought forefront TMG ?

I hope you can give me some help.
Aurel

I have a pre-sale service question. I'm not sure if I should post it here or other place. As I know, Microsoft Hyper-V Server is free to use. But how can I get technical supports like E-mail/phone/on-site support (except free forum support) in case of critical issues? Or I have to buy the expensive Windows Server and use the Hyper-V component? Thanks!

I have been asked to provide some comparative costs for either a  VMWare upgrade to vSphere 5.1 compared to a migration to Hyper-V. Has any body got any experience / advice on this?

Tools required?

V2V time scales?

Risks?

Windows 2012 Hyper V v3 Hardware compatibility?

Hi Team, 

I got 2 brand new servers installed with Hyper V Server 2012 as the base and build a couple VMs on top.  For WSUS, I have a WSUS 3.0 SP2 running on a Server 2008 R2 Machine.  From the WSUS Server, I have installed the KB for WSUS to shows correct Windows Version and the WSUS is showing 5 updates is needed for the Hyper V Server.  Therefore, I logged in as Admin for the Hyper V and 6) for Download and Install Update.  Then A for All Updates and the Hyper V still also listed as 5 updates required. Then, I select A for all updates.

So, the Hyper V Servers shows Downloading Update.... Installing Update....  but Listing of updates installed and individual installation results show NOTHING and ask for reboot.  Rebooted and no updates is installed.  6 for download and install updates again and still shows 5 updates required.  

How can i get this fix and deploy updates to the Hyper V Machine?

I've sort of asked this question before but I think I might need to rephrase it. Forgive the double post (this was ages ago).

OK I have a billion Hyper-V hosts running 2008 R2.

I want to profile the I/O of the VMs running on the hypervisor. I want to do this from the parent partition.

Information I need  in in order of importance:

- Combined IOPS

- Individual read\write IOPS figures

- I'm not holding out much hope, but maybe some idea of I/O block size?

Previously I was told to look at the 'Hyper-V Virtual Storage Device' counters. I see the following:

Error Count

Flush Count

Read Bytes / Sec

Write Bytes / Sec

Read Count

Write Count

I have some questions:

- Are these 'read\write count' counters operations over x period of time? If so, what is this period of time? Is it a second?

- Could someone please point me in the direction of documentation that would help me understand these counters in depth (I hate making assumptions) or otherwise school me in how I can get the information I need from these counters? I don’t mind what the sample rate is as long as I can derive operations per second from it.

- N.B. I really hate PerfMon, I will pull this data out with Powershell somehow. Right now I'm just looking in Perfmon so I can understand the information. I do understand that Perfmon has a sample duration etc but I can't translate the numbers into anything that looks sensible.

Just to note that I cannot look at the counters in the guest operating systems. So this is out of the window.

Second issue:

I also asked this question before, kinda, but I'll ask it again slightly rephrased.

Does anyone have information (white papers, documents) regarding the impact and i/o penalty of a.) differencing disks over fixed disks in terms of IOPS and also how the Hyper-V VHD container affects IOPS calculations? (I'll probably migrate everything to 2012 and VHDX if this makes any odds).

This might be me not understanding things properly, but I'm curious about how the container impacts the calculations I need to make.

For instance, we know that if you have a 'required  IOPS' figure for a given a deployment you have to take into account the penalty of a RAID array (and other things), we know that - for example - withRAID6 we have to bear in mind the increased parity calculations etc.

If I know that I need to run 2 VMs on a hypervisor and I know that both of those VMs will be doing 50 IOPSs, do either VHD format affect how many I/O operations the host actually to perform? I doubt this is the case? I guess that the only real impact is that a differencing disk is likely to (end up) necessitating more random I/O while a fixed disk is sequential (as far as the hypervisor is concerned)?

Bit of a ramble but hopefully makes sense.

If anyone knows of a Powershell I/O profiler for Hyper-V that would be the ideal solution :) If there isn’t one I will post back with the results of what I come up with. 

Hello,

I would like to know if it is possible to reduce the size of a dynamic xvhd disk.

Im running Windows Server 2012. The scenario is: 

40GB dynamic disk. 30 GB is used.. when I compact the disk it clearly shows on the VM settings (inspect disk) that it is reduce to 10GB.

The problem is when I check the actual .xvhd on windows explorer, it still shows the 30GB size.

Thanks!

I am running a esxi lab within hyper-v and the vms within this do not want to connect to the network. I have been told that I need to enable promiscuous mode on the virtual switch and I can't find the option anywhere. I cannot get guest OSes to connect to the network under the vmware vsphere. 

Is there any way to enable this? Please respond. Thank you. 

I am testing an unplanned failover. In the technet instructions (http://technet.microsoft.com/en-us/library/jj134169.aspx) to complete the failover I am supposed to issue this powershell command

Complete-VMFailover -VMName $ReplicaVM2

I receive an error (see screen shot below) when I try to execute this command and I am not sure how to trouble shoot this. The failover appears to have worked without isse, there are not errors in the logs, and the applications are all performing as expected, so I am not sure what this command actually does.

Any advice is greatly apprecated.

davidh