Hi All

Apologies if this is a long one but I thought the more information I can provide the better.

We have recently designed and built a new Hyper-V environment for a client, utilising Windows Server R2 / System Centre 2012 R2 however since putting it into production, we are now seeing problems with Virtual Machine Queues. These manifest themselves as either very high latency inside virtual machines (we’re talking 200 – 400 mSec round trip times), packet loss or complete connectivity loss for VMs. Not all VMs are affected however the problem does manifest itself on all hosts. I am aware of these issues having cropped up in the past with Broadcom NICs.

I'll give you a little bit of background into the problem...

Frist, the environment is based entirely on Dell hardware (Equallogic Storage, PowerConnect Switching and PE R720 VM Hosts). this environment was based on Server 2012 and a decision was taken to bring this up to speed to R2. This was due to a number of quite compelling reasons, mainly surrounding reliability. The core virtualisation infrastructure consists of four VM hosts in a Hyper-V Cluster.

Prior to the redesign, each VM host had 12 NICs installed:

• Quad port on-board Broadcom 5720 daughter card: Two NICs assigned to a host management team whilst the other two NICs in the same adapter formed a Live Migration / Cluster heartbeat team, to which a VM switch was connected with two vNICs exposed to the management OS. Latest drivers and firmware installed. The Converged Fabric team here was configured in LACP Address Hash (Min Queues mode), each NIC having the same two processor cores assigned. The management team is identically configured.

• Two additional Intel i350 quad port NICs: 4 NICs teamed for the production VM Switch uplink and 4 for iSCSI MPIO. Latest drivers and firmware. The VM Switch team spans both physical NICs to provide some level of NIC level fault tolerance, whilst the remaining 4 NICs for ISCSI MPIO are also balanced across the two NICs for the same reasons.

The initial driver for upgrading was that we were once again seeing issues with VMQ in the old design with the converged fabric design. The two vNics in the management OS for each of these networks were tagged to specific VLANs (that were obviously accessible to the same designated NICs in each of the VM hosts).

In this setup, a similar issue was being experienced to our present issue. Once again, the Converged Fabric vNICs in the Host OS would on occasion, either lose connectivity or exhibit very high round trip times and packet loss. This seemed to correlate with a significant increase in bandwidth through the converged fabric, such as when initiating a Live Migration and would then affect both vNICS connectivity. This would cause packet loss / connectivity loss for both the Live Migration and Cluster Heartbeat vNICs which in turn would trigger all sorts of horrid goings on in the cluster. If we disabled VMQ on the physical adapters and the team multiplex adapter, the problem went away. Obviously disabling VMQ is something that we really don’t want to resort to.

So…. The decision to refresh the environment with 2012 R2 across the board (which was also driven by other factors and not just this issue alone) was accelerated.

In the new environment, we replaced the Quad Port Broadcom 5720 Daughter Cards in the hosts with new Intel i350 QP Daughter cards to keep the NICs identical across the board. The Cluster heartbeat / Live Migration networks now use an SMB Multichannel configuration, utilising the same two NICs as in the old design in two isolated untagged port VLANs. This part of the re-design is now working very well (Live Migrations now complete much faster I hasten to add!!)

However…. The same VMQ issues that we witnessed previously have now arisen on the production VM Switch which is used to uplink the virtual machines on each host to the outside world.

The Production VM Switch is configured as follows:

• Same configuration as the original infrastructure: 4 Intel 1GbE i350 NICs, two of which are in one physical quad port NIC, whilst the other two are in an identical NIC, directly below it. The remaining 2 ports from each card function as iSCSI MPIO interfaces to the SAN. We did this to try and achieve NIC level fault tolerance. The latest Firmware and Drivers have been installed for all hardware (including the NICs) fresh from the latest Dell Server Updates DVD (V14.10).

• In each host, the above 4 VM Switch NICs are formed into a Switch independent, Dynamic team (Sum of Queues mode), each physical NIC hasRSS disabled and VMQ enabled and the Team Multiplex adapter also has RSS disabled an VMQ enabled. Secondly, each NIC is configured to use a single processor core for VMQ. As this is a Sum of Queues team, cores do not overlap and as the host processors have Hyper Threading enabled, only cores (not logical execution units) are assigned to RSS or VMQ. The configuration of the VM Switch NICs looks as follows when running Get-NetAdapterVMQ on the hosts:

Name                           InterfaceDescription              Enabled BaseVmqProcessor MaxProcessors NumberOfReceive
                                                                                                        Queues
----                           --------------------              ------- ---------------- ------------- ---------------
VM_SWITCH_ETH01                Intel(R) Gigabit 4P I350-t A...#8 True    0:10             1             7
VM_SWITCH_ETH03                Intel(R) Gigabit 4P I350-t A...#7 True    0:14             1             7
VM_SWITCH_ETH02                Intel(R) Gigabit 4P I350-t Ada... True    0:12             1             7
VM_SWITCH_ETH04                Intel(R) Gigabit 4P I350-t A...#2 True    0:16             1             7
Production VM Switch           Microsoft Network Adapter Mult... True    0:0                            28

Load is hardly an issue on these NICs and a single core seems to have sufficed in the old design, so this was carried forward into the new.

The loss of connectivity / high latency (200 – 400 mSec as before) only seems to arise when a VM is moved via Live Migration from host to host. If I setup a constant ping to a test candidate VM and move it to another host, I get about 5 dropped pings at the point where the remaining memory pages / CPU state are transferred, followed by an dramatic increase in latency once the VM is up and running on the destination host. It seems as though the destination host is struggling to allocate the VM NIC to a queue. I can then move the VM back and forth between hosts and the problem may or may not occur again. It is very intermittent. There is always a lengthy pause in VM network connectivity during the live migration process however, longer than I have seen in the past (usually only a ping or two are lost, however we are now seeing 5 or more before VM Nework connectivity is restored on the destination host, this being enough to cause a disruption to the workload).

If we disable VMQ entirely on the VM NICs and VM Switch Team Multiplex adapter on one of the hosts as a test, things behave as expected. A migration completes within the time of a standard TCP timeout.

VMQ looks to be working, as if I run Get-NetAdapterVMQQueue on one of the hosts, I can see that Queues are being allocated to VM NICs accordingly. I can also see that VM NICs are appearing in Hyper-V manager with “VMQ Active”.

It goes without saying that we really don’t want to disable VMQ, however given the nature of our clients business, we really cannot afford for these issues to crop up. If I can’t find a resolution here, I will be left with no choice as ironically, we see less issues with VMQ disabled compared to it being enabled.

I hope this is enough information to go on and if you need any more, please do let me know. Any help here would be most appreciated.

I have gone over the configuration again and again and everything appears to have been configured correctly, however I am struggling with this one.

Many thanks

Matt

VMMS Account does not have sufficient privilege to open attachment ".....\SRVVirt.vhd"

I am logged in as administrator, Services ara running with following accounts :

Hyper-V Image Management Service - Network Service

Hyper-V Networking Management Service - Local System

Hyper-V Virtual machine managemet - Local System

Operating system is clear installation of Windows Server Standard 2008 RTM with Hyper-V

Installed roles :

Application server

File Services

Hyper-V

Web Server

any suggestions on how to overcome this issue and get the hypervisor running would be greatly appreciated.

thanks.

Radek

.

Hi All,

I would like to ask you about a Hyper-V cluster networking best practice regarding the IP addressing and backup VLAN. The storage has SAS controllers so it’s not mentioned below. I have read through some blogs and I thought I will create my 2 node cluster this way:

Physical DC:

IP address: 10.0.1.5/24

Virtual DC:

IP address: 10.0.1.6/24

HOST1:

Management & Cluster communication: 10.0.1.11/24, DNS: 10.0.1.5, 10.0.1.6

Live Migration & Cluster communication: 10.0.2.11/24

Backup Network: 10.0.3.11/24

HOST2:

Management & Cluster communication: 10.0.1.12/24, DNS: 10.0.1.5, 10.0.1.6

Live Migration & Cluster communication: 10.0.2.12/24

Backup Network: 10.0.3.12/24

CLUSTER RESOURCE:

IP address: 10.0.1.10/24

BACKUP SERVER:

IP address: 10.0.3.20/24, DNS: 10.0.1.5, 10.0.1.6

So is it OK if I only have one Cluster Resource IP address in subnet 10.0.1.0/24? Or do I need a Cluster Resource IP in the Backup Network subnet too? I wonder if I use only the 10.0.1.10/24 IP address for the cluster, then this subnet will be used for backup traffic instead of 10.0.3.0/24.

Thanks,

Dvijne

We have a Windows 7 as VM on windows 2012 R2 host. This morning we try to install 25 Windows update. Now, the VM can't boot properly with this message: "Failed configuring Windows updates. Reverting changes. Do not turn off your computer". It has been over 5 hours. What should we do?

Bob Lin, MCSE & CNE Networking, Internet, Routing, VPN Networking, Internet, Routing, VPN Troubleshooting on http://www.ChicagoTech.net How to Install and Configure Windows, VMware, Virtualization and Cisco on http://www.HowToNetworking.com

Hi,

have problems with one of VMs on 2012R2 host

Host is HP DL160 Gen9 with E5-2620v3 (12cores), P440 storage controller, 2xSAS disks.

VM1 boots normally with 8 vcpu (but integration services doesn't report status - see below), however with 9-12vcpu system boots in a weird mode. The first blue screen is not blue, but light blue/magenta. I can type password to log in, but nothing happens when i click ok/enter - no failures and no result. Sometimes (another boot) password i type is typed with Chinese instead on English. In any case you cannot login. sometimes it says "a device attached to the system is not functioning"

host is: Windows Server 2012R2 (fully updated)

VM1 (the problematic one) is 2012R2 (fully updated), Integration services state/version not detected

State                      : Running
MemoryStatus               : OK
Status                     : Operating normally
Generation                 : 2
IntegrationServicesState   :
IntegrationServicesVersion :

VM2 (good) is 2012R2

State                      : Running
MemoryStatus               : OK
Status                     : Operating normally
Generation                 : 2
IntegrationServicesState   : Up to date
IntegrationServicesVersion : 6.3.9600.16384

So host is generally ok, but some issues with VM1. when i try to install Integration tools on VM1 i got:

---------------------------
Hyper-V Integration Services
---------------------------
This computer is already running the current version of integration services  (version 6.3.9600.16384).

get-service on VM1:

PS C:\> Get-Service vmi*

Status   Name               DisplayName
------   ----               -----------
Stopped  vmicguestinterface Hyper-V Guest Service Interface
Running  vmicheartbeat      Hyper-V Heartbeat Service
Stopped  vmickvpexchange    Hyper-V Data Exchange Service
Running  vmicrdv            Hyper-V Remote Desktop Virtualizati...
Running  vmicshutdown       Hyper-V Guest Shutdown Service
Running  vmictimesync       Hyper-V Time Synchronization Service
Stopped  vmicvss            Hyper-V Volume Shadow Copy Requestor


get-service on VM2:

PS C:\> Get-Service vmi*

Status   Name               DisplayName
------   ----               -----------
Stopped  vmicguestinterface Hyper-V Guest Service Interface
Running  vmicheartbeat      Hyper-V Heartbeat Service
Runningvmickvpexchange    Hyper-V Data Exchange Service
Running  vmicrdv            Hyper-V Remote Desktop Virtualizati...
Running  vmicshutdown       Hyper-V Guest Shutdown Service
Running  vmictimesync       Hyper-V Time Synchronization Service
Running vmicvss            Hyper-V Volume Shadow Copy Requestor

sfc /scannow doesn't find any issues

I have a feeling it's related to some bad integration driver. maybe some driver have replaced. Or maybe it's a service, that starts faster with more vcpu and fails. Not sure what else to do

Hi

I'm thinking of setting up Hyper-V (failover cluster) and since this is the first time I'll be venturing into this area, I'd like to clarify some points about it.

Can I set up Hyper-V on a different machine that my DC?

My DC has 4 separate volumes:

1 log

2 sysvol

3 database folder

4 shares

In case Hyper-v can be installed on a different machine that my DC, am I supposed to partition the hard drive of the machine where Hyper-V will be set up, or the virtualization process takes care of that?

Thank you for your help.

Hi guys,

I  have just installed  Hyper-v 2012 R2 and trying to enable GPU pass-through support for VM.

And I noticed that there is a RemoteFX GPU on the hyper-v server setting page.

RemoteFX GPU is not same as GPU pass-through(directly pass the whole GPU to vm), am i right?

thanks for your help

Hi!

Do we have any option to use a GPU with a Server Operating System installed as a virtual machine?

Thanks.

We have need to present the guest VM with removable media as can be done with VMWare during the boot up process. 

How can we present a Hyper-V guest VM with media that is recognized as removable media during the boot up process?

Cloud Production

It's easy to create a guest OS by defining a new VM, mounting an ISO to boot from, and starting it up for an attended install. However, my goal is to present an autoUnattend.xml file to the installing OS on removable media. This is a standard practice when installing to physical HW.  While there are many ways to make the contents of a flash drive available to a VM, none of them are recognized as removable media, and therefore the normal installation process doesn't look externally for an XML file. Alternatively, I could create a custom image with the file embedded in the \Sources folder, but that process requires way too much time for what is needed here. It should be possible to effectively, drop in the installation DVD, plug in the removable media, and boot up.

I've tried creating a virtual floppy and mounting that in the VM, but so far it hasn't worked.

How can I install a guest OS using the equivalent of dropping in a DVD and plugging in removable media that contains my XML file?

thanks for your help. 

Cloud Production

Server 2012 R2 with hyper-v host on server core.  In hyper-v manager:

I login to the hyper-v host and dir the C drive and have 213,557,248 bytes free! Yes, the VM's are located on the C drive. This host has been running fine with the same amount of VM's for months! It looks like something took up all the space. How do i find out what is hogging the space?

Hi guys, 

Server OS: Windows Server 2012 R2

I have a VM Gen2 with following settings: 

I can boot fine from the same .iso from a Gen1 VM so the .iso is bootable. 

I know that the Gen2 VM uses SCSI instead of IDE, but I have no clue regarding how to boot from the .iso..

Any ideas?

Thanks in advance!

Best regards,

Kenny

Hello, I just upgraded my machine to Windows 10 and now my Hyper-V Manager cannot connect to and manage our Server 2012R2 Hyper-V Hosts. I get the following error.

"An error occurred while attempting to connect to server xxxxx. Check that the Virtual Machine Management service is running and that you are authorized to connect to the server."

I've read in multiple places that Hyper-V Manager in Windows 10 and Server 10 will support managing Server 2012 and 2012R2 hosts. I'm not sure why it's not working though.

Any help would be appreciated.

 Dear 

   i have dual processor in my server Intel® Xeon® E5-2650 with 10 physical cores if i install hyper V 2012 R2 server 

Q1 is the maximum virtual cores is quad of physical cores? 

Q2 can i make two machines each one with 20 virtual cores? 

Q3 can i make one machine with 35 cores?

Regrads

I have a server 2012 Standard R2, that was initially going to be a virtual machine. Hardware restraints caused us to go physical, but the network adapter we are using is the vEthernet Virtual switch. The NIC 1 is still enabled, but the only item checked is the virtual switch.

What I need to know, is can I just insert the IP information in the NIC1 and delete the vEthernet virtual switch.  This is a production machine and I do not want to mess it up and cause other issues. 

The reason I want to do this, is every two or three weeks, the clients lose connection and the system has to be rebooted to recover.

Steve

As part of an application to detach a vhd from a VM, I am trying to execute the "RemoveResourceSettings" method ofMsvm_VirtualSystemManagementService class(https://msdn.microsoft.com/en-us/library/hh850278(v=vs.85).aspx).

Below if the mof of the method:

uint32 RemoveResourceSettings(
  [in]  CIM_ResourceAllocationSettingData REF ResourceSettings[],
  [out] CIM_ConcreteJob                   REF Job
);

    // Use the IWbemServices pointer(pSvc) to make requests in WMI namespace \ROOT\\VIRTUALIZATION\V2---

	BSTR VirtualSystemManagementService_ClassName = SysAllocString(L"Msvm_VirtualSystemManagementService");
	BSTR RemoveResourceSettings_MethodName = SysAllocString(L"RemoveResourceSettings");

	//Get resource allocation settings of the disk to be removed
	IEnumWbemClassObject* pEnumerator = NULL;
    hres = pSvc->ExecQuery(
        bstr_t("WQL"),
		bstr_t("select * from Msvm_ResourceAllocationSettingData where InstanceID like 'Microsoft:A87CEE11-889C-44E0-8A41-44452ABD0BEE%83F8638B-8DCA-4152-9EDA-2CA8B33039B4%1%1%D'"),
        WBEM_FLAG_FORWARD_ONLY | WBEM_FLAG_RETURN_IMMEDIATELY,
        NULL,&pEnumerator);

	IWbemClassObject *resourceAllocationSettingInstance;
	ULONG uReturn = 0;

	hres = pEnumerator->Next(WBEM_INFINITE, 1,&resourceAllocationSettingInstance, &uReturn);

	VARIANT vtProp;
	hres = resourceAllocationSettingInstance->Get(L"InstanceID", 0, &vtProp, 0, 0);
	wcout << "InstanceID(The disk drive to be removed) : " << vtProp.bstrVal << endl;

	hres = resourceAllocationSettingInstance->Get(L"__PATH", 0, &vtProp, 0, 0);
	wcout << "__PATH(The disk drive to be removed) : " << vtProp.bstrVal << endl;

	BSTR resourceAllocationSettingInstanceString= vtProp.bstrVal;

	IWbemClassObject* VirtualSystemManagementService_Class = NULL;
    hres = pSvc->GetObject(VirtualSystemManagementService_ClassName, 0, NULL, &VirtualSystemManagementService_Class, NULL);

	IWbemClassObject* pInParamsDefinition = NULL;
	IWbemClassObject* pOutParams = NULL;
	hres = VirtualSystemManagementService_Class->GetMethod(RemoveResourceSettings_MethodName, 0,&pInParamsDefinition, NULL);

	cout << "After GetMethod\n";

	IWbemClassObject* VirtualSystemManagementServiceClassInstance = NULL;
	hres = pInParamsDefinition->SpawnInstance(0, &VirtualSystemManagementServiceClassInstance);

	//Get a pointer to the "resourceAllocationSettingInstance" which should be detached or removed from the corresponding VM configuration
	void **ppvObject;
	const IID riid = IID_IUnknown;

	hres = resourceAllocationSettingInstance->QueryInterface(riid, ppvObject);
	if (FAILED(hres))
    {
        cout << "Failed to QueryInterface = 0x"<< hex << hres << endl;
        return 1;                  // Program has failed.
    }
	VARIANT object_var;
	VariantInit(&object_var);
	object_var.vt = VT_UNKNOWN;
	object_var.punkVal = (IUnknown*)*ppvObject;


	CComSafeArray<VARIANT> resourceSettingsInputSafeArray;
	resourceSettingsInputSafeArray.Add(object_var);

	VARIANT resourceAllocationSettingVariant;
	VariantInit(&resourceAllocationSettingVariant);
	resourceAllocationSettingVariant.parray = resourceSettingsInputSafeArray.Detach();
	resourceAllocationSettingVariant.vt = VT_ARRAY | VT_VARIANT ;

	VirtualSystemManagementServiceClassInstance->Put(L"ResourceSettings", 0, &resourceAllocationSettingVariant, NULL);

	if (FAILED(hres))
    {
        cout << "Failed to set input parameter. Error code = 0x"<< hex << hres << endl;
		return 1;                  // Program has failed.
	}

	VARIANT vtPath;
	VariantInit(&vtPath);
	if(FAILED(VirtualSystemManagementService_Class->Get(L"__Path", 0, &vtPath, NULL, NULL)))
	{
		cout << "Object has no __Path!" << endl;
		VirtualSystemManagementService_Class->Release();
	}
	else
	{
		wcout << "Object path on which method should be executed: " << vtPath.bstrVal <<endl;
	}

	hres = pSvc->ExecMethod(vtPath.bstrVal, RemoveResourceSettings_MethodName, 0,
    NULL, VirtualSystemManagementServiceClassInstance, &pOutParams, NULL);

	if (FAILED(hres))
    {
        cout << "Could not execute method. Error code = 0x"<< hex << hres << endl;
        return 1;               // Program has failed.
    }

Good day all!

This is my first post. So be nice :p. All day long the law of murphy is bugging me :(. Every thing goes wrong. Hope you can help me...

When I am trying to make a new Hard Disk in Hyper-V I get the following message:

The server encountered an error trying to create the virtual hard disk

and error code:

The requested operation could not be completed due to a file system limitation

We have a SAN sollution with two fysical servers (hyperv-01 and hyperv-02) where we installed 2008 r2 standard with Hyper-V, both servers are working in a failover cluster. We have a third fysical server (SCVMM01) where we use the failover cluster tool to manage the virtual machines. We use to work with SCVMM but that did not work so well. We have enough space to make new VHD's but with nog result. I have try to make the VHD's on SVMM01, hyperv-01 and hyperv-02 but the message stays. I have try the fixed size and dynamically modus. I have try 100 gb, 80 gb, 10 gb, 5 gb and 2 gb disk size's. Still the problems stays.

With the SAN is nothing wrong. No errors all LUN's are up. We have I/0 traffic.

Anyone have any idea what may be going on, or know of a workaround?