Hi all,
I have 2 Hyper-V Servers, one running Server Core 2016 and one running Server 2019 Datacenter w/ GUI.
On the 2019 server, all 5 of the VMs are Server 2019 Datacenter w/ GUI. I back them all up with Task Scheduler running Powershell scripts to a network share on a dedicated backup server (running 2019 Datacenter w/ GUI). They all work perfectly,
with a variety of schedules and other options including different destinations, etc. NO problems.
On the 2016 server, there are 10 VMs, 8 of which run Server 2016 Datacenter w/ GUI or Server 2019 Datacenter w/ GUI. All 8 of these work beautifully with the powershell scripts with wbadmin and task scheduler. (Same idea as the 2019 server.)
But on that 2016 host server, there are 2 VMs running Server 2012 R2 Standard w/ GUI. They are not domain controllers, exchange servers, or have any other SQL based applications. One hosts an encrypted file cabinet program and the other hosts
nothing more than an old version of Quickbooks 2008 as a Remote Desktop web application. Both of them act the same way with the exact same wbadmin powershell command as the working ones. They both do the same thing whether it's run as a scheduled
task or as a direct elevated Command Prompt command or an elevated Powershell command. They both do the exact same thing whether the command is run as a "Domain Admin" or the dedicated "Backup User" that is used to initiate all the
other scheduled tasks. They both act the same with different target servers and different shares. They both act the same whether they are online or offline.
Here is a sample powershell command like what I'm using (it works for all my 2016 and 2019 VMs):
wbadmin start backup -backupTarget:\\BackupServer\BackupShare -hyperv:"ServerName" -vssfull -quiet
(I know the -vssfull is redundant when going to a network share but I don't want to have to remember to change this part if I temporarily send it to a local destination.)
What happens when it runs on the 2012 R2 servers:
It does the usual "Creating a shadow copy of the volumes specified for backup..." for a few minutes. Then it says "Enumerating files for <ServerName>(Online)... 10 found" (ALL the ones that work have exactly 18 files, and
never say this 'enumerating files verbiage... they just move straight to 'Starting application backup'). Then it says "Copying files for <ServerName>(Online)..." on one line only (for just a few seconds) with a percentage that will be
somewhere randomly spaced between 0% and 25%. Then it says "The backup operation successfully completed. Backup of <ServerName>(Online) succeeded. Application backup succeeded. Log of files successfully backed up: <path
to log file>. (Examples of log files from successful and unsuccessful attempts below, but the TL;DR version is that the backup completes successfully but just plain doesn't even try to backup the actual Virtual Hard Disks, like they aren't even there.)
But of course, the "successful" backup set is always roughly 150-250 MB and doesn't include ANY data at all. You know how if you dig into the folders you see the virtual hard drive file that is essentially the size of the data on the server VHD.
Then if you mount that and look inside it, you see the actual backup of the VHD? Well... what I'm left with is a 150-250 MB VHD that if I mount it and look inside it, there is no virtual hard drive with the data. It's just not there.
Some notes: Both 2012 R2 servers are completely up to date and working perfectly otherwise. Both are generation 2 and have SCSI controllers (not IDE). One of these VMs is 670 GB and the other is only 21 GB, so size doesn't seem to matter
(also, I have another VM that's 2 TB on that same server and it has no problem, so having enough free resources isn't the issue.) The event viewer doesn't show any errors at all, because, like it said above, it thinks everything succeeded perfectly.
I've googled my little heart out and come up with dozens of postings and forums talking about generic fixes for wbadmin problems, and while I can't list everything I've tried here, it's been quite a bit. I just haven't been able to find something specific
enough to 2012 R2 on a 2016 host to get me where I'm going... which makes me think this is not a typical problem people face. Somebody out there has to have seen this right? I feel like it's going to be an easy answer like, 'you can't do that, stupid'
or 'check this box, silly' but I haven't found it. Also, for various reasons, upgrading these servers to newer operating systems is out of the question for the next year or two.
Successful Log File Example (from one of the working 2016+ VMs):
Backed up V:\
Backed up V:\Snapshots\
Backed up V:\Snapshots\397F9C6A-9CFE-44D0-8D4F-5A17D4175090.vmcx
Backed up V:\Snapshots\397F9C6A-9CFE-44D0-8D4F-5A17D4175090.VMRS
Backed up V:\Virtual Hard Disks\
Backed up V:\Virtual Hard Disks\SLC-AD-02_Disk_2-AutoRecovery.avhdx
Backed up V:\Virtual Hard Disks\SLC-AD-02_Disk_2.vhdx
Backed up V:\Virtual Hard Disks\slc_ad_02_disk_1-AutoRecovery.avhdx
Backed up V:\Virtual Hard Disks\SLC_AD_02_Disk_1.vhdx
Backed up V:\Virtual Machines\
Backed up V:\Virtual Machines\5FA4A6CE-38C8-4977-ADFE-0EB1AECB5F96.VMRS
Backed up V:\Virtual Machines\5FA4A6CE-38C8-4977-ADFE-0EB1AECB5F96.vmcx
Application backup
Writer Id: {66841CD4-6DED-4F4B-8F17-FD23F8DDC3DE}
Component: 5FA4A6CE-38C8-4977-ADFE-0EB1AECB5F96
Caption : Online\SLC-AD-02
Logical Path:
*-----------------------------*
Log file from a 2012 R2 attempt (that obviously didn't work, you'll notice the lines where the VHDs should be listed between 'Virtual Hard Disks' and 'Virtual Machines', is nothing more than a carriage return:
Backed up V:\
Backed up V:\Snapshots\
Backed up V:\Snapshots\426E2E7F-CA17-4C36-A744-A2FEE4A21C25.vmcx
Backed up V:\Snapshots\426E2E7F-CA17-4C36-A744-A2FEE4A21C25.VMRS
Backed up V:\Virtual Hard Disks\
Backed up V:\Virtual Machines\
Backed up V:\Virtual Machines\99C9F76D-5BC9-4CEF-96E7-450123D4DE66.VMRS
Backed up V:\Virtual Machines\99C9F76D-5BC9-4CEF-96E7-450123D4DE66.vmcx
Application backup
Writer Id: {66841CD4-6DED-4F4B-8F17-FD23F8DDC3DE}
Component: 99C9F76D-5BC9-4CEF-96E7-450123D4DE66
Caption : Online\SLC-RDAPP-03
Logical Path:
*-----------------------------*
EVEN MORE DETAILS: I went and tried including every combination of -include:c: -systemState -allCritical, etc. but all of those switches (of course) backup the specified resource for the hyperv host server, not the VM, and it works perfectly, but it's
obviously not backing up what I want. It looks like it's just plain failing to find the VHDs on the 2012 R2 VMs so it just backs up the VM component without the disks and then gives itself a pat on the back for job well done and heads off to its favorite
pub for a mid-strength beer.
The even crazier thing: IT USED TO WORK! I have VM backups of these machines from 5 days ago from that same server to a different destination (before we upgraded the backup destination server) that worked perfectly. So, just to prove the
theory that the destination was somehow the problem, I hooked up the old backup server and tried to send it back to the destination where it worked successfully 5 days ago... and it does the same thing now... a backup with no virtual hard disks.
So to sum up: It worked for all machines. I added a completely different destination server and replaced the scripts with a version with a few extra lines of house-keeping for moving old backups out of the way before the new one runs, and changed
the destination share. It works for all 2016 and 2019 servers, never worked right for either 2012 R2 VMs after the new scripts and destination. I tried sending it back to the old destination with the old script... and now that doesn't work either.
Tried rebooting 2012 VMs and the host server, no change.
WHAT AM I MISSING???
Any help is appreciated. Thanks in advance.