vSphere 5.1 is out! Among the enhancements is the new VMware Data Protection appliance that provides basic VM backups, deduplication, and file-level restore capabilities. It plugs directly into the vSphere Web Client interface and was very easy to setup. One re-occuring problem I have with VMs running Windows Server 2008 R2 is getting an application-consistent backup. I used vRanger Pro 5.5 before VDP came out and it had problems with this also, so I’m going just come right out and point the finger at VMware since the same issue is now plaguing me in their product also. The VMware Tools VSS API/Driver/whatever-you-want-to-call-it simply does not work with Windows Server 2008 R2.
After I upgraded my datacenter to vCenter 5.1 and ESXi 5.1 all of my backup jobs quit in vRanger Pro 5.5. Upgrading has always broken this product so I simply quit using it and installed the VDP appliance (the 2TB version FYI) figuring that it would solve my issue (and save me money in maintenance contracts). Not so fast! I couldn’t get a single backup job to finish. vCenter would show a snapshot being made, the VDP job start, and right about 10% of the way through the VDP job, the snapshot would be removed and the VDP job would fail. After some investigation on the VMware forms, I learned that this was a known issue and VMware recommended shutting down each VM, editing advanced settings, and changing the setting for “disk.enableUUID” to “false”. THIS IS THE SETTING THAT ALLOWS APPLICATION-CONSISTENT BACKUPS TO FUNCTION. Are you kidding me VMware? Your backup API doesn’t work for Windows Server 2008 R2? How long has that OS been out? It would be safe to say that the majority of your customers use that OS and need to make application-consistent backups of their VMs. Get with the program!
So are we stuck with backups that may have corrupt data in them (think Exchange and/or Microsoft SQL)? I say no. There is hope! You can use a backup freeze script to stop I/O during the snapshot process and allow an application-consistent backup to be made. Here’s how:
- Shutdown your VM, edit the advanced settings and change “disk.enableUUID” to “false” per VMware’s instructions. http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2035736
- Power the VM back on.
- Download the Windows Server 2008 SDK and install it on your PC. http://www.microsoft.com/en-us/download/details.aspx?id=11310
- Included with the SDK is a program called vshadow. There is a version for x86 (vshadow_x86.exe) and x64 (vshadow_x64.exe) installs. Pick the appropriate one for your install and copy it to the C: drive of your VM. More info on vshadow here: http://msdn.microsoft.com/en-us/library/windows/desktop/bb530725(v=vs.85).aspx
- Navigate to your VMware Tools folder and create a folder called “backupscripts.d”
- Create a batch file named “freeze.bat”
- In the batch file, enter a single line “C:\vshadow_xxx.exe c:” (replace xxx with x86 or x64 for which version you are using). You will need to add each drive on which you want to stop I/O to the end of the line, ex: “C:\vshadow_x64.exe c: e: f:”.
- Create your VDP job and run it!
You’re done! You can view the Windows Event Log and watch for the VSS activity. On an Exchange Server you will even see the log entry for “Backup Complete” and it will truncate transaction logs. This is the only fool proof way of getting application-consistent backups that I have found. New versions come every year, but this has never been fixed. Hopefully VMware will get with it soon and take care of this nagging problem. Hope this helps someone somewhere!
Thank you very much for the solution to the problem of the VDP backups!
Greetings from Argentina!
Genius. Thank you for taking the time!
Greetings from California.
worked! thx a lot!
Terrific! The only challenge now is that this method causes the Exchange server to truncate its logs. That is a problem because we use different software to perform weekly fulls and then hourly incrementals. When this runs the subsequent incrementals fail because the logs truncate. I guess I could set the UUID back to false to let this work and then do a two step restore — first the VDP then if that is hosed for Exchange the full from the up to a week before then all the incrementals but we were hoping to use VDP to supplement this strategy and not have to go through all of that.
Any suggestions?
But for non-Exchange machines this has been a lifesaver!
Not sure how to attack this scenario. Personally on my Exchange server let VDP handles all of the backups. It truncates the logs, but I can restore the VM to pretty much any point in time based on my backup schedule. I can see where you may want incrementals for restoring individual messages or something like that that someone might delete, but it would be better to have an archiving solution for that. I guess I need to know why you need the logs to answer this question more fully, because personally I wouldn’t worry about keeping them once you get a full backup to restore to.
So I think I figured out how to do this — not the absolute best since it is not using the Exchange writer but because it is using the rest I was able to restore and did not appear to have any data or messaging loss (it had to replay some of the logs when starting but mounted okay).
When running the vshadow command you can use the -wx={Writer Name} to exclude it from executing.
So for this I ran on my MS Exchange 2010 box with C: as system, K: as databases, and L: as logs:
vshadow_x64.exe -wx=”Microsoft Exchange Writer” C: K: L:
And this appeared to accomplish exactly what I wanted — to use VDP to make the backups (still using VSS for the files including the logs and databases) but without using the Exchange writer which invokes the truncation.
Awesome solution to your problem! Thanks for posting back in case anyone else needs to do the same thing!
Thanks OP and JFZ, great info!
I just want to provide a some additional details that helped me with the solution.
1. The SDK that worked for me is v7.0 AKA “Windows SDK for Windows 7 and .NET 3.5 SP1” (3.Download the Windows 7 / .NET 3.5 SP1 SDK and install it on your PC. http://www.microsoft.com/en-us/download/details.aspx?id=3138). v6.1 did not include the -wx={Writer Name} option.
2. vshadow differentiates its x86 and x64 version based on folder name. This is the freeze.bat I used for my 4 mailboxes, OS, and log drive:
“C:\Program Files\Microsoft SDKs\Windows\v7.0\Bin\x64\vsstools\vshadow.exe” -wx=”Microsoft Exchange Writer” C: F: G: H: J: K:
3. VMTools will try to run any file in the “backupscripts.d” folder, so if you have an existing freeze.bat you need to copy/back it up and overwrite with your new freeze.bat. For me when I tried to rename the existing one created by/for Backup Exec to freeze.bat.bak, it generated a separate backup job which fails immediately and you can see it listed in vCenter.
4. After I set the UUID to false and replaced the freeze.bat, VDP could not launch the backup and the error in vCenter stated: “VDP: An attempt was made to backup a virtual machine which no longer exists in vSphere.”
As per VMware KB: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2050333
1.Log into the VDP appliance and open a command line.
2.Run this command to stop the MCS services:
root@vdp:~/#: dpnctl stop mcs
Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)
dpnctl: INFO: Shutting down MCS…
dpnctl: INFO: MCS shut down.
3.Run this command to start the MCS services:
root@vdp:~/#: dpnctl start mcs
Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)
dpnctl: INFO: Starting MCS…
dpnctl: INFO: To monitor progress, run in another window: tail -f /tmp/dpnctl-mcs-start-output-4140
dpnctl: INFO: MCS started.
You saved the day!
thx a lot
Thank you!!!!
Question – in looking at some older threads for VCB… is there a need to “thaw” (e.g. un-freeze) after the backup is complete?
Thanks,
Brett
I think in the past a “thaw” script was necessary, but now the Microsoft Volume Shadow Copy service handles the thawing action automatically. I’ve never had any issues with this method and I’ve been using it for the last couple of years.
What should I be looking for in the Event Viewer on a SQL server to confirm this is working?
You will see some events telling you that I/O was frozen while a snapshot was created. You will also see an event that states something like “A backup has been created”. Hope this helps
Here’s my solution. Volume Shadow Copy is included in Server 2008 R2 and Server 2012, and is installed as a service with manual activation. The problem with Server 2008 and 2012 and creating a backup is not that VSC needs to be added. The problem is that the VCS service that is included with VMWare Tools is incompatible. So to fix this without running scripts, making special settings, etc etc, just do this:
1. Uninstall VMWare Tools
2. Reboot
3. Login to the O/S
4. From the VM console (not in VCenter Server) click: VM > Guest > Install/Upgrade VMWare Tools
5. If Propted, select “Interactive Tools Upgrade”, otherwise, continue.
6. Open your CD ROM
7. Run the 64 bit setup
8. Select Custom Install
9. Next
10. Deselect “Volume Shadow Copy Services Support”
11. Proceed with the installation.
12. Back up works now.
13. ???
14. Profit
Thanks for your post. Does this method quiesce the file system before it makes the backup? Is it truncating Exchange logs when done on a server with Exchange? If I’m not mistaken, removing Volume Shadow Copy Support from VMware Tools prevents any file system quiescing from happening period which won’t give you application consistent backups for servers running things like SQL and Exchange, but it will let backups complete.
Great method! Is it possible to use diskshadow instead of vshadow since diskshadow is now included with Server 2008 R2 and vshadow isn’t?
I would imagine it is. I will have to give it a try and see.
just checking if you ever gave diskshadow a go? about to implement this method. Thanks so much for sharing.
Today the problem is right with other product.
http://blogs.vmware.com/vsphere/?p=7335
diskshadow scripts that work: ##freeze.bat##
del c:\windows\metadata\backup.cab
C:\Windows\System32\diskshadow -s c:\windows\shadowScript.txt
date /t >> c:\windows\metadata\callme.txt
time /t >> c:\windows\metadata\callme.txt
##shadowScript.txt##
delete shadows all
reset
set context persistent
add volume c: alias new1
add volume g: alias new2
set metadata c:\windows\metadata\backup.cab
create
Did you actually check that the backups are indeed 100% consistent by restoring Exchange/SQL/etc VMs?
The vshadow command you use by default creates nonpersistent shadow copies meant to be accessed from the script specified with the “-exec=” parameter. After the program ends the shadow copies are deleted (that’s why you don’t need to do it in the VMware tools thaw scripts) and all writers are thawed. I guess there is still a small chance that something happens in SQL/Exchange/other writers in the instant between the vssadmin tool exits and vmware begins creating the actual snaphots.
I mean the method is far more better than no quiescing at all but is it really bulletproof?
i have a 2008 R2 VM that SQL installed on it, the SQL guy perform a schedule backup to the database, if i set the disk.EnableUUID=False,
does this affect his SQL backup and performance.