What a mess! We recently purchased a new backup system at work and initially I thought things were going to go smoothly getting things setup and running. We went with an IBM server configured with 6TB of storage, VMware ESX, and Quest/Vizioncore vRanger Pro DPP and vReplicator for the backups. We’d be able to do nightly backups with vRanger and have the ability to do file-level restores quickly and easily, plus with vReplicator we’d be able to replicate our virtual machines to the server and run them there in an emergency since the server would be housed offsite. For the majority of the VMs we have, this system ran flawlessly as our vendor said it would. Unfortunately we had just upgraded our most critical systems to Windows Server 2008 R2 and this is where the issues began.
With the new system I wanted to be able to make application aware and consistent backups of all of my virtual machines. I followed all of the documentation on setting this up, but could never get snapshots to complete in vRanger for longer than a week. We had the schedule for our virtual machine backups set so that we’d get a full backup at least once every 7 days, with differential backups in between. The full backups would only complete if guest quiescing was disabled which would only provide me with a crash consistent backup of the target server. With applications like Microsoft Exchange and Microsoft SQL running on these servers, this is a big no-no. I/O needs to be stopped before snapshot creation to provide a reliable backup. After the full backup would complete with quiescing turned off, it could be re-enabled and the differential backups would complete successfully for the next 6 days and then fail again on the next full backup.
I started by contacting vRanger support and opening a ticket. I tried all of the KB articles on snapshot failures, but always got a “snapshot configuration mismatch. Please commit all open snapshots before continuing backup” error on the log for the backup jobs that failed. Quest eventually started pointing the finger at VMware claiming they had a bug in ESX 4.1 and that I would have to wait for a fix from them before it would work. I was also told that this would be addressed in vRanger 5. I wasn’t told a timeline on these relases I couldn’t sit around waititng for an update. After 2 weeks back and forth with support, Quest simply told me that they couldn’t help me now, but would keep the case open an notify me when the issue was fixed. I contacted my vendor to return the software and instead purchase Veeam after downloading and demo and testing that this worked fine in their software. My vendor’s rep with Quest won’t return their calls or e-mails about returning the software!
Through all of this, I recognized inconsistencies in vRanger documentation for instance how to properly install their VSS agent vzshadow.exe, which incidentally is just the vshadow.exe program that is included in the Microsoft Windows SDK renamed to vzshadow.exe. Documentation needs to be updated on a regular basis for puducts such as this. Businesses rely on these products to save them in a disaster and if the documentation doesn’t tell you how to properly set their own software up, get ready for data loss when you have a disaster.
I was invited to participate in the BETA 2 program for vRanger Backup & Replication 5 a few weeks ago and participated in the first conference kicking everything off today. I figured maybe I could have some input and either get my issue fixed or get to the right person to get my money back. I was able to pose the question about this issue in the discussion and was baffled to find out that these backups should already be working in the current version of 4.5. To help everyone avoid the headaches I went through getting this to work, I’m going to provide some simple instructions for getting things going. You’ll need to schedule downtime for this.
- Shutdown the Windows Server 2008 VM
- Edit the .vmx file associated with the VM you want to backup and remove the line ‘disk.EnableUUID = “TRUE”‘ completely from the file.
- Save the .vmx file and startup the VM – at this point, only file system consistent backups are possible according to what I have been told.
- Install the VSS agent provided with either the Windows SDK or vRanger according to the documentation that comes with vRanger
NOTE: The documentation is wrong about where to put the freeze.bat file. It should go in C:\Program Files\VMware\VMware Tools\backupscripts.d\ folder. I suggest using the copy of vshadow.exe that comes with the Windows SDK because it is a newer version than the one included with vRanger. Make sure you get the appropriate copy of vshadow.exe for x64 or x86 depending on what you are using (Windows Server 2008 R2 is available only in x64) - Enable guest quiescing in vRanger for your backup job.
- Perform a backup.
- Verify the Windows application log hasVSS activity. MS SQL and Exchange are very vocal about freezing and thawing of databses.
This process basically disables the VMware Tools VSS agent and replaces it with a different one that works with Windows Server 2008. Why vRanger support couldn’t tell me this in any of the calls I made, I have no idea, but this solved the problem for me. They would either have me do one half of this or the other but never both at the same time. The funny thing is that this is really in the end a problem with VMware Tools like they said. Hopefully it will be fixed when VMware issues an update for ESX 4.1, but there is still no reason for vRanger support to not know how to work around it, put me off for 3 weeks, have wrong documentation, or for the rep not to return calls to my vendor about returing the product. I’m pretty sure I’m not the only person with these conbinations of products having these problems.
Leave a Reply