A virtual machine fails to respond when using backup software and VMware vSphere Replication

Products

VMware

Issue/Introduction

Symptoms:

During a snapshot consolidation, the virtual machine does not respond.
The ESXi host becomes disconnected in vCenter Server.
This issue occurs when you attempt to consolidate a snapshot on a virtual machine which has 255 snapshots created by vSphere Replication.

Note: The maximum supported number of snapshots is 32.
This issue occurs when vSphere Replication uses Volume Shadow Copy (VSS) to quiesce the snapshots.
In vCenter Server, you see the error:

hierarchy is too deep

The hostd.log file located at /var/log/ on the ESXi host contains entries similar to:

T02:22:35.984Z [50B87B90 error 'vm:/vmfs/volumes/4eae978d-008965a8-83fb-bc305be7c11e/APP120vmflt/APP120vmflt.vmx'] Invalid transition requested (VM_STATE_REMOVE_SNAPSHOT -> VM_STATE_CONSOLIDATE_ALL_DISKS): Tasks running

T02:22:35.984Z [50B87B90 info 'vm:/vmfs/volumes/4eae978d-008965a8-83fb-bc305be7c11e/APP120vmflt/APP120vmflt.vmx'] Consolidate Disks failed: N3Vim5Fault14TaskInProgress9ExceptionE(vim.fault.TaskInProgress)

T02:22:35.985Z [50B87B90 warning 'vm:/vmfs/volumes/4eae978d-008965a8-83fb-bc305be7c11e/APP120vmflt/APP120vmflt.vmx'] Failed operation

T02:22:35.985Z [50B87B90 info 'TaskManager'] Task Completed : haTask-1-vim.VirtualMachine.consolidateDisks-459720263 Status error

T02:22:35.985Z [50B87B90 info 'Default'] AdapterServer caught exception: vim.fault.TaskInProgress

T02:22:36.085Z [51401B90 warning 'Hbrsvc'] Remove snapshot for vm (vmId=1) failed with fault vim.fault.TaskInProgress
Events tab of the virtual machine displays entries similar to:

'VMname'in group 'Business Critical Protection Groups for SRM':Cannot resolve the file locations of the production VM for replication. error 6/9/2014 3:22:36PM

Cause

In vSphere 5.1, Microsoft Volume Shadow Copy (VSS) was introduced to replicate the quiesced state of the operating system or application. If quiescing is enabled for virtual machines running Windows 2003, 2008 or 2012, vSphere Replication creates and removes the virtual machine snapshot to create an application quiesced replica. If vSphere Replication and the backup software try to create or remove a snapshot at the same time, one process fails.

The virtual machine may have a backup snapshot. Whenever vSphere Replication or Host Based Replication (HBR) try to consolidate the disks, it does not add a new redo log to the disks. Later, the backup snapshot is removed and then, whenever HBR tries to consolidate the disks, it adds a new redo log to the disks.

This issue occurs because there is a parent disk open and HBR tries to consolidate the disks during replication.

Resolution

This is not a VMware issue. However, VMware has made code changes in ESXi 5.1 Update 2 to the improve the existing consolidation issue. For more information, see the VMware ESXi 5.1 Update 2 Release Note. You can download the latest release of ESXi 5.1 from the VMware Download Center.

There is a known interoperability issue between VMware vSphere Replication 5.1/5.5 and snapshot-based backup products. When a virtual machine is protected by vSphere Replication and it is also part of a snapshot-based backup job, snapshot conflicts can occur.

To avoid snapshot conflicts and related consolidation issues, perform one of these two options:

Ensure that vSphere Replication protected virtual machines which are also part of a backup schedule do not use quiescing, so that a snapshot will not be taken.

Or
Use a backup solution that is not snapshot-based.

Additional Information

For related information, see:

To be alerted when this document is updated, click the Subscribe to Article link in the Actions boxTroubleshooting Volume Shadow Copy (VSS) quiesce related issues
VSS Quiescing Support with vSphere Replication
バックアップソフトウェアおよび VMware vSphere Replication の使用時に仮想マシンが応答しない