InstantClone fails due to exceeding maximum number of delta disks

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

InstantClone operation on Virtual Machine(SourceVM) which succeeded during previous invocations starts failing with error similar to:

(vmodl.fault.SystemError) {
faultCause = (vim.fault.GenericVmConfigFault) {
faultCause = (vmodl.MethodFault) null,
faultMessage = (vmodl.LocalizableMessage) [
(vmodl.LocalizableMessage) {
key = "msg.migrate.prepare.failed",
arg = <unset>,
message = "Unable to prepare migration."
},
(vmodl.LocalizableMessage) {
...
reason = "Unable to prepare migration."
msg = "Unable to prepare migration."
},
...
reason = "Launch failure"

In the SourceVM, you see warning alert similar to:

This virtual machine has more than 100 redo logs in a single branch of its snapshot tree. Deleting some of the snapshots or consolidating the redo logs will improve performance.

Environment

VMware vCenter Server 6.7.x
VMware vCenter Server Appliance 6.5.x

Cause

This issue is caused due to a limitation of vSphere's delta disks (redo logs). The maximum number of redo log levels supported for a virtual disk is 255. When InstantClone is invoked on a SourceVM, a new delta disk is created for both the SourceVM and the GeneratedVM. These delta disks are not tied to snapshots, so they are not visible via the vSphere UI. At the moment, the only way to examine the presence of these delta disks is via the VM's FileLayoutEx API call or via examining the vmfolder of the SourceVM.
If numerous InstantClone operations are invoked against the same SourceVM, the delta disk chain will increase with each invocation. Once this number reaches 255, InstantClone operation will start failing.

Resolution

To resolve this issue:

Freeze the SourceVM:

The InstantCloneFreeze functionality can be used on the SourceVM to limit the number of new redo log delta disks created for the SourceVM to 1, regardless of the number of InstantClone operations invoked on the SourceVM. See Freezing the Source Virtual Machine for an Instant Clone Operation.

Note: Once a SourceVM is frozen, the VM will still be shown as PoweredOn and running, even though guest instructions are not running. There are no indicators on the vSphere UI to indicate a VM is instantCloneFrozen. The VIM API boolean Vim.VirtualMachine.runtime.instantCloneFrozen can be used to check if a SourceVM is instantCloneForzen. The only way to exit the instantCloneFrozen mode is to powerOff or reset the VM.

Cleanup existing redo log delta disks from SourceVM:

If a Virtual Machine is observed to have a large number of redo log delta disks, invoke the Vim.VirtualMachine.PromoteDisk(unlink=True), which will result in the VM from having a single flat vmdk file.

Clone with memory snapshot as an alternative to InstantClone:

If InstantCloneFreezing is not an option (eg. OS does not have vmtools available), then the Clone with memory snapshots may be used as an alternative way to achieve a very similar end result as InstantClone, with the downside that it will be slower and less memory efficient due to lack of memory page sharing.

By default, the Clone operation will attempt do a full disk copy from the SourceVM to the ClonedVM. To achieve better disk efficiency as well as Clone time, ensure there exists at least one snapshot on the SourceVM, and Clone with diskMoveType=moveChildMostDiskBacking, which will only perform a copy of the SourceVM's latest snapshot rather than the entire disk content.