Symptoms:

• Virtual machines with snapshots or delta disks stop responding or fail during a storage migration.
• When powering on a virtual machine, you see this error message:
  
  The parent virtual disk has been modified since the child was created.
   
   
• A virtual machine unexpectedly powers off and you see this error when you attempt to power it back on:
  
  A general system error occurred: internal error.
   
   
• Virtual machine log contains the entry:
  
  vcpu-0| Msg_Question: msg.hbacommon.outofspace reply=0
vcpu-0| Msg_Question:
vcpu-0| msg.hbacommon.outofspace There is no more space for the redo log of W2003Sbs1-000001.vmdk.
vcpu-0| You may be able to continue this session by freeing disk space on
vcpu-0| the relevant partition, and clicking Retry. Otherwise,
vcpu-0| click Abort to terminate this session.
   
   
• You are unable to remove snapshots for virtual machines on one or more datastores.

VMware ESXi 3.5.x Embedded
VMware ESX Server 3.0.x
VMware ESXi 4.1.x Installable
VMware ESX 4.0.x
VMware ESXi 4.0.x Installable
VMware ESXi 4.0.x Embedded
VMware ESX Server 3.5.x
VMware ESXi 4.1.x Embedded
VMware ESXi 3.5.x Installable
VMware ESX 4.1.x

Note: This article addresses ESX/ESXi 3.x and 4.x. For information on vSphere 5.x, see Consolidating snapshots in vSphere 5 (2003638).

• The base disk of the virtual machine has been modified after a virtual machine snapshot or delta disk has been created.
• You have run out of space on a datastore containing the snapshot or delta disk.
• The guest operating system suffered an exception or kernel panic when a virtual machine snapshot was taken.

For more details about snapshots and recreating missing virtual machine descriptor files, see: Cannot power on a virtual machine because the virtual disk cannot be opened (1004232).

Snapshot functionality has improved over VMware ESX product releases and updates. For additional information, see:

Overview

1. If 10GB of changes were completed on one snapshotted virtual machine, its delta disk file size will increase proportionally by 10GB.
    
2. Creating another snapshot causes the existing snapshot delta disk to retain its current size, however the next delta disk will begin to store all forthcoming modifications and writes. The base disk is still left unmodified since inception of the first snapshot.
3. If another 15GB of changes were performed by the guest operating system, a total of 25GB of snapshot delta has now been recorded over the respective virtual machine's two snapshot delta disk files.
    
4. For versions prior to VMware ESX 4.0 Update-2, the task of consolidating all snapshots (Remove All Snapshots task) causes unique changes stored only in the second snapshot delta disk to be copied upwards through the snapshot chain and into the first snapshot, or its "parent."
5. This effect is recursive for each preceding parent file. As a result, the first snapshot delta disk file will grow by up to 15GB, accommodating all new blocks. Any common changes stored in both snapshot levels does not require additional space, however.
    
6. The end result is a datastore requiring 40GB, or 25GB + 15GB.

Notes:

• Additional space is required if the virtual machine is running during a Remove All Snapshots task. The amount of space consumed is dependent the amount of snapshot delta information the virtual machine has to temporarily store while its consolidation process completes.
• If the same 10GB were changed in the second snapshot, and just 5GB of unique block changes occurred, then the first snapshot will increase by 5GB during the consolidation, not the complete 15GB.
• Opting to save virtual machine memory contents with each Create Snapshot task requires additional space from the VMFS datastore. The amount of space required is equivalent to the amount of memory assigned to the virtual machine. This requirement can apply multiple times over, if multiple snapshots including virtual machine memory contents are created.

VMware ESX 4.0 Update 2 or later

1. Power off the virtual machine.
2. Open the virtual machine's Snapshot Manager.
3. Remove one or more snapshots by click Delete or Delete All.
4. Allow time for the consolidation to complete. Larger snapshots and delta disks will take more time to complete.

VMware ESX 4.0 Update 1 or earlier

Solution Prerequisites

• Review the virtual machine's configuration and identify the locations and filenames of the attached virtual disks.
• Verify what specific delta virtual disk(s) the virtual machine is referencing. You can determine this either by reviewing the virtual machine hardware configuration, highlighting a virtual disk, and noting each disk's file name, or you can review the virtual machine's configuration ( .vmx) file.
• Also note the size of each virtual machine hard disk (specifically the base disk files). Destination clones will be of these determined sizes.
• Cloning of virtual disks may be required as part of your effort towards consolidation and reclaiming of space. Verify that you have enough space on one or more datastores to receive a consolidated clone of the virtual machine in entirety or in part (a virtual disk).
• Perform a sanity check of the virtual machine disk tree, verifying Content ID (CID) values between multiple snapshot delta disk files, and other virtual machine disk descriptor values. For more important information on this topic, see: Resolving "The parent virtual disk has been modified since the child was created" CID mismatch errors (1007969).
• Conduct a review of datastore space consumption and assess what datastores disk or virtual machine clones can be saved to, as part of a consolidation effort.

Solutions

Note: Third party backup software may fail to back up a virtual machine if there are orphaned delta or ctk files in the VM directory with an error about missing ctk.vmdk files. This occurs as the snapshot or change tracking files cannot be created because they are already in use. To work around this issue, rename or remove the orphaned .delta.vmdk or .ctk file(s).

For translated versions of this article, see:

• Portuguese: Como consolidar instantâneos (2032583)
• Spanish: Consolidando Imágenes de Disco Snapshots (1032486)

For other related information pertaining to resolving snapshot issues, see:

• Cannot power on a virtual machine because the virtual disk cannot be opened (1004232).
• Resolving "The parent virtual disk has been modified since the child was created" CID mismatch errors (1007969).
• Understanding virtual machine snapshots in VMware ESXi and ESX (1015180)

1. Remove the virtual machine hard disk from the virtual machine's configuration.
2. Complete your migration using the VMware vSphere or Infrastructure client's migrate functionality when connected to vCenter. Otherwise attempt this using the terminal, vMA, or vCLI with vmkfstools.
3. When the migration completes, re-add the virtual machine hard disk to the virtual machine's configuration.
4. Start the virtual machine.

Cleanup: Removing the original/old virtual machine disks via vMA and vCLI

Note: It is important that the virtual machine remains powered-on with its files locked throughout these steps. While powered on, deletion of in-use files (which are otherwise required) is not possible, in the event of a mistake being made anywhere in the deletion process. This procedure also guarantees an audit trail. Terminal access may not offer a complete command history.

1. Confirm the virtual machine is running.
2. List the virtual machine's datastore contents. Some or all of the snapshots should still be present.
    
   
   
   vifs --server -D '[Datastore Name] examplevm'
    
3. For the virtual disk that was successfully cloned, delete its original delta and base disk files:
    
   
   
   vmkfstools --server -U '[Datastore Name] examplevm/examplevm-000003.vmdk'
   vmkfstools --server -U '[Datastore Name] examplevm/examplevm-000002.vmdk'
   vmkfstools --server -U '[Datastore Name] examplevm/examplevm-000001.vmdk'
   vmkfstools --server -U '[Datastore Name] examplevm/examplevm.vmdk'
   
   This error is displayed if you attempt to remove disk files that are locked or in-use. This otherwise may be indicative of attempting to remove the wrong disk files:
   
   
    
4. Confirm that the disk files have been removed:
   
   vifs --server -D '[Datastore Name] examplevm'

 

Cleanup: Removing the base disk and related snapshot delta disks via the ESX host terminal

Note: See the section Confirming the Virtual Machine's Snapshot Delta Disk Layout Using the ESX Host Terminal below before proceeding with the provided steps.

1. Confirm that the respective delta disk files are not locked while this virtual machine is powered-on. They should not be, at this stage. Run this command:
   
   touch -a examplevm-??????-delta.vmdk
   
   Note: Running the command with "??????" would test all delta files in the directory. If you have multiple virtual hard disks, substitute "examplevm-??????" with applicable values to the virtual disk in question (previously cloned).
   
   If the command is successful to all files, no output is given and you are returned to the prompt. If there are failures due to locking, however:
   
   touch: cannot touch `examplevm-000001-delta.vmdk': Device or resource busy
   touch: cannot touch `examplevm-000002-delta.vmdk': Device or resource busy
    
2. Remove the snapshot delta and base disk extent/descriptor files using the vmkfstools command.
   
   vmkfstools -U examplevm-000003.vmdk
   vmkfstools -U examplevm-000002.vmdk
   vmkfstools -U examplevm-000001.vmdk
   vmkfstools -U examplevm.vmdk
   
   Note: The descriptor files are referenced, as opposed to the extents (*-delta.vmdk or *-flat.vmdk files)
   
   Again, any failures due to locked files will present an error:
   
   Failed to delete virtual disk: Device or resource busy (1048585).