Consolidating snapshots in ESX/ESXi 3.x and 4.x (1007849)
- Virtual machines with snapshots or delta disks stop responding or fail during a storage migration.
- When powering on a virtual machine, you see this error message:
The parent virtual disk has been modified since the child was created.
- A virtual machine unexpectedly powers off and you see this error when you attempt to power it back on:
A general system error occurred: internal error.
- Virtual machine log contains the entry:
vcpu-0| Msg_Question: msg.hbacommon.outofspace reply=0
vcpu-0| msg.hbacommon.outofspace There is no more space for the redo log of W2003Sbs1-000001.vmdk.
vcpu-0| You may be able to continue this session by freeing disk space on
vcpu-0| the relevant partition, and clicking Retry. Otherwise,
vcpu-0| click Abort to terminate this session.
- You are unable to remove snapshots for virtual machines on one or more datastores.
This error can occur if:
The base disk of the virtual machine has been modified after a virtual machine snapshot or delta disk has been created.
You have run out of space on a datastore containing the snapshot or delta disk.
The guest operating system suffered an exception or kernel panic when a virtual machine snapshot was taken.
For more details about snapshots and recreating missing virtual machine descriptor files, see: Cannot power on a virtual machine because the virtual disk cannot be opened (1004232).
Snapshot functionality has improved over VMware ESX product releases and updates. For additional information, see:
- VMware ESX 3.5, Patch ESX350-201006401-SG: Updates VMkernel, VMware Tools, hostd, VMX, VMnix (1022899)
- VMware ESXi 3.5, Patch ESXe350-201006401-I-SG: Updates firmware (1020052)
- VMware ESX 4.0, Patch ESX400-201006201-UG: Updates the VMware ESX 4.0 Core and CIM components (1017721)
- VMware ESXi 4.0, Patch ESXi400-201006201-UG: Updates Firmware (1017739)
- If 10GB of changes were completed on one snapshotted virtual machine, its delta disk file size will increase proportionally by 10GB.
- Creating another snapshot causes the existing snapshot delta disk to retain its current size, however the next delta disk will begin to store all forthcoming modifications and writes. The base disk is still left unmodified since inception of the first snapshot.
- If another 15GB of changes were performed by the guest operating system, a total of 25GB of snapshot delta has now been recorded over the respective virtual machine's two snapshot delta disk files.
- For versions prior to VMware ESX 4.0 Update-2, the task of consolidating all snapshots (Remove All Snapshots task) causes unique changes stored only in the second snapshot delta disk to be copied upwards through the snapshot chain and into the first snapshot, or its "parent."
- This effect is recursive for each preceding parent file. As a result, the first snapshot delta disk file will grow by up to 15GB, accommodating all new blocks. Any common changes stored in both snapshot levels does not require additional space, however.
- The end result is a datastore requiring 40GB, or 25GB + 15GB.
- Additional space is required if the virtual machine is running during a Remove All Snapshots task. The amount of space consumed is dependent the amount of snapshot delta information the virtual machine has to temporarily store while its consolidation process completes.
- If the same 10GB were changed in the second snapshot, and just 5GB of unique block changes occurred, then the first snapshot will increase by 5GB during the consolidation, not the complete 15GB.
- Opting to save virtual machine memory contents with each Create Snapshot task requires additional space from the VMFS datastore. The amount of space required is equivalent to the amount of memory assigned to the virtual machine. This requirement can apply multiple times over, if multiple snapshots including virtual machine memory contents are created.
VMware ESX 4.0 Update 2 or later
To consolidate the snapshots for a virtual machine currently registered to an ESX 4.0 Update 2 or later host:
- Power off the virtual machine.
- Open the virtual machine's Snapshot Manager.
- Remove one or more snapshots by click Delete or Delete All.
- Allow time for the consolidation to complete. Larger snapshots and delta disks will take more time to complete.
VMware ESX 4.0 Update 1 or earlier
Review the virtual machine's configuration and identify the locations and filenames of the attached virtual disks.
- Verify what specific delta virtual disk(s) the virtual machine is referencing. You can determine this either by reviewing the virtual machine hardware configuration, highlighting a virtual disk, and noting each disk's file name, or you can review the virtual machine's configuration ( .vmx) file.
- Also note the size of each virtual machine hard disk (specifically the base disk files). Destination clones will be of these determined sizes.
Cloning of virtual disks may be required as part of your effort towards consolidation and reclaiming of space. Verify that you have enough space on one or more datastores to receive a consolidated clone of the virtual machine in entirety or in part (a virtual disk).
Perform a sanity check of the virtual machine disk tree, verifying Content ID (CID) values between multiple snapshot delta disk files, and other virtual machine disk descriptor values. For more important information on this topic, see: Resolving "The parent virtual disk has been modified since the child was created" CID mismatch errors (1007969).
- Conduct a review of datastore space consumption and assess what datastores disk or virtual machine clones can be saved to, as part of a consolidation effort.
- Clone the virtual machine by using the VMware vSphere or Infrastructure Client. See: Cloning Virtual Machines in VMware vCenter and VirtualCenter (1027865).
- Clone individual virtual machine disks using the vMA or vCLI. See: Cloning Individual Virtual Machine Disks via vMA and vCLI (1027872).
- Clone individual virtual machine disks using the terminal. See: Cloning individual virtual machine disks via the VMware ESX host terminal (1027876).
Note: Third party backup software may fail to back up a virtual machine if there are orphaned delta or ctk files in the VM directory with an error about missing ctk.vmdk files. This occurs as the snapshot or change tracking files cannot be created because they are already in use. To work around this issue, rename or remove the orphaned .delta.vmdk or .ctk file(s).
- Portuguese: Como consolidar instantâneos (2032583)
- Spanish: Consolidando Imágenes de Disco Snapshots (1032486)
- Cannot power on a virtual machine because the virtual disk cannot be opened (1004232).
- Resolving "The parent virtual disk has been modified since the child was created" CID mismatch errors (1007969).
- Understanding virtual machine snapshots in VMware ESXi and ESX (1015180)
After understanding the snapshot or delta disk layout, you may begin to clone disks, as required. The information collected above provides the ability to roll back changes, wherever required, as well as necessary details for the cleanup (deletion) process.
This is possible using vmkfstools provided in the ESX host terminal, vMA, and vCLI. The -E switch will move the disk, and the -i switch will clone it. We recommend you to use -i option as this will preserve the source disk. For more information, see Cloning and converting virtual machine disks with vmkfstools (1028042)
When you have selected a process, perform it in this order:
- Remove the virtual machine hard disk from the virtual machine's configuration.
- Complete your migration using the VMware vSphere or Infrastructure client's migrate functionality when connected to vCenter. Otherwise attempt this using the terminal, vMA, or vCLI with vmkfstools.
- When the migration completes, re-add the virtual machine hard disk to the virtual machine's configuration.
- Start the virtual machine.
Cleanup: Removing the original/old virtual machine disks via vMA and vCLINote: It is important that the virtual machine remains powered-on with its files locked throughout these steps. While powered on, deletion of in-use files (which are otherwise required) is not possible, in the event of a mistake being made anywhere in the deletion process. This procedure also guarantees an audit trail. Terminal access may not offer a complete command history.
- Confirm the virtual machine is running.
- List the virtual machine's datastore contents. Some or all of the snapshots should still be present.
vifs --server <vCenter or ESX IP/Hostname> -D '[Datastore Name] examplevm'
- For the virtual disk that was successfully cloned, delete its original delta and base disk files:
vmkfstools --server <vCenter or ESX IP/Hostname> -U '[Datastore Name] examplevm/examplevm-000003.vmdk'
vmkfstools --server <vCenter or ESX IP/Hostname> -U '[Datastore Name] examplevm/examplevm-000002.vmdk'
vmkfstools --server <vCenter or ESX IP/Hostname> -U '[Datastore Name] examplevm/examplevm-000001.vmdk'
vmkfstools --server <vCenter or ESX IP/Hostname> -U '[Datastore Name] examplevm/examplevm.vmdk'
This error is displayed if you attempt to remove disk files that are locked or in-use. This otherwise may be indicative of attempting to remove the wrong disk files:
- Confirm that the disk files have been removed:
vifs --server <vCenter or ESX IP/Hostname> -D '[Datastore Name] examplevm'
Cleanup: Removing the base disk and related snapshot delta disks via the ESX host terminalNote: See the section Confirming the Virtual Machine's Snapshot Delta Disk Layout Using the ESX Host Terminal below before proceeding with the provided steps.
Confirm that the respective delta disk files are not locked while this virtual machine is powered-on. They should not be, at this stage. Run this command:
touch -a examplevm-??????-delta.vmdk
Note: Running the command with "??????" would test all delta files in the directory. If you have multiple virtual hard disks, substitute "examplevm-??????" with applicable values to the virtual disk in question (previously cloned).
If the command is successful to all files, no output is given and you are returned to the prompt. If there are failures due to locking, however:
touch: cannot touch `examplevm-000001-delta.vmdk': Device or resource busy
touch: cannot touch `examplevm-000002-delta.vmdk': Device or resource busy
Remove the snapshot delta and base disk extent/descriptor files using the vmkfstools command.
vmkfstools -U examplevm-000003.vmdk
vmkfstools -U examplevm-000002.vmdk
vmkfstools -U examplevm-000001.vmdk
vmkfstools -U examplevm.vmdk
Note: The descriptor files are referenced, as opposed to the extents (*-delta.vmdk or *-flat.vmdk files)
Again, any failures due to locked files will present an error:
Failed to delete virtual disk: Device or resource busy (1048585).