Consolidating/Committing snapshots in VMware ESXi

Products

VMware vSphere ESXi

Issue/Introduction

Identifying the symptoms before considering the right steps to successfully consolidate the Virtual machine snapshots.

The most typical causes for VMs to shows ‘Virtual Machine disks consolidation is needed’ Alert.

Snapshots cannot be deleted/consolidated after completing backups.
There is not enough space on the Datastore to perform consolidation. VM disk/disks would be residing on the datastore which has less than 1 GB available space.
Third-party backup application (Veeam, Unitrends, Dataprotect) has locked snapshot files, and failed to remove the snapshot after completing backups or failed to initiate backups.
vCenter server and the ESXi host connectivity issues.
When there are more than the VMware recommended number of snapshots, consolidation may fail. (VMware recommends only 32 as the maximum number of snapshots under best practices).
When large snapshots are undergoing consolidation, VM may show unresponsive/frozen but the alert continues to show up.

Check for the appropriate scenarios

Validate the error message observed while trying to consolidate snapshot.

Troubleshooting issues resulting from locked virtual disks (2107795)
"An error occurred while consolidating disks" when deleting one or more snapshots or consolidate disks (2150414)
Snapshot removal task stops at 99% in ESXi (1007566)
Determining if a virtual machine is using snapshots (1004343)
Was the snapshot created manually or by a backup appliance?

"Failed to lock the file or One or more disks are busy" deleting virtual machine snapshots (2017072)

Monitoring VMware Snapshots and Best practices

Environment

VMware vSphere ESXi 6.5
VMware vSphere ESXi 5.5
VMware vSphere ESXi 7.0.0
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.0

Resolution

Overview

Typically, the snapshot file is in use by the virtual machine until the snapshot is consolidated/deleted. When a snapshot is deleted, any additional files in the hierarchy that are not identified by the Snapshot Manager are included in the commit process.
Sometimes, if the VMs show consolidation needed and no snapshots are present under the snapshot manager, creating a new snapshot and deleting it clears the entire hierarchy. This means that all snapshot files on the virtual machine are committed, then deleted.

A small amount of free space is required to create the new snapshots. Check this KB to validate the required space. If the virtual machine needs to remain running, more space must be allowed as the new snapshot grows (accepting new changes to the virtual disks) as the older snapshots commit.

Checking if virtual machine consolidation is required with the Web Client
Checking if virtual machine is running on a snapshot, via Command Line
Commit snapshots using the vSphere Web Client and HTML5 Client 6.x/7.x
Commit snapshots using the vSphere Client 6.0
Commit snapshots from the ESXi command line

Checking if virtual machine consolidation is required with the Web Client

To check if virtual machine consolidation is required:

Select a vCenter Server host or a cluster and click the Virtual Machines tab.
Right-click any of the virtual machine and click Snapshot Manager/Manage snapshots, it would show "Needs Consolidation/Delete Snapshots".
Note: A Configuration Issue warning is also displayed in the Summary tab of the virtual machine indicating that virtual machine disk consolidation is needed.

Alternatively, you can edit the settings of the virtual machine and see if any of the virtual disks are using a -00000X.vmdk file. If no disks are using -00000X.vmdk, this virtual machine is not using any of these files. Although unlikely, it is possible that another virtual machine is storing its snapshots in this directory. Check the other virtual machines. If none of them refer to these files, they can be safely erased.

Checking if virtual machine is running on a snapshot, via Command Line

Open the virtual machine configuration file (.vmx). Entries in the configuration file (.vmx) file related to virtual disks start with scsix:x. You can do this while connected to an ESXi host through DCUI or SSH. This command shows you the information of the virtual disks of a virtual machine:

grep file /vmfs/volumes/datastoreName/VMname/VMname.vmx

Example:
grep vmdk "/vmfs/volumes/lun1/TestVM/TestVM.vmx"
scsi0:0.fileName = "TestVM-000001.vmdk"

You can also use the vim-cmd commands which is mentioned under the
"Commit snapshots from the ESXi command line" section.

Commit snapshots using the vSphere Web Client and HTML5 Client 6.x/7.x

To commit all snapshots by using the vSphere Client:

Take a Snapshot. For more information, see the Take a Snapshot section in the vSphere Product Documentation .
Delete all Snapshots. For more information, see the Delete all Snapshots section in the vSphere Product Documentation.
Consolidate Snapshots. If the Delete All snapshots does not work, with vSphere 6.x/7.x you can try to Consolidate snapshots. For more information, see the Consolidate Snapshots section in the vSphere Product Documentation.

Commit snapshots using the vSphere Client 6.0

To commit all snapshots by using the vSphere Client:

Take a Snapshot. For more information, see the Take a Snapshot section in the vSphere Product Documentation .
Delete all Snapshots. For more information, see the Delete all Snapshots section in the vSphere Product Documentation .

Note: If you have unsuccessfully attempted to consolidate using the Take a Snapshot then Delete all Snapshots technique, attempt the technique again, but this time select the Quiesce guest file system checkbox to enable quiesced snapshots when you Take a Snapshot.

Commit snapshots from the ESXi command line

To commit all snapshot from the command line on ESXi:

Log into the ESXi host as root via the DCUI console or a SSH session.
Run this command to get a list of virtual machines and the Virtual Machine ID (vmid) for each virtual machine:

vim-cmd vmsvc/getallvms

The output appears similar to:

Vmid Name File Guest OS Version Annotation
1 vm1 [datastore1] vm1/vm1.vmx windows7Server64Guest vmx-08
3 testvm [iscsi1] testvm/testvm.vmx winNetDatacenterGuest vmx-08

Make a note of the Vmid for the specific virtual machine.
To verify if the snapshot exists, run this command and check the Snapshot Name, Snapshot Created On, and Snapshot State:

vim-cmd vmsvc/snapshot.get [Vmid]

The output appears similar to:

Get Snapshot:
|-ROOT
--Snapshot Name : SnapshotTest
--Snapshot Desciption :
--Snapshot Created On : 11/27/2019 13:49:55
--Snapshot State : powered on
Run this command to create a new snapshot:

vim-cmd vmsvc/snapshot.create [VmId] [snapshotName] [snapshotDescription] [includeMemory] [quiesced]

For example, to create a snapshot on the virtual machine named testvm:

vim-cmd vmsvc/snapshot.create 3 SnapshotTest AsnapshotDescription 0 0

Notes:
- You can use any name you like. The name appears in the snapshot manager.
- A file .vmsd may interfere with the creation or removal of the snapshots if a memory snapshot was left behind. For more information, see Understanding virtual machine snapshots in VMware ESXi and ESX (1015180).
Run this command to remove all snapshots:

vim-cmd vmsvc/snapshot.removeall [Vmid]

Additional Information

The remove snapshot process can take a long time to complete if the snapshots are large.
To monitor the commit of snapshots, see Commands to monitor snapshot deletion in VMware ESX/ESXi (1007566).
The commit process has no progress that you can follow. As long as the date on the files continues to update, the process is working. Also, if the virtual machine is off, you can use the file * command to see if any files are in use. If any of the files return the error message can't read `filename' (Device or resource busy, they are locked by the VMkernel. The commit process is actively being performed on those files.
When the commit has completed successfully, there are no -00000X.vmdk or -00000X-delta.vmdk files left unless they were not part of the snapshot tree. These files can be deleted. To confirm that the commit succeeded, view the .vmx file, and verify that virtual disks are now pointing to a base disk (-flat.vmdk).
If the .vmx contains a disk that is still pointing to a snapshot file, the commit process failed. If the attempt was made with the virtual machine running, plan a maintenance windows and try again with the virtual machine powered off. If that does not work, the virtual disk must be cloned using vmkfstools -i. The source file name is the current active -00000X.vmdk as identified in the .vmx file. When the clone is complete, point the virtual machine to use the newly cloned disk. The original base disk and snapshot tree can be deleted.
As an alternative to using the command line, you can use the vSphere Client, vSphere Web Client or HTML5 Client to perform the clone operation on the virtual machine. The cloned virtual machine retains the content of the associated snapshot disks at the time of cloning. When the virtual machine has been cloned successfully, test the operation of the resulting cloned virtual machine and decommission the previous virtual machine. For more information on cloning a virtual machine using the Web Client, see Clone an Existing Virtual Machine.
This issue may be due to left over snapshots from the Virtual Machine Backup tool left unconsolidated, Please do follow the article Unable to delete the virtual machine snapshots.

Note: Cloning using the vSphere Client, vSphere Web Client or HTML5 Client requires vCenter Server and all applicable licenses and does not allow you to select individual virtual disks connected to the virtual machine. Relative to using the vmkfstools -i command to perform the same operation on a single virtual disk, consolidating snapshots using vSphere Clients clones all disks and may require more disk space.