Changed Block Tracking (CBT) on virtual machines

Products

VMware vSphere ESXi

Issue/Introduction

This article provides an insight on Changed Block Tracking (CBT) and alerts observed while backing up virtual machine related to this feature.

Changed Block Tracking (CBT) is a VMkernel feature that keeps track of the storage blocks of virtual machines as they change over time. The VMkernel keeps track of block changes on virtual machines, which enhances the backup process for applications that have been developed to take advantage of VMware’s vStorage APIs.

Backup appliance uses the VMware API calls to request that a snapshot is created. VMware then takes the snapshot and presents it back so that a backup can be made.
Changed Block Tracking (CBT) is a feature that identifies blocks of data that have changed or in use. It enables incremental backups to identify changes from the last previous backup, writing only changed or in-use blocks.

Symptoms:
When backing up a virtual machine, you experience these symptoms:

The backup of the virtual machine fails.
You see this error similar to:

One or more VM disks have incorrect changed block tracking configuration.
The backup job show errors similar to:

Disk "Hard disk #" has incorrect changed block tracking configuration.
One or more VM disks have incorrect changed block tracking configuration.

Environment

VMware ESXi 4.0.x Installable
VMware ESX 4.0.x
VMware vSphere ESXi 5.5
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.5
VMware ESXi 4.1.x Installable
VMware vSphere ESXi 5.1
VMware vSphere ESXi 6.0
VMware ESXi 4.1.x Embedded
VMware ESXi 4.0.x Embedded
VMware vSphere ESXi 5.0

Resolution

Virtual machines running on ESXi hosts can track disk sectors that have changed. This feature is called Changed Block Tracking (CBT).

On many file systems, CBT identifies the disk sectors altered between two change set IDs. On VMFS partitions, CBT can also identify all the disk sectors that are in use.

If CBT is disabled or not functioning properly, an incremental backup might back up a complete virtual machine disk instead of only changed blocks or revert to a full backup.

Virtual disk block changes are tracked from outside virtual machines, in the virtualization layer. When software performs a backup, it can request transmission of only the blocks that changed since the last backup, or the blocks in use.

The CBT feature can be accessed by third-party backup applications as part of the vSphere APIs. Applications call VAPI to request that the VMkernel return blocks of data that have changed on a virtual disk since the last backup snapshot.

For CBT to identify altered disk sectors since the last change ID, these items are required:

The host must be ESXi 4.0 or later.
The virtual machine owning the disks to be tracked must be hardware version 7 or later.
I/O operations must go through the ESXi storage stack. NFS is supported, RDM in virtual compatibility mode, VMFS is supported, whether backed by SAN, iSCSI, or local disk.
CBT must be enabled for the virtual machine (see below & by default it is enabled).
Virtual machine storage must not be (persistent or non-persistent) independent disk, meaning unaffected by snapshots.

Note: RDM in physical compatibility mode is not supported

For CBT to identify disk sectors in use with the special "*" change ID, these items are required:

The virtual disk must be located on a VMFS volume, backed by SAN, iSCSI, or local disk.
The virtual machine must have zero (0) snapshots when CBT is enabled, for a clean start.

In some cases, such as a power failure or hard shutdown while virtual machines are powered on, CBT might reset and lose track of incremental changes.

Note: Ensure that there are no snapshots on the virtual machine before enabling Changed Block Tracking. If you create snapshots before enabling CBT, the QueryChangedDiskAreas API might not return any error or the data returned by QueryChangedDiskAreas might be incorrect.

To check if a virtual disk has CBT enabled:

Open the vSphere Client and select a powered-off virtual machine.
Right-click the virtual machine and navigate to Edit Settings > Options > Advanced/General > Configuration Parameters.

The option ctkEnabled = "TRUE" is a VMX option which controls the general CBT configuration for the virtual machine. To disable CBT for the complete VM, set the value to False to disable CBT. This setting is stored in the Virtual Machines configuration file <vm_name>,vmx in the VM folder. For more information, see Changed Block Tracking (CBT) on virtual machines (1020128).

For each virtual disk fo the VM, there are additional options named scsix:x.ctkEnabled = "TRUE". To disable CBT on individually on each disk, set the specific option to "FALSE". These options can also be found in the virtual machine configuration file <vm_name>.vmx in the VM folder.

Note: scsi0:0 in scsi0:0.ctkEnabled indicates the SCSI device assigned to the hard disk that is added to the virtual machine. Every hard disk added to the virtual machine is given a SCSI device that appears similar to scsi0:0, scsi0:1, or scsi 1:1. CBT is enabled (or disabled) individually on each disk.

For each virtual disk and snapshot disk there is a .ctk file. For example:

vmname.vmdk
vmname-flat.vmdk
vmname-ctk.vmdk
vmname-000001.vmdk
vmname-000001-delta.vmdk
vmname-000001-ctk.vmdk

Power on the virtual machine.

In the home directory of the virtual machine, verify that each disk having CBT enabled has

also a vmname-ctk.vmdk file.

Note: After a successful backup and full snapshot consolidation, there should be no snapshot related .ctk files remaining in the datastore. For example: vmname-000001-ctk.vmdk.

To disable CBT, ensure that there are no snapshots on the virtual machine. For more information, see:

To disable CBT:

Power off the virtual machine.
Right-click the virtual machine and click Edit Settings.
Click the Options tab.
Click General under the Advanced section and then click Configuration Parameters. The Configuration Parameters dialog opens.
Set the ctkEnabled parameter to false for the desired SCSI disk(s).
Power on the virtual machine.

Additional Information

For more information, see the section Changed Block Tracking on Virtual Disks in the Designing vSphere Backup Solutions section of the Virtual Disk API Programming Guide.

If you are using VMware Horizon View and linked clone or instant clone virtual machines, you should not be using CBT. Always ensure that CBT is disabled for the parent virtual machine.