Knowledge Base

The VMware Knowledge Base provides support solutions, error messages and troubleshooting guides
 
Search the VMware Knowledge Base (KB)   View by Article ID
 

QueryChangedDiskAreas API returns incorrect sectors after extending virtual machine VMDK file with Changed Block Tracking (CBT) enabled (2090639)

Symptoms

When using backup software that uses the Virtual Disk Development Kit (VDDK) API call QueryChangedDiskAreas() to return a list of allocated disk sectors, you experience these symptoms:
  • The list of allocated virtual machine disk sectors returned is incorrect.
  • Backups appear to be corrupt or missing data.

Cause

The Virtual Disk Development Kit (VDDK) method QueryChangedDiskAreas() is used by some backup applications to determine which areas of a virtual disk have changed since the last backup, this information is used to decide which blocks to include in a backup.

This issue occurs when you expand a virtual disk .vmdk file which has Change Block Tracking (CBT) enabled past any 128 GB boundary. In ESXi 4.x and earlier versions of ESXi 5.x when the disk is extended, the change tracking data becomes unreliable. Due to the faulty changed block information, some changed blocks might not be captured by a backup. Consequently, a restore operation that uses the incomplete backup could cause a data loss.

Resolution

This is a known issue affecting VMware ESXi 4.x and ESXi 5.x.

This issue is resolved in these releases:

Currently, there is no resolution for ESXi 4.x.

To work around this issue, you must disable CBT and then enable it for disks extended past a 128 GB boundary. You can do this through a vCenter Client only on a powered off virtual machine.
 
To disable CBT and then enable it for disks extended past a 128 GB boundary using the vCenter Client, perform these steps:
  1. Power off the virtual machine.
  2. Use a vCenter client to disable CBT on each disk extended past a 128 GB boundary.
  3. Power on the virtual machine.
  4. Power off the virtual machine.
  5. Use a vCenter client to enable CBT on each disk where you disabled it in step 2.
  6. Power on the virtual machine.

To work around this issue where the virtual machine must remain powered on, perform these steps:

  1. Disable CBT using either PowerCLI or the VDDK API.
  2. Take a snapshot.
  3. Delete the snapshot (To recover space and performance).
  4. Enable CBT using either PowerCLI or the VDDK API.
  5. Take a snapshot.
  6. Delete the snapshot.
The next backup after toggling CBT is the full backup of the virtual machine.

Note: Discard any backups that were captured after growing the disk, as they can be incomplete.

Additional Information

For more information about the VDDK API call QueryChangedDiskAreas(), see the vSphere 5.0 SDK documentation and vSphere 5.1 SDK documentation.

Frequently Asked Questions:

Is the fix retroactive on corrupted .ctk files that already exist?
Basically, the answer is no. If the .ctk files are corrupted the way forward is to trigger a full backup using the methods in the KB. Since making the fix retroactive would require repairing .ctk files and since the data needed to do so is unavailable, repairing suspect backups (or the .ctk files they are based on) is not an available option.

Is there a way to determine if a virtual disk has been expanded?
You should rely on your own change control records to determine whether a virtual disk has been expanded. This information is not tracked in the virtual machine.

Is virtual machine growth in smaller increments affected?
The amount of space the virtual disk is extended is not relevant, the increment of space by which a virtual disk is extended is not relevant. The virtual machine has this issue when its disk is grown past any 128 GB boundary in absolute size. The issue is triggered at other sizes which are a power of 2 from 128 GB up. For example: 256 GB, 512 GB, and 1024 GB.

Check whether CBT is enabled
  • In the home directory of the virtual machine, verify whether a vmname-ctk.vmdk file is saved for one or more virtual hard disks.
  • To Query the advanced configuration parameters for the virtual machine:
    1. Right-click the virtual machine and click Edit Settings.
    2. Click the Options tab.
    3. Click General in the Advanced section and then click Configuration Parameters. The Configuration Parameters dialog opens.
    4. Search for the ctkEnabled parameter entry for each disk and note whether it is enabled.
Note: The VDDK API call configSpec.changeTrackingEnabled = new Boolean(true); can enable and disable CBT dynamically. In this case, you need not power off the virtual machine to change the CBT configuration as taking and deleting a snapshot can apply these settings.

Important: The use of both the VDDK API and scripts is only supported with VMware SDK Developer Support. For more information, see the API 5.5 Documentation and VMware SDK Developer Support Program (1029840).
 
Applying the fix
Using the ExtendVirtualDisk function to grow virtual disks is not an ESX host or a virtual machine specific operation. The vCenter Server can use any host that has access to the datastore for growing a virtual disk. Therefore, you must apply the above resolution to all ESX hosts with access to the datastore managed by vCenter Server.
 

See Also

Update History

12/19/2014 - Updated the Resolution section 01/29/2015 - Updated resolution section with latest patch ESXi 5.5 Patch 4 04/21/2015 - Made the above mentioned yet missing update to include reference to ESXi 5.5 Patch 4 in the resolution section.

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 165 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.

What can we do to improve this information? (4000 or fewer characters)




Please enter the Captcha code before clicking Submit.
  • 165 Ratings
Actions
KB: