This article provides information regarding a potential data inconsistency when very specific conditions are met for VMDK's residing on VMware vSAN 6.6 and later.

Note: This behavior is exhibited only under very specific conditions.

Symptoms:
Applications such as databases report in-guest data inconsistency on a VMDK that has been extended while residing on VMware vSAN 6.6 and later.

This issue occurs in VMware vSAN 6.6 and later when virtual machine disks residing on vSAN storage are extended. This is a rare occurrence, requiring specific conditions to be met.

This is a known issue affecting VMware vSAN 6.6 and later versions. VMware recommends applying the configuration changes suggested in the "Workaround" section to mitigate the risk from encountering this issue on future VMDK extend operations in un-patched environments.

The patch to remediate future impact, and address existing VMs which may be exposed to potential impact, has been released and is available here:
For vSAN 6.7 - Express Patch 4 Release: http://kb.vmware.com/kb/58848
For vSAN 6.6 - Express Patch 9 Release: http://kb.vmware.com/kb/58852


NOTE: VMs which have experienced in-guest data inconsistencies will not be corrected by this patch. Please contact VMware Support for assistance if your virtual machine applications have noted data inconsistency.
 
Customers that run vSAN on a VMware Cloud Foundation (VCF) system should refer to https://kb.vmware.com/s/article/58873 for instructions on how to remediate for this issue.
VCF customers will be getting notification through SDDC Manager in the next few days with the availability of this patch and can schedule the upgrades through their SDDC Manager.


 
VMware has developed a process to discover VMs that are potentially impacted, and we are recommending that customers contact VMware Support prior to upgrading to this patch so that validations can take place for virtual machines. We are suggesting the following update process:
 
- Contact VMware Support and request to speak to a vSAN engineer about this KB# and upgrade process. You can do this through the Customer Connect Portal, or by contact Support at 1-877-4-VMWARE
- Request an evaluation of the vSAN environment and VMs that have potential exposure
- Review the output provided, ensuring backups of any virtual machines that may be impacted
- Patch the VMware vSAN environment with the remediation patch
 
 
Please contact VMware Support for any questions regarding this process and the patch release.
 
 
 
Notes on applying the patch:
- Additional resync will not impact virtual machines further, as the patch is preventative once the first Host has been updated

Workaround:
To work around this issue, and avoid future occurrence of this issue, set the advanced parameter on each Host for in-place expansion to "0". This must be completed on all vSAN nodes, including any vSAN Witness Hosts.

Notes:

• No reboot or service restart is required for this change to take effect, it will become effective within 60 seconds. There is no loss of functionality once the workaround is applied, and VM disks can continue to be extended when needed.
• This change is persistent across Host reboots, and can be made at any time with no impact to running Production workloads.
• Please note that this has to done on all hosts including the witness host in case of ROBO or stretch cluster.

1. Log in to the ESXi host through SSH as root.
2. Run this command to set the value to "0".
   
   esxcfg-advcfg -s 0 /VSAN/ClomEnableInplaceExpansion
    
3. Run this command to confirm the value is set to "0".
   
   esxcfg-advcfg -g /VSAN/ClomEnableInplaceExpansion
    
4. Repeat this process on all ESXi hosts in the vSAN Cluster.

The following PowerCLI script is provided as an example that will ask for a cluster name, and then apply the setting across the entire cluster: Note: If your application reports of data inconsistency errors, contact VMware Support and note this Knowledge Base article ID (58715) in the problem description. To contact VMware support, see Filing a Support Request in Customer Connect (2006985) or How to Submit a Support Request.

Impact/Risks:
Applications such as databases may report in-guest data inconsistency errors on VMDK's residing on VMware vSAN under specific conditions.