Knowledge Base

The VMware Knowledge Base provides support solutions, error messages and troubleshooting guides
 
Search the VMware Knowledge Base (KB)   View by Article ID
 

ESXi 5.x hosts fail to mount VMFS5 volumes that are formatted with ATS-only capabilities (2006858)

Symptoms

  • If a virtual machine is registered and the ESX host is rebooted, the virtual machine becomes in-accessible (orphaned) after reboot.
  • If a virtual machine is configured for Fault Tolerance, and if the primary ESX host is restarted, the secondary virtual machine will not be created since the VMFS5 volume may fail to mount on the original primary server after restart.
  • If virtual machines are configured for auto-start after poweron/reboot, the virtual machines will fail to start as the VMFS5 volume will not mount.
  • If a VMFS5 volume is unmounted and its device is detached, and removed from an existing ESX server and then presented to a new server (or presented back to the same server again) it will fail to mount the VMFS5 volume.
  • You see the following events in the vmkernel.log log file on the ESXi 5.x host having the issue mounting the VMFS5 volume:

    Setting pulse [HB state abcdef02 offset 3354624 gen 1 stampUS 12741765582 uuid 4e6f910f-9f0cbd23-bb3d-14feb5c9a0ec jrnl <FB 0> drv 14.54] on vol 'jrb163_52' failed: Not supported
    2011-09-13T20:52:48.325Z cpu30:5872)WARNING: FSAts: 1263: Denying reservation access on an ATS-only vol 'jrb163_52'
    2011-09-13T20:52:48.325Z cpu30:5872)WARNING: HBX: 1889: Failed to initialize VMFS3 distributed locking on volume 4e6e0c49-a955df7d-b6a3-14feb5c9aa1d: Not supported
    2011-09-13T20:52:48.325Z cpu30:5872)FSS: 890: Failed to get object f530 28 1 4e6e0c49 a955df7d fe14b6a3 1daac9b5 0 0 0 0 0 0 0 :Not supported
    2011-09-13T20:52:48.325Z cpu30:5872)WARNING: Fil3: 2032: Failed to reserve volume f530 28 1 4e6e0c49 a955df7d fe14b6a3 1daac9b5 0 0 0 0 0 0 0
    2011-09-13T20:52:48.325Z cpu30:5872)FSS: 890: Failed to get object f530 28 2 4e6e0c49 a955df7d fe14b6a3 1daac9b5 4 1 0 0 0 0 0 :Not supported

  • Attempting to perform Test Failover on Site Recovery Manager (SRM) 5.0 fails the failover host may not mount the replicated VMFS5 volume correctly. The following error will be displayed:

    Error - Failed to recover datastore 'DatastoreName'. VMFS volume residing on recovered devices "DeviceNumber" and expected to be auto-mounted during HBA rescan cannot be found

    The SRM 5.0 vmware-dr.log file shows this related error:

    Failed to recover datastore 'DatastoreName'. VMFS volume residing on recovered devices '"DeviceNumber"' and expected to be auto-mounted during HBA rescan cannot be found

Cause

When a VMFS-5 volume formatted on a VAAI capable device, it uses ATS locking, exclusively termed as ATS only. To check if a VMFS volume has been configured with ATS only, use this command:

# vmkfstools -Ph -v1 /vmfs/volumes/VMFS-volume-name

The output appears similar to:

VMFS-5.54 file system spanning 1 partitions.
File system label (if any): ats-test-1
Mode: public ATS-only

When an ESX host discovers and registers new and existing devices, the VAAI plugin module is loaded if the device is deemed to be VAAI capable during PSA device registration. In this particular case, ESX may fail to register device correctly with VAAI plugin module as you are not reading the capacity of the volume correctly to apply the equivalent VAAI pluging module. As a result, the VMFS5 volume may fail to mount.

This issue is seen to occur in, but is not restricted to, EMC arrays Clariion CX4, NS, and VNX arrays.

Resolution

This issue is resolved in ESXi 5.0 Update 1, To download this release see: Download VMware ESXi 5.0 Installable Update 1
For more information see: VMware ESXi 5.0 Update 1 Release Notes

To work around this issue in ESXi 5.0, disable VAAI ATS locking mechanism on your hosts:
  1. Check if VAAI features are enabled on the ESXi 5.x host from a console or SSH session using this command:

    # esxcli system settings advanced list -o /VMFS3/HardwareAcceleratedLocking

    You see an output similar to:

    Path: /VMFS3/HardwareAcceleratedLocking
       Type: integer
       Int Value: 1
       Default Int Value: 1
       Min Value: 0
       Max Value: 1
       String Value:
       Default String Value:
       Valid Characters:
       Description: Enable hardware accelerated VMFS locking (requires compliant hardware)

    The Int Value of 1 is indicating that VAAI features are enabled.

  2. Disable Hardware Accelerated Locking to ensure that the VMFS datastores are stable on the cluster:

    1. Log in to vCenter Server 5.0 or ESXi 5.0 host using the vSphere Client.
    2. In the vSphere Client inventory panel, choose the ESXi 5.x host.
    3. Click the Configuration tab, then click Advanced Settings under Software.
    4. Change the value of this parameter to 0:

      VMFS3.HardwareAcceleratedLocking

      Alternatively this can be disabled from the console with the command:

      # esxcli system settings advanced set -i 0 -o /VMFS3/HardwareAcceleratedLocking

  3. Verify that VAAI has been disabled using the esxcli again:

    # esxcli system settings advanced list -o /VMFS3/HardwareAcceleratedLocking

    You see an output similar to:

    Path: /VMFS3/HardwareAcceleratedLocking
       Type: integer
       Int Value: 0
       Default Int Value: 1
       Min Value: 0
       Max Value: 1
       String Value:
       Default String Value:
       Valid Characters:
       Description: Enable hardware accelerated VMFS locking (requires compliant hardware)
       
    The Int Value of 0 is indicates that VAAI features are disabled.


Notes:

  • Disabling ATS locking takes effect immediately and does not require a host reboot. 
  • Enabling or disabling ATS should not impact any VMs running but it is advisable to make this change during a maintenance window.
  • This change must be implemented on each host individually. All hosts able to connect to this storage must be configured consistently in regards to ATS.

Additional Information

To be alerted when this article is updated, click Subscribe to Document in the Actions box.

See Also

Update History

10/29/2012 - Added ESXi 5.1 to products

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 18 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.
What can we do to improve this information? (4000 or fewer characters)
  • 18 Ratings
Actions
KB: