Search the VMware Knowledge Base (KB)
View by Article ID

ESXi hosts fail to mount VMFS5 volumes that are formatted with ATS-only capabilities (2006858)

  • 32 Ratings

Symptoms

  • If a virtual machine is registered and the ESX host is rebooted, the virtual machine becomes inaccessible (orphaned) after reboot.
  • If a virtual machine is configured for Fault Tolerance and if the primary ESX host is restarted, the secondary virtual machine will not be created since the VMFS5 volume may fail to mount on the original primary server after restart.
  • If virtual machines are configured for auto-start after power on/reboot, the virtual machines fail to start as the VMFS5 volume mounts.
  • If a VMFS5 volume is unmounted and its device is detached, removed from an existing ESX server, and then presented to a new server (or presented back to the same server again), it fails to mount the VMFS5 volume.
  • In vCenter Server, you see an error similar to:

    A general system error occurred: Timed-out while waiting to get datastore information from host


  • In the vmkernel.log file of the ESXi 5.0 host that has issues mounting the VMFS5 volume, you see entries similar to:

    Setting pulse [HB state abcdef02 offset 3354624 gen 1 stampUS 12741765582 uuid 4e6f910f-9f0cbd23-bb3d-14feb5c9a0ec jrnl <FB 0> drv 14.54] on vol 'jrb163_52' failed: Not supported
    YYYY-MM-DD TIME 325Z cpu30:5872)WARNING: FSAts: 1263: Denying reservation access on an ATS-only vol 'jrb163_52'
    YYYY-MM-DD TIME 325Z cpu30:5872)WARNING: HBX: 1889: Failed to initialize VMFS3 distributed locking on volume 4e6e0c49-a955df7d-b6a3-14feb5c9aa1d: Not supported
    YYYY-MM-DD TIME 325Z cpu30:5872)FSS: 890: Failed to get object f530 28 1 4e6e0c49 a955df7d fe14b6a3 1daac9b5 0 0 0 0 0 0 0 :Not supported
    YYYY-MM-DD TIME 325Z cpu30:5872)WARNING: Fil3: 2032: Failed to reserve volume f530 28 1 4e6e0c49 a955df7d fe14b6a3 1daac9b5 0 0 0 0 0 0 0
    YYYY-MM-DD TIME 325Z cpu30:5872)FSS: 890: Failed to get object f530 28 2 4e6e0c49 a955df7d fe14b6a3 1daac9b5 4 1 0 0 0 0 0 :Not supported

    Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment. 

  • Attempting to perform Test Failover on Site Recovery Manager (SRM) 5.0 fails.
  • The failover host may not mount the replicated VMFS5 volume correctly and reports this error:

    Error - Failed to recover datastore 'DatastoreName'. VMFS volume residing on recovered devices "DeviceNumber" and expected to be auto-mounted during HBA rescan cannot be found


  • In the SRM 5.0 vmware-dr.log file, you see the error similar to:

    Failed to recover datastore 'DatastoreName'. VMFS volume residing on recovered devices '"DeviceNumber"' and expected to be auto-mounted during HBA rescan cannot be found.

Cause

When a VMFS-5 volume formatted on a VAAI capable device, it uses ATS locking, exclusively termed as ATS only. To check if a VMFS volume is configured with ATS only, use this command:

# vmkfstools -Ph -v1 /vmfs/volumes/VMFS-volume-name

You see output similar to:

VMFS-5.54 file system spanning 1 partitions.
File system label (if any): ats-test-1
Mode: public ATS-only
 
When an ESX host discovers and registers new and existing devices, the VAAI plug-in module is loaded if the device is deemed to be VAAI capable during PSA device registration. In this particular case, ESX may fail to register device correctly with VAAI plug-in module as you are not reading the capacity of the volume correctly to apply the equivalent VAAI Plugin module. As a result, the VMFS5 volume may fail to mount.

This issue is seen to occur in EMC arrays Clariion CX4, NS, and VNX arrays, but is not restricted to these.

Resolution

This issue is resolved in ESXi 5.0 Update 1, available at VMware Downloads. For more information about this version, see the VMware ESXi 5.0 Update 1 Release Notes.

This issue is also resolved in ESXi 5.1 (See mentions of VAAI in 5.1, 5.1 U1, and 5.1U2 Release Notes in 5.1 Documentation Library for exception.)

To work around this issue in ESXi 5.0, disable VAAI ATS locking mechanism on your hosts. For more information, see Disabling the VAAI functionality in ESXi/ESX (1033665)

 

Additional Information

  • Disabling VAAI entirely on the ESXi host may introduce issues in the environment. Instead of disabling VAAI for all devices, you can disable it only for the affected LUN without impacting other LUNs.

    • To downgrade VMFS-5 from ATS capability, run this command:

      vmkfstools --configATSOnly 0 device path


    • To upgrade VMFS-5 to ATS capability, run this command:

      vmkfstools --configATSOnly 1 device path

      Where device path is the path of the head extent device on which VMFS-5 was formatted.

      For example:

      ~ # vmkfstools --configATSOnly 0 /vmfs/devices/disks/naa.600601607ae02600e85affc3f022e411:1

      VMware ESX Question:
      VMFS on device /dev/disks/naa.600601607ae02600e85affc3f022e411:1 will be upgraded to or downgraded from ATS capability. Please ensure that the VMFS-5 volume is not in active use by any local or remote ESX 4.x servers.

      Continue with configuration of ATS capability?

      0) _Yes
      1) _No

      Select a number from 0-1: 0

      Checking if remote hosts are using this device as a valid file system. This may take a few seconds...
      Downgrading VMFS-5 on '/dev/disks/naa.600601607ae02600e85affc3f022e411:1' from ATS capability...done


  • In ESXi 5.0 and later, the HP P2000 MSA iSCSI/FC Array requires the VIB hp_vaaip_p2000_offline-bundle-210.zip to be installed. HP does not include the VIB on their vSphere ESXi 5.x install media. If you have installed ESXi from an older media, it may still have the VIB and therefore VAAI hardware acceleration. The re-installed hosts lacking the VIB show entries similar to:

    YYYY-MM-DD TIME 519Z [65AE2B70 info 'Vimsvc.ha-eventmgr'] Event 2467 : ATS-Only VMFS volume 'HP 2000 SAN Vol1' not mounted. Host does not support ATS or ATS initialization has failed.

    To resolve this issue, install the hp_vaaip_p2000_offline-bundle-210.zip after the upgrade. To download hp_vaaip_p2000_offline-bundle-210.zip, see the HP depot.

    Note: The preceding link was correct as of 21 January 2015. If you find the link is broken, provide a feedback and a VMware employee will update the link.

To be alerted when this article is updated, click Subscribe to Document in the Actions box.

See Also

Update History

07/02/2013 - Added additional symptom. 02/14/2014 - Added vCenter Server and ESXi 5.1 and 5.5 to Product Versions. 04/02/2014 - Changed under Cause # vmkfstools -Ph -v1 /vmfs/volumes/VMFS-volume-name

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 32 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.

What can we do to improve this information? (4000 or fewer characters)




Please enter the Captcha code before clicking Submit.
  • 32 Ratings
Actions
KB: