VMFS (ATS-only) volumes on SAS Storage failed to automount when upgraded to 7.0 U2 and onwards
search cancel

VMFS (ATS-only) volumes on SAS Storage failed to automount when upgraded to 7.0 U2 and onwards

book

Article ID: 339566

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

This KB provides the troubleshoot elements needed to determine if you have this issue and will provide a workaround specifically to address VMFS (ATS-only) volumes on SAS Storage(Active/Active or Active/Passive) failed to automount when ESXi host is upgraded to 7.0 U2 and onwards.
Note: This KB can be useful to all four VAAI ops we support : ATS Locking, Clone, Zero and Delete. Here the focus is on Locking.


Symptoms:
  • ESXi host recently updated or upgraded to 7.0 U2 and onwards.
  • SAS is used for Storage provisioning VMFS.
  • You can see the impacted datastore's backing device size.
  • You can consult its partition table and metadata.
  • VMFS ATS-only datastores are not mounted.
  • You can see in vmkernel the following logs (names and timestamps may vary depending on your environment):
2021-03-11T12:45:18.375Z cpu5:2113297)WARNING: HBX: 2419: ATS-Only VMFS volume 'Datastore' is not mounted. This host does not support ATS, or ATS initialization failed.
2021-03-11T12:45:18.375Z cpu5:2113297)WARNING: HBX: 2439: Failed to initialize VMFS distributed locking on volume 5fb4c81e-c702312a-3133-9440c93e9d60: Not supported
2021-03-11T12:45:18.375Z cpu5:2113297)Vol3: 4339: Failed to get object 28 type 1 uuid 5fb4c81e-c702312a-3133-9440c93e9d60 FD 0 gen 0 :Not supported
2021-03-11T12:45:18.375Z cpu5:2113297)WARNING: Fil3: 1534: Failed to reserve volume f532 28 1 5fb4c81e c702312a 40943133 609d3ec9 0 0 0 0 0 0 0
2021-03-11T12:45:18.375Z cpu5:2113297)Vol3: 4339: Failed to get object 28 type 2 uuid 5fb4c81e-c702312a-3133-9440c93e9d60 FD 4 gen 1 :Not supported
2021-03-11T12:45:18.425Z cpu4:2113309)HBX: 1016: 'Datastore': HB at offset 3145728 - Setting pulse failed: Not supported:
2021-03-11T12:45:18.425Z cpu4:2113309) [HB state abcdef02 offset 3145728 gen 71 stampUS 3782277213 uuid 604a0231-493248f2-ec8c-9440c93e9d60 jrnl <FB 0> drv 24.82 lockImpl 4 ip 172.xx.xx.51]
2021-03-11T12:45:18.425Z cpu4:2113309)WARNING: FSAts: 1568: Denying reservation access on an ATS-only vol 'Datastore'
  • When you look at the device on SSH using the command "esxcli storage core device list", you see the following:
naa.600c0ff000514ed010c6b45f01000000:
   Display Name: Local HPE Disk (naa.600c0ff000514ed010c6b45f01000000)
   Has Settable Display Name: false
   Size: 4768368
   Device Type: Direct-Access
   Multipath Plugin: HPP
   Devfs Path: /vmfs/devices/disks/naa.600c0ff000514ed010c6b45f01000000
   Vendor: HPE
   Model: MSA 2050 SAS
   Revision: V270
   SCSI Level: 6
   Is Pseudo: false
   Status: on
   Is RDM Capable: true
   Is Local: true
   Is Removable: false
   Is SSD: false
  • The command "esxcli storage core device vaai status get" returns the following:
naa.600c0ff000514ed010c6b45f01000000:
   VAAI Plugin Name:
   ATS Status: unsupported
   Clone Status: unsupported
   Zero Status: unsupported
   Delete Status: unsupported
  • If you still have a host in 7.0U1 to compare you can see the following:
naa.600c0ff000514ed010c6b45f01000000:
   Display Name: HPE Serial Attached SCSI Disk (naa.600c0ff000514ed010c6b45f01000000)
   Has Settable Display Name: true
   Size: 4768368
   Device Type: Direct-Access
   Multipath Plugin: NMP
   Devfs Path: /vmfs/devices/disks/naa.600c0ff000514ed010c6b45f01000000
   Vendor: HPE
   Model: MSA 2050 SAS
   Revision: V270
   SCSI Level: 6
   Is Pseudo: false
   Status: on
   Is RDM Capable: true
   Is Local: false
   Is Removable: false
   Is SSD: false

naa.600c0ff000514ed010c6b45f01000000:
   VAAI Plugin Name:
   ATS Status: supported
   Clone Status: supported
   Zero Status: supported
   Delete Status: supported


Environment

VMware vSphere ESXi 7.0.x
VMware vSphere ESXi 7.0.0

Cause

Until 7.0 U2, VMWare default MPP plugin NMP claimed SAS devices. In 7.0 U2, the default MPP for SAS devices was changed to HPP. Like NMP, HPP also cannot differentiate between local/remote SAS targets unless in case of ALUA targets. In case of NMP also user would have added a NMP SATP claim rule forcing the device to be claimed by VMW_SATP_DEFAULT_AA/VMW_SATP_DEFAULT_AP which will in turn mark the device remote. We need a similar setting with HPP as well which user will have to explicitly setup for the device to mark it as remote. Because this setting is not done, the device is marked as local and hence VAAI operations will be disabled as VMWare disables VAAI operations on local HDDs.

Resolution

Currently there is no resolution.
While we are working on the configuration option with HPP , the current plan is to use NMP itself for the remote SAS target. To workaround the issue follow the below steps.

Workaround:
Case 1: If ESXi is already updated to 7.0 U2 and onwards, add the claimrule using the steps below and then do an unclaim and reclaim of the devices, and if that doesn’t work do a host reboot.
Case 2: If planning to upgrade to 7.0 U2 and onwards, prior to upgrade add the claimrule using the steps below to overcome this issue.
Steps to Workaround:
  1. Make a note of the Vendor/Model string of the device using the command as below. If multiple devices from same target vendor/model, the setting will apply to all devices.

    [root@******:~] esxcli storage core device list -d <device naa/mpx/t10 ID>

    Example output:
    naa.600c0ff0005150c90ec6b45f01000000: 
       Display Name: Local HPE Disk
(naa.600c0ff0005150c90ec6b45f01000000) 
   Has Settable Display Name: false 
   Size: 4768368 
   Device Type: Direct-Access  
   Multipath Plugin: HPP 
   Devfs Path: /vmfs/devices/disks/naa.600c0ff0005150c90ec6b45f01000000 
   Vendor: HPE      
   Model: MSA 2050 SAS     

- Add the claim rule (rule number below 50)

[root@******:~] esxcli storage core claimrule add -r 49 -t vendor -V <Your device vendor> -M <Your device model> -P NMP --force-reserved

- Load the claim rule

[root@******:~] esxcli storage core claimrule load

Check claim rule list to confirm claim rule added successfully

[root@******:~] esxcli storage core claimrule list

 
  • For case 1 do the an unclaim and and reclaim of the devices, and if that doesn’t help do a ESXi host reboot.
  • For case 2 after setting the claimrule go ahead with the ESXi host upgrade.


 

Check whether the devices with the above vendor model are claimed by NMP.

 


Additional Information

Reverting to a previous version of ESXi https://kb.vmware.com/s/article/1033604
Disabling hardware accelerated locking https://kb.vmware.com/s/article/2146451

Powered by Wolken Software