Datastore will not mount or LUN resignaturing fails with error "Invalid datastore format"

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

Datastore will not mount
Datastore re-signature fails with error "Invalid datastore format"
hostd.log shows

2019-02-07T05:34:17.300Z info hostd[2099664] [Originator@6876 sub=Vimsvc.TaskManager opID=AddDatastoreWizard-add-23584-ngc:70002264-ec-44-a114 user=vpxuser:Administrator] Task Created : haTask-ha-host-vim.host.DatastoreSystem.resignatureUnresolvedVmfsVolume-1817572049
2019-02-07T05:34:17.300Z info hostd[2395423] [Originator@6876 sub=Libs opID=AddDatastoreWizard-add-23584-ngc:70002264-ec-44-a114 user=vpxuser:Administrator] UnresolvedVmfsVolumeNoSnapShotCache::_Populate: Getting list of UnresolvedVmfsVolume
2019-02-07T05:34:17.315Z info hostd[2395423] [Originator@6876 sub=Libs opID=AddDatastoreWizard-add-23584-ngc:70002264-ec-44-a114 user=vpxuser:Administrator] UnresolvedVmfsVolume: deviceName=naa.600000e00d2c0000002c043b00030000:1,lvmName=5c4446e4-cb507084-911e-4c526214afa0,label=DatastoreName(VMFS),fsUuid=5c4446e4-d9d9fa4a-9dc1-4c526214afa0
2019-02-07T05:34:17.384Z warning hostd[2395423] [Originator@6876 sub=Libs opID=AddDatastoreWizard-add-23584-ngc:70002264-ec-44-a114 user=vpxuser:Administrator] VmFileSystem: Unable to get device properties for volume / device
2019-02-07T05:34:17.385Z info hostd[2395423] [Originator@6876 sub=Vimsvc.TaskManager opID=AddDatastoreWizard-add-23584-ngc:70002264-ec-44-a114 user=vpxuser:Administrator] Task Completed : haTask-ha-host-vim.host.DatastoreSystem.resignatureUnresolvedVmfsVolume-1817572049 Status error
2019-02-07T05:34:17.329Z info hostd[2100090] [Originator@6876 sub=Hostsvc.DatastoreSystem] LvmUpdate: got LVM message [N11HostdCommon17VmkernelUpdateLvmE:0x000000bc825341d0] timestamp=944676853788 discovered=true name=[LikewiseGetDomainJoinInfo:354] QueryInformation(): ERROR_FILE_NOT_FOUND (2/0):

vmkernel.log shows

2019-02-07T05:34:17.362Z cpu50:2395423 opID=40e292ac)WARNING: NFS: 1227: Invalid volume UUID snap-06c78f99-5c4446e4-cb507084-911e-4c526214afa0
2019-02-07T05:34:17.367Z cpu50:2395423 opID=40e292ac)WARNING: Vol3: 3102: DatastoreName/5c4446e4-d9d9fa4a-9dc1-4c526214afa0: Invalid physDiskBlockSize 512
2019-02-07T05:34:17.373Z cpu50:2395423 opID=40e292ac)WARNING: Vol3: 3102: DatastoreName/5c4446e4-d9d9fa4a-9dc1-4c526214afa0: Invalid physDiskBlockSize 512
2019-02-07T05:34:17.376Z cpu50:2395423 opID=40e292ac)WARNING: Vol3: 3102: DatastoreName/5c4446e4-d9d9fa4a-9dc1-4c526214afa0: Invalid physDiskBlockSize 512
2019-02-07T05:34:17.380Z cpu50:2395423 opID=40e292ac)WARNING: Vol3: 3102: DatastoreName/5c4446e4-d9d9fa4a-9dc1-4c526214afa0: Invalid physDiskBlockSize 512
2019-02-07T05:34:17.384Z cpu50:2395423 opID=40e292ac)FSS: 6092: No FS driver claimed device 'snap-06c78f99-5c4446e4-cb507084-911e-4c526214afa0': No filesystem on the device

vCenter vpxd.log shows

2019-02-07T13:34:23.079+08:00 info vpxd[12272] [Originator@6876 sub=Default opID=AddDatastoreWizard-add-23584-ngc:70002264-ec] [VpxLRO] -- ERROR task-1147 -- datastoreSystem-29 -- vim.host.DatastoreSystem.resignatureUnresolvedVmfsVolume: vim.fault.InvalidDatastore:
-- > Result:
-- > (vim.fault.InvalidDatastore) {
-- > faultCause = (vmodl.MethodFault) null,
-- > faultMessage = < unset >,
-- > datastore = < unset >,
-- > name = ""
-- > msg = "Invalid datastore format ''."
-- > }
-- > Args:
-- >
-- > Arg resolutionSpec:
-- > (vim.host.UnresolvedVmfsResignatureSpec) {
-- > extentDevicePath = (string) [
-- > "/vmfs/devices/disks/naa.600000e00d2c0000002c043b00030000:1"
-- > ]
-- > }

Running esxcli storage core device capacity list shows similar to below

Device Physical Blocksize Logical Blocksize Logical Block Count Size Format Type

   --------------------                                                 ------------------           -----------------                     -------------------           -----------               -----------

naa.600000e00d2800000028087800030000     512                           512                               209715200                102400 MiB             512n
naa.600000e00d2800000028087800010000     4096                         512                               734003200                358400 MiB             512e
naa.600000e00d2800000028087800080000     4096                         512                               209715200                102400 MiB             512e

Note: The preceding log excerpts are only examples. Date,time and environmental variables may vary depending on your environment.

Environment

VMware ESXi 6.7.x
VMware vSphere ESXi 6.7

Cause

This happens when original LUN and Backup LUN physical block size/sector format is different.

Resolution

This issue is fixed in ESXi6.7U1. Upgrade the ESXi hosts to 6.7 U1 or later.

To download ESXi 6.7 U1 go to Customer Connect download

Additional Information

For more information on device sector formats refer to VMware documentation
For more information on LUN resignaturing refer to vSphere handling of LUNs detected as snapshot LUNs

Impact/Risks:
ESXi host reboot is required once upgraded.