ESXi 6.5 / 6.7 does not recognize datastores created on the device which has more than 1 VML IDs associated

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

VMFS datastores may be absent after upgrading to ESXi 6.5 despite path availability.
After upgrading an ESXi host version 6.5, one or more datastores may be absent.
The vmkernel log reports an error around UUIDs and device-registration failures:

2017-06-02T17:12:54.076Z cpu23:65974)ScsiUid: 403: Existing device naa.6001f931059e80000113000200000000 already has uid vml.02000400006001f931059e80000113000200000000495345323430
2017-06-02T17:12:54.076Z cpu23:65974)ScsiDevice: 4163: Failing registration of device 'naa.6001f931059e80000113000200000000'
2017-06-02T17:12:54.076Z cpu23:65974)ScsiDevice: 4165: Failed to add legacy uid vml.02000400006001f931059e80000113000200000000495345323430 on path vmhba2:C0:T0:L4: Already exists
2017-06-02T17:12:54.076Z cpu23:65974)WARNING: NMP: nmp_RegisterDevice:851: Registration of NMP device with primary uid 'naa.6001f931059e80000113000200000000' failed. Already exists
The storage path list will reveal multiple paths with No associated device.

# esxcli storage core path list

fc.20000025b5ab007c:20000025b5ac0049-fc.2000001f931059e8:2000001f931059e8-
UID: fc.20000025b5ab007c:20000025b5ac0049-fc.2000001f931059e8:2000001f931059e8-
Runtime Name: vmhba2:C0:T0:L4
Device: No associated device
Device Display Name: No associated device
Adapter: vmhba2
Channel: 0
Target: 0
LUN: 4

Note: This log excerpt is an example. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vSphere ESXi 6.5

Cause

This issue has been observed when a device from the storage array has been presented with multiple LUN numbers to the ESXi hosts. Such devices possess multiple vml identifiers on the ESXi hosts, because a vml identifier is constituted from the LUN identfier (i.e. the identifier beginning, naa, eui, etc.) and the LUN number.

For more information on vml id: Correlating VML ID with naa ID in ESXi 5.x (2078730)

Prior versions of ESXi, handled devices with multiple vmls. Changed behaviour in ESXi 6.5 causes registration failure for such devices.

Example:

# esxcfg-scsidevs -l

naa.600c0ff000261b5db72dc75601000000

Device Type: Direct-Access

Size: 5145644 MB

Display Name: HP Fibre Channel Disk (naa.600c0ff000261b5db72dc75601000000)

Multipath Plugin: NMP

Console Device: /vmfs/devices/disks/naa.600c0ff000261b5db72dc75601000000

Devfs Path: /vmfs/devices/disks/naa.600c0ff000261b5db72dc75601000000

Vendor: HP Model: MSA 1040 SAN Revis: G210

SCSI Level: 6 Is Pseudo: false Status: on

Is RDM Capable: true Is Removable: false

Is Local: false Is SSD: false

Other Names:

vml.0200030000600c0ff000261b5db72dc756010000004d5341203130

vml.0200020000600c0ff000261b5db72dc756030000004d5341203130

VAAI Status: supported

### The above output shows multiple VML ids for the same device.

Resolution

This issue is resolved in ESXi 6.5 U1 release.

Workaround:
To resolve this issue, LUN presentation should be made consistent for every host participating in a cluster that could run the virtual machine, the raw device mapping metadata file should be consistent with that presentation, and vCenter Server’s cache of this information should be accurate.

Additional Information

To avoid service disruptions, always follow the published best practices from your storage vendor when making LUN presentation/mapping changes.

Prior to upgrading to ESXi 6.5, you can determine if you are exposed to this issue by examining device registrations from a pre-6.5 host. When examining the device registration, you should see only one VML identifier associated with each storage device. If you observe multiple VML identifiers, the device will fail registration after upgrade to ESXi 6.5.

For example:

# esxcli storage core device list

naa.6001f931059e80000113000200000000

Display Name: Fibre Channel Disk (naa.6001f931059e80000113000200000000)

Has Settable Display Name: true

Size: 10240

Device Type: Direct-Access

Multipath Plugin: NMP

Devfs Path: /vmfs/devices/disks/naa.6001f931059e80000113000200000000

[ … ]

Is VVOL PE: false

Is Offline: false

Is Perennially Reserved: false

Queue Full Sample Size: 0

Queue Full Threshold: 0

Thin Provisioning Status: unknown

Attached Filters:

VAAI Status: supported

Other UIDs:

vml.02000600006001f931059e80000113000200000000495345323430,

vml.02000800006001f931059e80000113000200000000495345323430

Note: Command output was reduced to improve clarity.

Alternatively, you may examine the short-form path list to list all paths to a device and check for differing LUN numbers.

For example:

# esxcfg-mpath -b

naa.6001f931059e80000113000200000000 : Fibre Channel Disk (naa.6001f931059e80000113000200000000)

vmhba2:C0:T0:L6 LUN:6 state:active fc Adapter: WWNN: 20:00:00:25:b5:ab:00:4f WWPN: 20:00:00:25:b5:ac:00:3f Target: WWNN: 20:00:00:1f:93:10:59:e8 WWPN: 20:00:00:1f:93:10:59:e8

vmhba2:C0:T1:L6 LUN:6 state:active fc Adapter: WWNN: 20:00:00:25:b5:ab:00:4f WWPN: 20:00:00:25:b5:ac:00:3f Target: WWNN: 20:00:00:1f:93:10:59:e8 WWPN: 20:00:00:1f:93:10:59:ed

vmhba1:C0:T3:L8 LUN:8 state:active fc Adapter: WWNN: 20:00:00:25:b5:ab:00:4f WWPN: 20:00:00:25:b5:ac:00:2f Target: WWNN: 20:00:00:1f:93:10:59:e8 WWPN: 20:00:00:1f:93:10:59:e9

vmhba1:C0:T5:L8 LUN:8 state:active fc Adapter: WWNN: 20:00:00:25:b5:ab:00:4f WWPN: 20:00:00:25:b5:ac:00:2f Target: WWNN: 20:00:00:1f:93:10:59:e8 WWPN: 20:00:00:1f:93:10:59:ec

To be alerted when this document is updated, click the Subscribe to Article link in the Actions box in the Action box.

Correlating VML ID with naa ID in ESXi 5.x (2078730) describes how the legacy identifier is generated.ESXi 5.x または 6.0 で認識されたデータストアが ESXi 6.5 で認識されない
ESXi 6.5 无法识别之前 ESXi 5.x 或 6.0 已识别的数据存储