Knowledge Base

The VMware Knowledge Base provides support solutions, error messages and troubleshooting guides

Search the VMware Knowledge Base (KB)

View by Article ID

Removing a LUN containing a datastore from VMware ESXi 4x and ESX 4.x (1029786)

Purpose

This article provides steps for unpresenting a LUN containing a datastore from an ESXi/ESX 4.x host without removing the contents of the datastore.

Note: The way ESX rescans storage has changed with patch ESX400-200912401-BG. For more information, see:

Deleting a datastore from vCenter Server and subsequent unpresentation of the LUN containing that datastore is effective most of the time. However, in ESXi/ESX 4.x, removing a LUN without masking the LUN causes the host to continuously retry access to the LUN, and leads to an All-Paths-Down (APD) condition on the host. For more information, see ESX hosts in APD may appear Not Responding in vCenter Server (1030980).

To avoid such issues, you may have to mask the LUN and then unpresent the LUN.

Note: To unpresent a LUN on an ESXi 5.x host, see Unpresenting a LUN in ESXi 5.x (2004605).

Resolution

Unpresenting a LUN containing a datastore from an ESXi/ESX 4.x host includes these steps:

Note: These steps must be performed using the ESX service console or the vSphere command line. These steps cannot be performed using the Manage Paths dialog in the vSphere Client.

Step 1: Identifying the volume in question

To identify the volume in question:

From the vSphere Client, identify (by name) the datastore that you want to remove. If this is an RDM, go to Step 3.
From an SSH session to the host, associate this name to an NAA identifier.

To do this, run the command:

esxcfg-scsidevs --vmfs

Example output:

mpx.vmhba0:C0:T0:L0:5 /dev/cciss/c0d0p5 4a9fd860-6098929b-70e1-001e0b1df518 0 Storage2 naa.60a9800050334c424a6f55566d466f31:1 /dev/sdd1 4b5f0b60-a09c425d-3953-001e0b1df518 0 NetApp_iSCSI naa.6006016033201c00a43a4ab9be9cde11:1 /dev/sda1 4bc37846-18f3c301-2ecb-001e0b1df518 0 delete_datastore

In this case, the datastore delete_datastore has naa.6006016033201c00a43a4ab9be9cde11, which is the NAA ID. You may also see an EUI ID, or a T10 ID, which are fully interchangeable.
For RDMs:
1. Edit the settings of the virtual machine in question, select the RDM you want to remove, and copy the vml.XXX number. This is the vml.ID to be used in the next step.
2. Remove the RDM from the virtual machine and select Delete from disk before clicking OK. This will destroy the mapping file, but not affect LUN content.
3. On the host, run this command:
  
  esxcfg-scsidevs –u | grep –i vml.ID
  
  This command returns the NAA ID of that volume.
From an SSH session to the host, associate this NAA ID with the vmhba identifiers.

To do this, run the command:

esxcfg-mpath –L | grep naa.ID

For example:

# esxcfg-mpath -L | grep -i naa.6006016033201c00a43a4ab9be9cde11

Example output:

vmhba1:C0:T1:L1 state:standby naa.6006016033201c00a43a4ab9be9cde11 vmhba1 0 1 1 NMP standby san fc.20000000c9739842:10000000c9739842 fc.50060160c1e0b7ec:5006016941e0b7ec vmhba1:C0:T0:L1 state:active naa.6006016033201c00a43a4ab9be9cde11 vmhba1 0 0 1 NMP active san fc.20000000c9739842:10000000c9739842 fc.50060160c1e0b7ec:5006016141e0b7ec vmhba0:C0:T1:L1 state:standby naa.6006016033201c00a43a4ab9be9cde11 vmhba1 0 1 1 NMP standby san fc.20000000c9739842:10000000c9739842 fc.50060160c1e0b7ec:5006016941e0b7ec vmhba0:C0:T0:L1 state:active naa.6006016033201c00a43a4ab9be9cde11 vmhba1 0 0 1 NMP active san fc.20000000c9739842:10000000c9739842 fc.50060160c1e0b7ec:5006016141e0b7ec

In this example, the initial portion of the output, such as vmhba1:C0:T1:L1 and vmhba1:C0:T0:L1, breaks down to HBA1/0, Controller 0, Target (SP) 1/0, Lun 1.

Note the portions that change (normally HBA and Target) between paths, and which stay the same (LUN).

Some storage arrays use the same LUN number but change targets, such as Equalogic and Lefthand. If you are not sure if your array uses differing targets or differing LUNs, DO NOT PROCEED and contact VMware Support.

Step 2: Deleting the datastore when you do not want to preserve the data from the VMFS volumes for later use

Note: For RDMs, skip this section.

To delete the VMFS datastore, from the vSphere Client right-click the specific virtual machine volume and click Delete. If this fails, DO NOT PROCEED and contact VMware Support.

Step 3: Masking the volume when you want to preserve data from the VMFS volumes for later use or if the volume is already deleted

For this step, consider for example that the HBAs are changing. In this case, you need two rules to mask the volume: One for HBA1 and another for HBA2. You may use any rule number from 101 to 200. This example uses 192 and 193.

Before proceeding, ensure that you:

Verify that the datastore is not in use.
Unregister all objects stored on the datastore from the inventory, such as virtual machines and templates.
Verify that there are no third-party scripts or utilities running on a service console which could access the LUN in question.

Connect to each ESX host using SSH.
Run these commands:

esxcli corestorage claimrule add --rule 192 –t location –A vmhba1 –L 1 –P MASK_PATH esxcli corestorage claimrule add --rule 193 –t location –A vmhba0 –L 1 –P MASK_PATH

If your environment uses only a single HBA (or Software iSCSI), you need only a single command.

The -t location and -P MASK_PATH are mandatory. The other options are optional and are based on your environment. This example specifies both HBAs (vmhba0 and vmhba1), no controller (masking all controllers), no target (masking all targets), and LUN1 (masking only LUN1). If your storage uses targets to differentiate LUNs, you may want to specify the appropriate target with -T X. For more information, see the example at the end of this article.
Run this command on each host to load the claimrule:

esxcli corestorage claimrule load
Run this command on each host and verify that the claimrule has loaded:

esxcli corestorage claimrule list

Example output:

# esxcli corestorage claimrule list Rule Class Type Plugin Matches 0 runtime transport NMP transport=usb 1 runtime transport NMP transport=sata 2 runtime transport NMP transport=ide 3 runtime transport NMP transport=block 4 runtime transport NMP transport=unknown 101 runtime vendor MASK_PATH vendor=DELL model=Universal Xport 101 file vendor MASK_PATH vendor=DELL model=Universal Xport 192 runtime location MASK_PATH adapter=vmhba1 channel=* target=* lun=1 192 file location MASK_PATH adapter=vmhba1 channel=* target=* lun=1 193 runtime location MASK_PATH adapter=vmhba1 channel=* target=* lun=1 193 file location MASK_PATH adapter=vmhba1 channel=* target=* lun=1 65535 runtime vendor NMP vendor=* model=*

Verify that each new rule has entries for show file and runtime. This indicates that the rules are saved and loaded. If you do not see this, DO NOT PROCEED and contact VMware Support.
Run this command on all hosts to unclaim the volume in question:

esxcli corestorage claiming reclaim –d naa.ID
Confirm that steps 1-5 have been completed on all hosts.
Run this command:

In ESX:
cat /var/log/vmkernel

In ESXi:
grep vmkernel /var/log/messages
In the output of the command, locate entries similar to:

Apr 12 16:18:03 ds-tse-h12 vmkernel: 4:03:31:44.853 cpu7:4109)WARNING: NMP: nmp_UnclaimPath: Physical path "vmhba1:C0:T1:L1" is the last path to NMP device "Unregistered Device". The NMP device has been unregistered.

Warning: If you see any of these entries, DO NOT PROCEED and contact VMware support immediately for assistance:
- Unable to unclaim path vmhba33:C0:T0:L1 on device naa.60a9800050334c424a6f55566d466f31. Some paths may be left in an unclaimed state. You will need to claim them manually using the appropriate commands or wait for periodic path claiming to reclaim them automatically.
- Apr 12 16:30:51 ds-tse-h12 vmkernel: 4:03:44:33.242 cpu5:4109)ScsiDevice: 1930: Can't unregister device naa.60a9800050334c424a6f55566d466f31 because it is in use. OpenCount:1 RefCount:2
- Apr 12 16:30:51 ds-tse-h12 vmkernel: 4:03:44:33.242 cpu5:4109)WARNING: NMP: nmp_DeviceUnregister: Could not unregister NMP device "naa.60a9800050334c424a6f55566d466f31". Busy

Step 4: Unpresenting the LUN from the array

After the LUN is masked from all the hosts, unpresent the LUN from the SAN side. For more information, contact your storage array vendor.

Note: Ensure that the correct LUN is unpresented by verifying the NAA ID on the host and the array.

Step 5: Rescanning all hosts

To rescan all hosts:

Connect to vCenter Server using the vSphere Client.
Right-click the cluster and click Rescan for Datastores.

Alternatively, navigate to Configuration > Storage adapters > Rescan on each host.
To verify that the rescan was successful, run this command from all hosts:

In ESX:
egrep –i apd /var/log/vmkernel

In ESXi:
cat /var/log/messages |grep -i apd

If this command returns anything, contact VMware Support immediately.

Step 6: Restoring normal claimrules

To restore normal claimrules, perform these steps for every host that had visibility to the LUN, or from all hosts on which you created rules earlier:

To remove the rule created in Step 3: Masking the volume when you want to preserver data from the VMFS volumes for later use, run this command:

esxcli corestorage claimrule delete --rule 192

In this example, run this command for rules 192 and 193. Your environment may differ based on the number that you provided in the Step 3: Masking the volume when you want to preserver data from the VMFS volumes for later use section.
Run this command to reload the claimrules:

esxcli corestorage claimrule load
Run this command and verify that the created rule is deleted:

esxcli corestorage claimrule list

Example output:

# esxcli corestorage claimrule list Rule Class Type Plugin Matches 0 runtime transport NMP transport=usb 1 runtime transport NMP transport=sata 2 runtime transport NMP transport=ide 3 runtime transport NMP transport=block 4 runtime transport NMP transport=unknown 101 runtime vendor MASK_PATH vendor=DELL model=Universal Xport 101 file vendor MASK_PATH vendor=DELL model=Universal Xport 65535 runtime vendor NMP vendor=* model=*

Note: You do not see the rules that you created earlier.

Perform a rescan on all ESX hosts that had visibility to the LUN. If all of the hosts are in a cluster, right-click the cluster and click Rescan for Datastores. Previously masked LUNs should now be accessible to the ESX host.

Additional Information

Appendix 1: Storage by Target

Some arrays use storage by target instead of storage by LUN. On a traditional array, such as Clariion and HP EVA, each controller is a target and individual volumes are specified by LUN number. On a target-based array, such as Equallogic and LeftHand, each volume is a separate SCSI target, each with LUN0. On these arrays, the output of the esxcfg-mpath appears similar to:

vmhba33:C0:T2:L0 state:active naa.6000eb395d4b82660000000000000010 vmhba33 0 2 0 NMP …
vmhba33:C0:T1:L0 state:active naa.6000eb395d4b82660000000000000012 vmhba33 0 1 0 NMP …
vmhba33:C0:T0:L0 state:active naa.6000eb395d4b82660000000000000014 vmhba33 0 0 0 NMP …

On each line of the output, you see L0 for LUN 0 and T0/1/2 for each of the actual volumes.

In this case the masking commands are different and you must mask the target instead of the LUN. For example, use a command similar to:

esxcli corestorage claimrule add --rule 192 -t location -A vmhba33 -T 1 -P MASK_PATH

This command masks target 1, LUN 0.

This Article Replaces

1015084

Update History

05/08/2012 - Added information and link to KB 1030980

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

1 2 3 4 5
26 Ratings

1 2 3 4 5
26 Ratings

Actions

Bookmark Document

KB:

Did this article help you?
This article resolved my issue. This article did not resolve my issue. This article helped but additional information was required to resolve my issue.
What can we do to improve this information? (4000 or fewer characters)