Knowledge Base

The VMware Knowledge Base provides support solutions, error messages and troubleshooting guides
 
Search the VMware Knowledge Base (KB)   View by Article ID
 

Permanent Device Loss (PDL) and All-Paths-Down (APD) in vSphere 5.0 (2004684)

Symptoms

Permanent Device Loss (PDL)

  • A datastore is shown as unavailable in the Storage view.
  • A storage adapter indicates the Operational State of the device as Lost Communication.
  • All paths to the device are marked as Dead.
  • The /var/log/vmkernel.log file shows messages similar to:

    2011-08-09T10:43:26.857Z cpu2:853571)VMW_SATP_ALUA: satp_alua_issueCommandOnPath:661: Path "vmhba3:C0:T0:L0" (PERM LOSS) command 0xa3 failed with status Device is permanently unavailable. H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x25 0x0.
    2011-08-09T10:43:26.857Z cpu2:853571)VMW_SATP_ALUA: satp_alua_issueCommandOnPath:661: Path "vmhba4:C0:T0:L0" (PERM LOSS) command 0xa3 failed with status Device is permanently unavailable. H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x25 0x0.
    2011-08-09T10:43:26.857Z cpu2:853571)WARNING: vmw_psp_rr: psp_rrSelectPathToActivate:972:Could not select path for device "naa.60a98000572d54724a34642d71325763".
    2011-08-09T10:43:26.857Z cpu2:853571)WARNING: ScsiDevice: 1223: Device :naa.60a98000572d54724a34642d71325763 has been removed or is permanently inaccessible.
    2011-08-09T10:43:26.857Z cpu3:2132)ScsiDeviceIO: 2288: Cmd(0x4124403c1fc0) 0x9e, CmdSN 0xec86 to dev "naa.60a98000572d54724a34642d71325763" failed H:0x8 D:0x0 P:0x0
    2011-08-09T10:43:26.858Z cpu3:2132)WARNING: NMP: nmp_DeviceStartLoop:721:NMP Device "naa.60a98000572d54724a34642d71325763" is blocked. Not starting I/O from device.
    2011-08-09T10:43:26.858Z cpu2:2127)ScsiDeviceIO: 2316: Cmd(0x4124403c1fc0) 0x25, CmdSN 0xecab to dev "naa.60a98000572d54724a34642d71325763" failed H:0x1 D:0x0 P:0x0 Possible sense data: 0x5 0x25 0x0.
    2011-08-09T10:43:26.858Z cpu2:854568)WARNING: ScsiDeviceIO: 7330: READ CAPACITY on device "naa.60a98000572d54724a34642d71325763" from Plugin "NMP" failed. I/O error
    2011-08-09T10:43:26.858Z cpu2:854568)ScsiDevice: 1238: Permanently inaccessible device :naa.60a98000572d54724a34642d71325763 has no more open connections. It is now safe to unmount datastores (if any) and delete the device.
    2011-08-09T10:43:26.859Z cpu3:854577)WARNING: NMP: nmpDeviceAttemptFailover:562:Retry world restore device "naa.60a98000572d54724a34642d71325763" - no more commands to retry

All Paths Down (APD)

  • A datastore is shown as unavailable in the Storage view.
  • A storage adapter indicates the Operational State of the device as Dead or Error.
  • All paths to the device are marked as Dead.
  • You are unable to connect directly to the ESXi host using the vSphere Client.
  • The ESXi host shows as Disconnected in vCenter Server.
  • The /var/log/vmkernel.log file shows messages similar to:

    2011-07-30T14:47:41.187Z cpu1:2049)WARNING: NMP: nmp_IssueCommandToDevice:2954:I/O could not be issued to device "naa.60a98000572d54724a34642d71325763" due to Not found
    2011-07-30T14:47:41.187Z cpu1:2049)WARNING: NMP: nmp_DeviceRetryCommand:133:Device "naa.60a98000572d54724a34642d71325763": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device.
    2011-07-30T14:47:41.187Z cpu1:2049)WARNING: NMP: nmp_DeviceStartLoop:721:NMP Device "naa.60a98000572d54724a34642d71325763" is blocked. Not starting I/O from device.
    2011-07-30T14:47:41.361Z cpu1:2642)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60a98000572d54724a34642d71325763" - issuing command 0x4124007ba7c0
    2011-07-30T14:47:41.361Z cpu1:2642)WARNING: NMP: nmpDeviceAttemptFailover:658:Retry world failover device "naa.60a98000572d54724a34642d71325763" - failed to issue command due to Not found (APD), try again...
    2011-07-30T14:47:41.361Z cpu1:2642)WARNING: NMP: nmpDeviceAttemptFailover:708:Logical device "naa.60a98000572d54724a34642d71325763": awaiting fast path state update...
    2011-07-30T14:47:42.361Z cpu0:2642)WARNING: NMP: nmpDeviceAttemptFailover:599:Retry world failover device "naa.60a98000572d54724a34642d71325763" - issuing command 0x4124007ba7c0
    2011-07-30T14:47:42.361Z cpu0:2642)WARNING: NMP: nmpDeviceAttemptFailover:658:Retry world failover device "naa.60a98000572d54724a34642d71325763" - failed to issue command due to Not found (APD), try again...
    2011-07-30T14:47:42.361Z cpu0:2642)WARNING: NMP: nmpDeviceAttemptFailover:708:Logical device "naa.60a98000572d54724a34642d71325763": awaiting fast path state update...

  • A restart of the management agents may show these errors:

    Not all VMFS volumes were updated; the error encountered was 'No connection'.
    Errors:
    Rescan complete, however some dead paths were not removed because they were in use by the system. Please use the 'storage core device world list' command to see the VMkernel worlds still using these paths.
    Error while scanning interfaces, unable to continue. Error was Not all VMFS volumes were updated; the error encountered was 'No connection'.
  • You may also see that the device was not listed any more:

    2012-10-23T09:56:17.751Z cpu17:10107)WARNING: Vol3: 1717: Failed to refresh FS 4beb089b-68037158-2ecc-00215eda1af6 descriptor: Device is permanently unavailable
    2012-10-23T09:56:17.751Z cpu17:10107)ScsiDeviceIO: 2316: Cmd(0x412442939bc0) 0x28, CmdSN 0x367bb6 from world 10107 to dev "eui.00173800084f0005" failed H:0x1 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
    2012-10-23T09:56:17.751Z cpu17:10107)Vol3: 1767: Error refreshing PB resMeta: Device is permanently unavailable

Purpose

This article discusses a Permanent Device Loss (PDL) and All-Paths-Down (APD) in ESXi 5.0, and provides information on dealing with each of these scenarios.

Resolution

In vSphere 4.x, an All-Paths-Down (APD) situation occurs when all paths to a device are down. As there is no indication whether this is a permanent or temporary device loss, the ESXi host keeps reattempting to establish connectivity. APD-style situations commonly occur when the LUN is incorrectly unpresented from the ESXi/ESX host. The ESXi/ESX host, still believing the device is available, retries all SCSI commands indefinitely. This has an impact on the management agents, as their commands are not responded to until the device is again accessible. This causes the ESXi/ESX host to become inaccessible/not-responding in vCenter Server.

In vSphere 5.0, a clear distinction has been made between a device that is permanently lost (PDL) and a transient issue where all paths are down (APD) for an unknown reason.

For example, in the VMkernel logs, if a SCSI Sense code of H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x25 0x0 or Logical Unit Not Supported is logged by the storage device to the ESXi 5.0 host, this indicates that the device is permanently inaccessible to the ESXi host, or is in a Permanent Device Loss (PDL) state. The ESXi host no longer attempts to re-establish connectivity or issue commands to the device.

Devices that suffer a non-recoverable hardware error are also recognized as being in a Permanent Device Loss (PDL) state.

This table outlines possible SCSI sense codes that determine if a device is in a PDL state:

SCSI Sense Code Description
H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x25 0x0 LOGICAL UNIT NOT SUPPORTED
H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x4c 0x0 LOGICAL UNIT FAILED SELF-CONFIGURATION
H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x3e 0x3 LOGICAL UNIT FAILED SELF-TEST
H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x3e 0x1 LOGICAL UNIT FAILURE

For more information about SCSI sense codes on vSphere, see Interpreting SCSI sense codes (289902).

Note: Some iSCSI arrays map LUN-to-Target as a one-to-one relationship. That is, there is only ever a single LUN per Target. In this case, the iSCSI arrays do not return the appropriate SCSI sense code, so PDL on these arrays types cannot be detected.

All-Paths-Down (APD)

If PDL SCSI Sense codes are not returned from a device (when unable to contact the storage array, or with a storage array that does not return the supported PDL SCSI codes), then the device is in an All-Paths-Down (APD) state, and the ESXi host continues to send I/O requests until the host receives a response.

As the ESXi host is not able to determine if the device loss is permanent (PDL) or transient (APD), it indefinitely retries SCSI I/O, including:
  • Userworld I/O (hostd management agent)
  • Virtual machine guest I/O

    Note: If an I/O request is issued from a guest, the operating system should timeout and abort the I/O.
Due to the nature of an APD situation, there is no clean way to recover.
  • The APD situation needs to be resolved at the storage array/fabric layer to restore connectivity to the host.
  • All affected ESXi hosts may require a reboot to remove any residual references to the affected devices that are in an APD state.
Note: Performing a vMotion of unaffected virtual machines is not possible, as the management agents may be affected by the APD condition, and the ESXi server may become unmanaged. As a result, a reboot of an affected ESXi host forces an outage to all non-affected virtual machines on that host.

Planned versus unplanned PDL

A planned PDL occurs when there is an intent to remove a device presented to the ESXi host. The datastore must first be unmounted, then the device detached before the storage device can be unpresented at the storage array. For more information on how to correctly unpresent a LUN in ESXi 5.0, see Unpresenting a LUN in ESXi 5.x (2004605).

An unplanned PDL occurs when the storage device is unexpectedly unpresented from the storage array without the unmount and detach being executed on the ESXi host.

To clean up an unplanned PDL:
  1. All running virtual machines from the datastore must be powered off and unregistered from the vCenter Server.
  2. From the vSphere Client, go to the Configuration tab of the ESXi host, and click Storage.
  3. Right-click the datastore being removed, and click Unmount.

    The Confirm Datastore Unmount window displays. When the prerequisite criteria have been passed, the OK button appears.

    If you see this error when unmounting the LUN:

    Call datastore refresh for object <name_of_LUN> on vCenter server <name_of_vCenter> failed

    You may have a snapshot LUN presented. To resolve this issue, remove that snapshot LUN on the array side.

  4. Perform a rescan on all of the ESXi hosts that had visibility to the LUN.

    Note: If there are active references to the device or pending I/O, the ESXi host still lists the device after the rescan. Check for virtual machines, templates, ISO images, floppy images, and raw device mappings which may still have an active reference to the device or datastore.

  5. If the LUN is still being used and available again, go to each host, right-click the LUN, and click Mount.

    Note: One possible cause for an unplanned PDL is that the LUN ran out space causing it to become inaccessible.

Tags

permanent-device-loss-pdl  all-paths-down-apd

See Also

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 29 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.
What can we do to improve this information? (4000 or fewer characters)
  • 29 Ratings
Actions
KB: