Knowledge Base

The VMware Knowledge Base provides support solutions, error messages and troubleshooting guides

Search the VMware Knowledge Base (KB)

View by Article ID

Virtual machines stop responding when any LUN on the host is in an all-paths-down (APD) condition

Details

The issue addressed in this article occurs when a rescan is issued while an all-paths-down state exists for any LUN in the vCenter Server cluster. Therefore, a virtual machine on one LUN stops responding (temporarily or permanently) because a different LUN in the vCenter Server cluster is in an all-paths-down (APD) state.

These symptoms may indicate you have an all-paths-down state:

You might see intermittent Request timed out (out of 6 or 7 successful pings) while trying to ping the virtual machine.
When powering on a virtual machine with a raw device mapping (RDM) the progress bar stops at 50% and the virtual machine console becomes unresponsive at the VMware splash/loading/BIOS screen.

All-Paths-Down State (APD)

The all-paths-down state is a condition where no working path exists to a storage device (LUN). These situations can cause an all-paths-down state to a LUN:

A hardware failure (permanent or transient)
Removal of a LUN

Rescanning

This article is concerned with the type of rescanning that can cause virtual machines on other LUNs to become unresponsive:

Configuration changes in ESX/ESXi 4.x involving VMFS (Virtual Machine File System)

Configuration of VMFS that involves creating, deleting, or increasing datastores can cause an automated rescan. vCenter Server issues a vCenter Server-wide rescan as part of the workflow to discover storage changes. This rescan allows automatic discovery to maintain a consistent view of storage across all hosts in the cluster. For example, to perform a rescan in ESX 4.0, use this command:

esxcfg-rescan -d vmhba#

Note: The command to perform a rescan on the vmhba using various methods, such as RCLI, vMA, and PowerCLI varies. For appropriate commands, refer the documentation.
Removal of a LUN

The removal of a LUN through an array-based administration action followed by a manual rescan can cause virtual machines on other LUNs to also become unresponsive.

Solution

The issue is resolved in ESX/ESXi 4.1 Update 1 and the fix has also been included with ESXi 5.0.

Notes:

With ESX/ESXi 4.1 Update 1 and ESX/ESXi 4.0 Update 3, you no longer have to make the modification to the advanced setting. Virtual machines that are not associated with the APD Volume(s) do not become unresponsive upon a rescan.
When unpresenting a LUN containing a datastore, follow the instructions in Unpresenting a LUN containing a datastore from ESX 4.x and ESXi 4.x (1015084). If the issue still persists, contact VMware Technical Support.

Workaround

ESX/ESXi 4.x can list all of the LUNs it detects, as well as the state of these LUNs. If none of the paths to a storage device are in the ACTIVE state, then ESX/ESXi considers the device to be in an all-paths-down state. If an all-paths-down state does exist, then this is likely the issue causing LUNs to be unresponsive, either for a limited period of time or permanently, when a rescan occurs. For more information, see Identifying disks when working with VMware ESX (1014953).

If virtual machines are not responding on an ESX/ESXi 4.0 host, determine if an all-paths-down condition exists by executing:

# esxcfg-mpath --list-paths --device <device naa> | grep state

or

# esxcfg-mpath --list-paths --device <device mpx> | grep state

where:

<device naa> is the Network Addressing Authority (NAA) unique address for the full storage device
<device mpx> is the identifier if a NAA ID is not available

Note: For information about using the command line with ESXi, see Tech Support Mode for Emergency Support (1003677).

Starting with ESX/ESXi 4.0 Update 1, you can set an advanced configuration option on all hosts in the vCenter Server cluster to reduce rescan times and to prevent virtual machines from not responding. By default this option is disabled.

Caution: Not every all-paths-down condition is permanent. Some all-paths-down conditions, such as those that occur briefly during a network re-configuration, are transient. Enabling this option can cause devices in a transient all-paths-down state to become unavailable. It is recommended to disable this option after the rescan operation completes.

To enable this option, execute:

# esxcfg-advcfg -s 1 /VMFS3/FailVolumeOpenIfAPD

To disable and reset to the default value without requiring downtime, execute:

# esxcfg-advcfg -s 0 /VMFS3/FailVolumeOpenIfAPD

To check the value of this option, execute:

# esxcfg-advcfg -g /VMFS3/FailVolumeOpenIfAPD

To minimize the amount of time that the virtual machines are unresponsive, apply patch:

Note: This does not apply for ESX/ESXi 4.0 Update 2 and 4.1, because the patch is integrated in these versions.

With ESX 4.1 Update 1 and ESX 4.0 Update 3, you no longer have to make the modification to the advanced setting. Virtual machines that are not associated with the APD Volume(s) do not becomes unresponsive upon a rescan.

Additional Information

For translated versions of this article, see:

Update History

06/24/2011 - Added link to French version. 02/13/2012 - Added command to perform a rescan in ESX 4.0

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

1 2 3 4 5

1 2 3 4 5

Actions

Bookmark Document

Did this article help you?
This article resolved my issue. This article did not resolve my issue. This article helped but additional information was required to resolve my issue.
What can we do to improve this information? (4000 or fewer characters)