Knowledge Base

The VMware Knowledge Base provides support solutions, error messages and troubleshooting guides
 
Search the VMware Knowledge Base (KB)   View by Article ID
 

Software iSCSI logins fail to complete in time causing datastore unavailability on EqualLogic arrays (2007829)

Symptoms

  • Software iSCSI based EqualLogic array cannot failover.
  • The EqualLogic array event log reports failed logins due to an initiator disconnect during the iSCSI login.

Cause

The EqualLogic array uses a load balancing algorithm to spread out iSCSI sessions between the initiator and the array iSCSI Ethernet ports. Prior to moving or creating a new iSCSI session to the Ethernet port, the array uses an ICMP ping to verify that the initiator's network port can connect to that Ethernet port.

When multiple VMkernel ports on the same subnet, the VMkernel TCP/IP stack uses the first VMkernel port as the route to the local subnet. For example, if four VMkernel ports are configured on the same subnet, as would be the case when configuring iSCSI for Round Robin MPIO, and a ping is sent to the first, second, third, or fourth VMkernel port, the reply to that ping is sent from the first VMkernel port.

The configuration requirements for iSCSI multipathing stipulate that each VMkernel port uses a single physical NIC as an uplink. As a result, if the physical uplink for the first VMkernel port goes down, no failover is possible at the network level. Any pings to the other three VMkernel ports fail, even though they are still up and reachable.

The failed ping replies cause an additional delay to the iSCSI login processing, and can potentially delay the iSCSI login beyond the default VMware iSCSI login timeouts of 5 seconds for vSphere 5.0 and 15 seconds for vSphere 4.1. This causes the ESXi/ESX host to mark the datastore as unavailable.

Note: This behavior is changed in ESXi 5.1 and later hosts. For more information, seeĀ Change to ICMP ping response behavior in ESXi 5.1 (2042189).

Resolution

Create a highly available VMkernel port

Dell recommends creating the highly available VMkernel port as described in the Dell EqualLogic technical report TR1075, Configuring iSCSI Connectivity with VMware vSphere 5 and Dell EqualLogic PS Series Storage.

If you are using the Dell EqualLogic multipathing extension module, see Dell EqualLogic technical report TR1074, Configuring and Installing the EqualLogic Multipathing Extension Module for VMware vSphere 5.0 and PS Series SANs.

Note: The preceding links were correct as of July 18, 2013. If you find a link is broken, provide feedback and a VMware employee will update the link.

These technical reports provide a detailed explanation on how to create the highly available VMkernel port, which ensures that ping replies can be transmitted to the array.

For more information on the above articles, contact the storage vendor, Dell.

Disable Delayed ACKs

To disable delayed ACKs from the vSphere client:
  1. Click the ESXi/ESX host you want to modify.
  2. Navigate to the Configuration tab and click Storage Adapters.
  3. Click the iSCSI VMHBA to be modified and click Properties.
  4. Modify the delayed ACK setting using the option that best matches your needs using one of these processes:

    • To modify the delayed ACK setting on a discovery address (recommended):
      1. Click the Dynamic Discovery tab.
      2. Click the Server Address tab.
      3. Click Settings > Advanced.

    • To modify the delayed ACK setting on a specific target:
      1. Click the Static Discovery tab and select the target.
      2. Click Settings > Advanced.

    • To modify the delayed ACK setting globally:
      1. Click the General tab.
      2. Click Advanced.

  5. In the Advanced Settings dialog box, scroll to the delayed ACK setting.
  6. Deselect Inherit from parent.
  7. Deselect DelayedAck.
  8. Reboot the host.
  9. Repeat for all hosts that have access to the iSCSI storage.

Adjust the iSCSI login timeout on ESXi 5.0

In ESXi 5.x, the iSCSI login timeout is currently set 5 seconds. This means that after 5 seconds the ESXi host kills the iSCSI session if there is no response, and tries to log in again immediately after. This places additional load on the Storage Array, and can result in a "login storm".

The ability to change this setting from the vSphere Client has been added with VMware ESXi 5.0 Patch Release ESXi500-201112001 (2007680). To help alleviate this problem, extend the login timeout to 15 seconds, or 30 seconds if necessary.

To change the login timeout from the vSphere Client:
  1. Go to Storage Adapters > iSCSI Software Adapter > Properties.
  2. Click Advanced and scroll down to LoginTimeout.
  3. Change the value from 5 seconds to a larger value, such as 15 or 30 seconds.
To adjust the iSCSI login timeout in ESXi 5.0 from the command line, run this command:
esxcli iscsi adapter param set -A adapter_name -k LoginTimeout -v value_in_sec
For example:
esxcli iscsi adapter param set -A vmhba33 -k LoginTimeout -v 60
Notes:
  • This option is grayed out if you are not running ESXi 5.0 patch 2 (build 515841).
  • You must reboot the host to reflect the correct timeout settings.
  • As per the Release Notes for ESXi/ESX 4.1 U3, it is now possible to set the timeout value settings for iSCSI initiator login.

Additional Information

There are reports of performance improvements on EqualLogic arrays by disabling the delayed ACK parameter. For more information, see ESXi/ESX hosts might experience read/write performance issues with certain storage arrays (1002598).

See Also

Update History

11/09/2012 - Added command to adjust the iSCSI login timeout in ESXi 5.0

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 22 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.
What can we do to improve this information? (4000 or fewer characters)
  • 22 Ratings
Actions
KB: