Knowledge Base

The VMware Knowledge Base provides support solutions, error messages and troubleshooting guides
 
Search the VMware Knowledge Base (KB)   View by Article ID
 

Software iSCSI logins fail to complete in time causing datastore unavailability on EqualLogic arrays (2007829)

Symptoms

  • Software iSCSI based EqualLogic array cannot failover.
  • EqualLogic array event log reports logins failed due to an initiator disconnect during the iSCSI login.

Cause

The EqualLogic array uses a load balancing algorithm to spread out iSCSI session between the initiator and the array iSCSI eth ports. Prior to moving or creating a new iSCSI session to the eth port, the array uses an ICMP ping to verify that the initiator’s network port can connect to that eth port.
 
When multiple vmkernel ports on the same subnet, the VMkernel TCP/IP stack uses the first vmkernel port as the route to the local subnet. For example, if four vmkernel ports are configured on the same subnet, as would be the case when configuring iSCSI for Round Robin MPIO, and a ping is sent to the first, second, third, or fourth vmkernel port, the reply to that ping is sent from the first vmkernel port.
 
The configuration requirements for iSCSI multipathing stipulate that each vmkernel port uses a single physical NIC as an uplink. As a result, if the physical uplink for the first vmkernel port goes down, no failover is possible at the network level. As a result, any pings to the other 3 vmkernel ports fail, even though they are still up and reachable.
 
The failed ping replies causes additional delay to the iSCSI login processing, and can potentially delay the iSCSI login beyond the default VMware iSCSI login timeouts of 5 seconds for vSphere 5.0 and 15 seconds for vSphere 4.1. This causes the ESX/ESXi host to mark the datastore as unavailable.

Resolution

Create a highly available vmkernel port

Dell recommends creating the highly available vmkernel port as described in the Dell EqualLogic technical report TR1075 Configuring iSCSI Connectivity with VMware vSphere 5 and Dell EqualLogic PS Series Storage.
 
If you are using the Dell EqualLogic multipathing extension module, see Dell EqualLogic technical report TR1074 Configuring and Installing the EqualLogic Multipathing Extension Module for VMware vSphere 5.0 and PS Series SANs.
 
These technical reports provide a detailed explanation on how to create the highly available vmkernel port, which ensures that ping replies can be transmitted to the array.
 
For more information on the above articles, refer to storage vendor Dell.

Disable Delayed ACKs

To disable delayed ACKs from the vSphere client:
  1. Click the ESX/ESXi host you want to modify.
  2. Navigate to the Configuration tab and click Storage Adapters.
  3. Click the iSCSI vmhba to be modified and click Properties.
  4. Modify the delayed ACK setting using the option that best matches your needs using one of the processes below.

        To modify the delayed ACK setting on a discovery address (recommended):
    • Click the Dynamic Discovery tab.
    • Click the Server Address tab.
    • Click Settings > Advanced.

        To modify the delayed ACK setting on a specific target:
    • Click the Static Discovery tab and select the target.
    • Click Settings > Advanced.

        To modify the delayed ACK setting globally:
    • Click the General tab.
    • Click Advanced.

  5. In the Advanced Settings dialog box, scroll to the delayed ACK setting.
  6. Deselect Inherit From parent.
  7. Deselect DelayedAck.
  8. Reboot the host.
  9. Repeat for all hosts that have access to the iSCSI storage.
Adjust the iSCSI login timeout on ESXi 5.0
 
With ESXi 5.0 the iSCSI login timeout is currently set 5 seconds, this means that after 5 seconds the ESXi host kills the iSCSI session if there is no response, and tries to login again immediately after. This places additional load on the Storage Array, and can result in a ‘login storm’.

 
With ESXi5.0 patch 2 (build 515841 - ESXi500-201112001) there is the ability to change this setting from the vSphere client. To help alleviate this problem extend the login timeout to 15 seconds, or 30 seconds if necessary.
 
To change the login timeout from the vSphere client:
  1. Go to Storage Adapters > iSCSI Software Adapter > Properties.
  2. Select Advanced and scroll down to LoginTimeout.
  3. Change the value from 5 seconds to a larger value, such as 15 or 30 seconds.
To adjust the iSCSI login timeout in ESXi 5.0, run this command from the command line:

esxcli iscsi adapter param set -A adapter_name -k LoginTimeout -v value_in_sec

For example:

esxcli iscsi adapter param set -A vmhba33 -k LoginTimeout -v 60

Notes:

  • This option is grayed out if you are not running ESXi 5.0 patch 2 (build 515841).
  • You need to reboot the host to reflect the correct timeout settings.
  • As per the Release Notes for ESX/ESXi 4.1 U3, it is now possible to set the timeout value settings for iSCSI initiator login.
 
There are reports of performance improvements on Equalogic arrays by disabling the delayed ACK parameter. For further/related information, see ESX/ESXi hosts might experience read performance issues with certain storage arrays (1002598).

Update History

11/09/2012 - Added command to adjust the iSCSI login timeout in ESXi 5.0

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 16 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.
What can we do to improve this information? (4000 or fewer characters)
  • 16 Ratings
Actions
KB: