ESXi 5.5/6.0 hosts running Emulex HBAs become unresponsive in vCenter Server during periods of high I/O
search cancel

ESXi 5.5/6.0 hosts running Emulex HBAs become unresponsive in vCenter Server during periods of high I/O

book

Article ID: 320046

calendar_today

Updated On:

Products

VMware

Issue/Introduction


Symptoms:
  • ESXi 5.5/6.0 host running Emulex FC or FCoE HBAs becomes unresponsive in vCenter Server.

    Note: The host still responds to ping and SSH but appears to be stopped at the console.

  • The ESXi 5.5/6.0 host uses the Emulex native driver (lpfc). This issue occurs when you are using lpfc driver version earlier than 10.2.340.18

  • Disconnections during periods of high I/O can cause the Emulex HBA driver to abort I/O.

  • In the /var/log/ vmkernel.log file, you see the entries similar to:

    YYYY-MM-DD TIME 793Z cpu15:789970)WARNING: lpfc: lpfc_sli4_iocbq_lookup:2678: 1:0372 iotag x563 not on txcmplq - max iotag (xec0)

  • In the lpfc driver, you see entries similar to:

    YYYY-MM-DD TIME 512Z cpu3:3472551)WARNING: lpfc: lpfc_abort_handler:2945: 1:(0):0748 abort handler timed out waiting for aborting I/O xri x13ea to complete: ret xbad0001, cmd x2a, tgt_id x0, lun_id x0

  • In the /var/log/vmkernel.log file, you see entries similar to:

    2015-11-24T13:46:11.128Z cpu48:33636)WARNING: lpfc: lpfc_mbx_cmpl_read_topology:3158: 0:1305 Link Down Event x2 received Data: x2 x20 x110 x0
    ...
    2015-11-24T13:46:21.131Z cpu55:33636)WARNING: lpfc: lpfc_dev_loss_tmo_handler:212: 0:(0):0203 Devloss timeout on WWPN 50:00:09:73:00:23:05:30 NPort x430000 Data: x0 x8 x0
    2015-11-24T13:46:21.131Z cpu55:33636)WARNING: lpfc: lpfc_dev_loss_tmo_handler:212: 0:(0):0203 Devloss timeout on WWPN 50:00:09:73:00:23:05:28 NPort x430020 Data: x0 x8 x0
    2015-11-24T13:46:21.131Z cpu55:33636)WARNING: lpfc: lpfc_dev_loss_tmo_handler:212: 0:(0):0203 Devloss timeout on WWPN 50:00:09:73:00:2a:01:30 NPort x430080 Data: x0 x8 x0
    2015-11-24T13:46:21.131Z cpu55:33636)WARNING: lpfc: lpfc_dev_loss_tmo_handler:212: 0:(0):0203 Devloss timeout on WWPN 50:00:09:73:00:2a:01:28 NPort x4300a0 Data: x0 x8 x0
    2015-11-24T13:46:21.131Z cpu55:33636)WARNING: lpfc: lpfc_dev_loss_tmo_handler:212: 0:(0):0203 Devloss timeout on WWPN 50:00:09:73:00:2a:29:30 NPort x430120 Data: x0 x8 x0
    2015-11-24T13:46:21.132Z cpu55:33636)WARNING: lpfc: lpfc_dev_loss_tmo_handler:212: 0:(0):0203 Devloss timeout on WWPN 50:00:09:73:00:2a:29:28 NPort x430140 Data: x0 x8 x0

    Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.


Cause

This issue occurs when you are using lpfc driver version earlier than 10.2.340.18.

The lpfc driver version earlier than 10.2.340.18 causes Emulex HBAs to abort during heavy I/O.

Resolution

This is a known issue affecting ESXi 5.5 and ESXi 6.0 hosts.

To resolve the issue, upgrade the Emulex driver version to 10.2.340.18 available at VMware Downloads.
For more information on installing the driver, see Installing async drivers on VMware ESXi 5.x and ESXi 6.0.x (2005205).

Note:
  • The linked Emulex driver version 10.2.340.18 is also compatible with ESXi 6.0.
  • To obtain the recommended firmware version to use with lpfc driver 10.2.340.18, contact your server vendor.

    Note: The preceding links were correct as of December 22, 2015. If you find the link is broken, provide feedback and a VMware employee will update the link.


Additional Information


How to download and install async drivers in ESXi 5.x/6.x
I/O の負荷が高いときに Emulex HBA を実行する ESXi 5.5/6.0 ホストが vCenter Server で応答しなくなる
在高 I/O 期间,vCenter Server 中运行 Emulex HBA 的 ESXi 5.5/6.0 主机变得无响应