ESXi host that uses Intel Corporation Ethernet Controller X710 for 10GbE SFP+ NIC with the i40e driver fails with PSOD

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

ESXi host that uses Intel Corporation Ethernet Controller X710 for 10GbE SFP+ NIC fails with a PSOD error.
Multiple driver/firmware combinations of the i40e driver are affected.
The purple diagnostic screen contains entries similar to:
@BlueScreen: #PF Exception 14 in world wwww:WorldName IP 0xnnnnnnnnnn addr 0x0 PTEs:0x0;
0xnnnnnnnnnnnn [0xnnnnnnnnnnnn] i40e_lan_xmit_frame@<None>#<None>+0x3a4
0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn] [email protected]#9.2+0x137
0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn] UplinkDevTransmit@vmkernel#nover+0x295
0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn] NetSchedFIFORunLocked@vmkernel#nover+0x1a5
0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn] NetSchedFIFOInput@vmkernel#nover+0x24e
0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn] NetSchedInput@vmkernel#nover+0x191
0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn] IOChain_Resume@vmkernel#nover+0x247
0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn] PortOutput@vmkernel#nover+0xe3
0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn] TeamOutputPerFrame@<None>#<None>+0x26f
0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn] TeamES_Output@<None>#<None>+0x115
0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn] EtherswitchPortDispatch@<None>#<None>+0x13e6
0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn] Port_InputResume@vmkernel#nover+0x146
0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn] Port_Input_Committed@vmkernel#nover+0x29
0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn] Vmxnet3VMKDevTQDoTx@vmkernel#nover+0x2f8
0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn] Vmxnet3VMKDev_AsyncTx@vmkernel#nover+0xd7
0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn] NetWorldletPerVMCB@vmkernel#nover+0xae
0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn] WorldletProcessQueue@vmkernel#nover+0x489
0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn] WorldletBHHandler@vmkernel#nover+0x60
0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn] BH_Check@vmkernel#nover+0x185
0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn] VMMVMKCall_Call@vmkernel#nover+0x27a
0xnnnnnnnnnnnn:[0xnnnnnnnnnnnn] __vmk_versionInfo_str@<None>#<None>+0xf14dd015

Environment

VMware vSphere ESXi 5.5
VMware vSphere ESXi 6.5
VMware vSphere ESXi 5.1
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.0

Cause

This issue occurs with Intel X710/XL710/X722/XXV710 ESXi (VMKLinux API based) driver versions up to 2.0.6, which occasionally cause hosts with TSO enabled to fail with a purple diagnostic screen.

Resolution

This is a known issue affecting ESXi 6.x series using the i40e driver.

To work around this issue, use one of the options:

Intel recommends ESXi Native mode driver for Intel X710/XL710/X722/XXV710 products info here. When using ESXi native drivers for these devices, hosts with TSO enabled do not fail with a purple diagnostic screen. Refer to https://kb.vmware.com/s/article/2147565
Alternatively, if using native drivers is not an option for you, you can disable TSO, TSO6, and LRO on the ESXi host.
For more information, see Understanding TCP Segmentation Offload (TSO) and Large Receive Offload (LRO) in a VMware environment (2055140).

To disable TSO:

Run this command to determine if the hardware TSO is enabled on the host:
esxcli system settings advanced list -o /Net/UseHwTSO
Run this command to disable TSO at the host level:
esxcli system settings advanced set -o /Net/UseHwTSO -i 0
Run this command to disable TSO6 at the host level:
esxcli system settings advanced set -o /Net/UseHwTSO6 -i 0

To disable LRO:

Run this command to determine if LRO is enabled for the VMkernel adapters on the host:
esxcli system settings advanced list -o /Net/TcpipDefLROEnabled
Run this command to disable LRO for all VMkernel adapters on a host:
esxcli system settings advanced set -o /Net/TcpipDefLROEnabled -i 0

Note : The preceding command can only take effect after reboot.

Additional Information

To be alerted when this document is updated, click the Subscribe to Article link in the Actions box

Understanding TCP Segmentation Offload (TSO) and Large Receive Offload (LRO) in a VMware environment
使用 Intel Corporation Ethernet Controller X710 for 10GbE SFP+ NIC 的 ESXi 主机失败并出现紫色诊断屏幕
Intel Corporation Ethernet Controller X710 for 10GbE SFP+ NIC を使用する ESXi ホストが PSOD で失敗する