ESXi host fails with purple diagnostic screen with "vmk_PktListPopFirstPkt" in the backtrace

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

The PSOD backtrace may contain entries similar to:

2021-12-13T15:58:10.047Z cpu7:2098219)Backtrace for current CPU #7, worldID=2098219, fp=0x453a1c49be60
2021-12-13T15:58:10.047Z cpu7:2098219)0x453a1c49bbe8:[0x42001baa3da3]vmk_PktListPopFirstPkt@vmkernel#nover+0xb stack: 0x0, 0x0, 0x420000000000, 0x0, 0x0
2021-12-13T15:58:10.047Z cpu7:2098219)0x453a1c49bbf0:[0x42001c531024][email protected]#1.0.0+0x201 stack: 0x0, 0x420000000000, 0x0, 0x0, 0x0
2021-12-13T15:58:10.047Z cpu7:2098219)0x453a1c49be70:[0x42001ba6bf0a]Port_InputResume@vmkernel#nover+0x10b stack: 0x42001ba6bef9, 0x0, 0x430700402b80, 0x4301d3e723c0, 0x430700402b80
2021-12-13T15:58:10.047Z cpu7:2098219)0x453a1c49bec0:[0x42001b9e5793]Net_AcceptRxList@vmkernel#nover+0x264 stack: 0x0, 0x4301d3e72280, 0x4301d3e722d8, 0x4313d8444180, 0x4313d8430c00
2021-12-13T15:58:10.047Z cpu7:2098219)0x453a1c49bf70:[0x42001baa5143]NetPollWorldCallback@vmkernel#nover+0xb8 stack: 0x32, 0x42001c73df28, 0x42001baa520e, 0x0, 0x0
2021-12-13T15:58:10.047Z cpu7:2098219)0x453a1c49bfe0:[0x42001bbb1775]CpuSched_StartWorld@vmkernel#nover+0x86 stack: 0x0, 0x42001b8c4700, 0x0, 0x0, 0x0
2021-12-13T15:58:10.047Z cpu7:2098219)0x453a1c49c000:[0x42001b8c46ff]Debug_IsInitialized@vmkernel#nover+0xc stack: 0x0, 0x0, 0x0, 0x0, 0x0
2021-12-13T15:58:10.077Z cpu7:2098219)[45m[33;1mVMware ESXi 7.0.3 [Releasebuild-18825058 x86_64][0m
#PF Exception 14 in world 2098219:vmnic1-pollW IP 0x42001baa3da3 addr 0x0 PTEs:0x801962d023;0x8019633023;0xc14152d023;0x0;

In the /var/run/log/vmkernel.log, we see entries similar to:

2021-12-13T15:58:05.967Z cpu66:2098643)ql_fcoe:vmhba67:SendFCoEVlanSolicitation:1404:Info: Sending FCoEVlanSolicitation request (0x1)
2021-12-13T15:58:06.827Z cpu68:2098638)ql_fcoe:vmhba65:SendFCoEDiscoverySolicitation:1146:Info: Sending FIP discovery for vlan_id = 0x3ea (0x431264972000
2021-12-13T15:58:07.969Z cpu47:2098642)ql_fcoe:vmhba67:SendFCoEVlanSolicitation:1404:Info: Sending FCoEVlanSolicitation request (0x2)
2021-12-13T15:58:08.830Z cpu68:2098638)ql_fcoe:vmhba65:SendFCoEDiscoverySolicitation:1146:Info: Sending FIP discovery for vlan_id = 0x3ea (0x431264972000[7m
2021-12-13T15:58:09.386Z cpu44:2097460)WARNING: qfle3: ecore_state_wait:317: timeout waiting for state 7m
2021-12-13T15:58:09.386Z cpu44:2097460)WARNING: qfle3: ecore_queue_state_change:4992: [vmnic1] Ecore Wait Complete Timed out, manually polling fp[2]. from f01 to 10da
2021-12-13T15:58:09.970Z cpu49:2098642)ql_fcoe:vmhba67:SendFCoEVlanSolicitation:1404:Info: Sending FCoEVlanSolicitation request (0x3)
2021-12-13T15:58:10.013Z cpu7:2098219)World: 3069: PRDA 0x420041c00000 ss 0x0 ds 0xf50 es 0xf50 fs 0xf50 gs 0xf50
2021-12-13T15:58:10.013Z cpu7:2098219)World: 3071: TR 0xf58 GDT 0x45398000e000 (0xf77) IDT 0x42001b94f000 (0xfff)
2021-12-13T15:58:10.013Z cpu7:2098219)World: 3072: CR0 0x8001003d CR3 0x6ffff000 CR4 0x10216c7m
2021-12-13T15:58:10.013Z cpu44:2097460)WARNING: qfle3: ecore_queue_state_change:4999: [vmnic1] fp[2], pending bits cleared

Note:The preceding log excerpts are only examples.Date,time and environmental variables may vary depending on your environment.

Environment

VMware vSphere ESXi 7.0.x
VMware vSphere ESXi 6.7

Cause

This issue is caused due to ecore timeout

Resolution

This issue is resolved from below versions of Qlogic driver

For ESXi 6.7, Qfle3 driver version 1.1.25.0 available at:

https://customerconnect.vmware.com/downloads/details?downloadGroup=DT-ESXI67-MARVELL-MRVL-E3-ETHERNET-ISCSI-FCOE-201820&productId=742

For ESXi 7.0, Qfle3 driver version 1.4.25.0 available at:

https://customerconnect.vmware.com/downloads/details?downloadGroup=DT-ESXI70-MARVELL-E3-ETHERNET-ISCSI-FCOE-301820&productId=974

Workaround:
None

Additional Information

Impact/Risks:
Requires ESXi host reboot