Large packet loss in the guest OS using VMXNET3 in ESXi

Products

VMware vSphere ESXi

Issue/Introduction

To resolve the issue of significant packet loss during periods of very high traffic bursts:

Ensure that there is no traffic filtering occurring (for example, with a mail filter).
Check the driver/firmware of the ESXi's physical NICs and update if needed. To check the driver/firmware, see Determining Network/Storage firmware and driver version in ESXi.
1. NOTE: if physical NIC errors are also incrementing, please investigate with physical NIC vendor. See Troubleshooting network receive traffic faults and other NIC errors in ESXi
After eliminating the above possibilities, slowly increase the number of buffers in the guest operating system.

Symptoms:
When using the VMXNET3 driver on a virtual machine on ESXi, you see significant packet loss during periods of very high traffic bursts. The virtual machine may even freeze entirely. Doing one of the following may resolve the issue:

vMotion the VM to another host
disconnect and reconnect the VM's adapter

Environment

VMware vSphere ESXi 6.7
VMware vSphere ESXi 5.5
VMware vSphere ESXi 5.0
VMware vSphere ESXi 5.1
VMware ESXi 4.1.x Embedded
VMware ESXi 4.1.x Installable
VMware vSphere ESXi 7.0.0
VMware vSphere ESXi 6.0
VMware vSphere ESXi 6.5

Cause

This can occur due to a lack of receive and transmit buffer space or when receive traffic which is speed-constrained using, for example a traffic filter.

Resolution

To resolve the issue of significant packet loss during periods of very high traffic bursts with the VMXNET3 vNIC:

Ensure that there is no traffic filtering occurring (for example, with a mail filter).
Check the driver/firmware of the ESXi's physical NICs and update if needed. To check the driver/firmware, see Determining Network/Storage firmware and driver version in ESXi.
After eliminating the above possibilities, slowly increase the number of buffers in the guest operating system.

To reduce burst traffic drops in Windows Buffer Settings:

Click Start > Control Panel > Device Manager.
Right-click vmxnet3 and click Properties.
Click the Advanced tab.
Click Small Rx Buffers and increase the value (The maximum value is 8192).
Click Rx Ring #1 Size and increase the value (The maximum value is 4096).

Notes:

No reboot is required for these changes to take affect. However, any application sensitive to TCP session disruption can fail and have to be restarted. This applies to RDP, so it is better to do this work in a console window.
This issue is seen in the Windows guest operating system with a VMXNET3 vNIC.
It is important to increase the value of Small Rx Buffers and Rx Ring #1 gradually to avoid drastically increasing the memory overhead on the host and possibly causing performance issues if resources are close to capacity.
If this issue occurs on only 2-3 virtual machines, set the value of Small Rx Buffers and Rx Ring #1 to the maximum value. Monitor virtual machine performance to see if this resolves the issue.
The Small Rx Buffers and Rx Ring #1 variables affect non-jumbo frame traffic only on the adapter.
Windows Servers must be on NDIS 6.3 or higher to change RX Ring #2 (See Internal Notes)
Overview of NDIS versions - Windows driver

Additional Information

VMware Skyline Health Diagnostics for vSphere - FAQ
The output of esxtop show dropped receive packets at the virtual switch
Troubleshooting ESX/ESXi virtual machine performance issues
ESXi の VMXNET3 vNIC でゲスト OS レベルのパケットロスが大量に発生する
在 ESXi 中的 VMXNET3 vNIC 上，客户机操作系统级别发生大数据包丢失
Hoher Paketverlust auf Gastbetriebssystem-Ebene auf der VMXNET3-vNIC in ESXi
Network receive traffic faults and other NIC errors in ESXi