The output of esxtop show dropped receive packets at the virtual switch

Products

VMware vSphere ESXi

Issue/Introduction

The network screen output in esxtop show dropped receive packets (%DRPRX) at the virtual switch port.

Warning: The changes described in this article may impact guest networking at the time of the change. You may need to reboot the guest to recover. VMware recommends you to schedule downtime for applications running within the guest to which you are making changes prior to performing the steps.

Environment

VMware vSphere ESXi 6.0
VMware ESX 4.1.x
VMware vSphere ESXi 5.5
VMware ESX Server 3.5.x
VMware vSphere ESXi 5.0
VMware vSphere ESXi 5.1
VMware ESXi 6.5.x
VMware ESXi 3.5.x Embedded
VMware ESXi 3.5.x Installable
VMware ESX 4.0.x
VMware vSphere ESXi 6.5
VMware ESXi 4.0.x Installable
VMware ESXi 4.1.x Installable
VMware ESX Server 3.0.x
VMware ESXi 4.0.x Embedded
VMware ESXi 4.1.x Embedded

Resolution

Esxtop might show receive packets dropped at the virtual switch if the virtual machine’s network driver runs out of receive (RX) buffer memory the packets are treated in a FIFO (First in first out basis) which the network can be degraded. Even though esxtop shows the packets as dropped at the virtual switch, they are actually dropped between the virtual switch and the guest operating system driver.

The number of dropped packets can be reduced by increasing the Rx buffers for the virtual network driver.

Warning: The changes described in this article may impact guest networking at the time of the change. You may need to reboot the guest to recover. VMware recommends you to schedule downtime for the applications running within the guest to which you are making changes prior to performing the steps.

E1000 Virtual Network Driver

Linux:

For the E1000 virtual network driver in a Linux guest operating system, Rx buffers can be modified from the guest operating system in exactly the same way as on the physical machine. The default value is 256, and the maximum value that can be manually configured is 4096. Determine an appropriate setting by experimenting with different buffer sizes.

To determine the appropriate setting by running the ethtool -G ethX rx value command.

Windows:

For the Intel pro driver in Windows, Receive Buffers can be modified from the guest operating system in exactly the same way as on the physical machine. The default value for the inbox driver on Windows 2008 R2 is 256, this may very depending on the driver used. To determine the appropriate setting by experimenting with different buffer size, load the Intel pro driver to the guest operating system and modify the Receive Buffers in the driver’s property.

For more information on a related issue, see High rate of dropped packets for guests using E1000 or E1000E virtual network adapter on the same ESXi 5.x host (2056468) and vCenter Server 5.1 performance charts report dropped network packets (2052917).

VMXNET2 / Enhanced VMXNET2 Virtual Network Driver on ESX/ESXi 3.x.x

The maximum Rx buffers supported are 128, whereas the default is 120. To increase the default value, add this line in the virtual machine’s .vmx configuration file:

Ethernet x.numRecvBuffers=128

Where x refers to your virtual NIC.

VMXNET2 / Enhanced VMXNET2 Virtual Network Driver on ESX/ESXi 4.x.x

The maximum Rx buffers is increased to 512, whereas the default is 150. To increase the default value, add this line in the virtual machine’s .vmx configuration file.

Ethernet x.numRecvBuffers=value

Where x refers to your virtual NIC and value refers to the new value for the Rx buffer size.

VMXNET3

For Linux guest operating systems

The default Rx ring size is 256 and the maximum is 4096. The default setting can be modified from within the guest operating system.

For example, for a Linux guest operating system:

ethtool -G ethX rx value

Where X refers to the Ethernet interface ID in the guest operating system, and value refers to the new value for the Rx ring size.

Additionally, a Linux virtual machine enabled with Large Receive Offload (LRO) functionality on a VMXNET3 device might experience packet drops on the receiver side when the Rx ring #2 runs out of memory. This occurs when the virtual machine is handling packets generated by LRO.

As of ESXi 5.1 Update 3, the Rx ring #2 can be configured through the rx-jumbo parameter in ethtool. The maximum ring size for this parameter is 2048.

ethtool -G ethX rx-jumbo value

Where X refers to the Ethernet interface ID in the guest operating system, and value refers to the new value for the Rx ring #2.

Note: Even though ESXi supports a maximum of 2048 for the Rx ring #2 size, the maximum value used will depend on the Linux guest operating system's kernel version and other variables.

For Windows guest operating systems

In ESXi/ESX 3.x.x, you cannot configure the ring size of the VMXNET3 network interface card in Windows guest operating systems. In ESXi/ESX 4.0 Update 2 and later, you can configure these parameters from the Device Manager (a Control Panel dialog box) in Windows guest operating systems: Rx Ring #1 Size, Rx Ring #2 Size, Tx Ring Size, Small Rx Buffers, and Large Rx Buffers.

The default value of the size of the first Rx ring, Rx Ring #1 Size, is 512. You can modify the number of Rx buffers separately using the Small Rx Buffers parameter. The default value is 1024.

For some processes (for example, traffic that arrives in burst), you might need to increase the size of the ring, while for others (for example, applications that are slow in processing receive traffic) you might increase the number of the receive buffers.

When jumbo frames are enabled, you might use a second ring, Rx Ring #2 Size. The default value of RX Ring #2 Size is 32. The number of large buffers that are used in both RX Ring #1 and #2 Sizes when jumbo frames are enabled is controlled by Large Rx Buffers. The default value of Large Rx Buffers is 768.

NOTE: Windows Servers must be on NDIS 6.3 or higher to change RX Ring #2 (See Internal Notes)
Overview of NDIS versions - Windows drivers

For Solaris guest operating systems

If the driver is already loaded, unload it by entering the following command:

# rem_drv vmxnet3s
To change the size of Rx rings, edit the vmxnet3.conf file available in the /kernel/drv/amd64 and /kernel/drv/ directories.
Reload the driver by entering the following commands:

# add_drv -i "pciex15ad,7b0" vmxnet3s

# ifconfig vmxnet3sx plumb