Low virtual network I/O performance with Intel 1Gbps NIC using igb driver
search cancel

Low virtual network I/O performance with Intel 1Gbps NIC using igb driver

book

Article ID: 326272

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • In a virtual environment, for certain workloads and (or) configurations, the network performance achieved on an Intel 1Gbps NIC using the igb driver might be low because the interrupt throttling rate for the igb driver is not optimal for that workload.
  • The network Bandwidth of Intel 82576 and Intel i340T4/i350 do not go up to the supported 1 Gb/s
  • It performs slower than other 1Gb/s cards


Environment

VMware vSphere ESXi 6.0
VMware vSphere ESXi 5.1
VMware vSphere ESXi 5.5
VMware vSphere ESXi 5.0

Resolution

The InterruptThrottleRate parameter of the igb driver that regulates the rate at which interrupts are generated by the adapter affects network and system performance. The igb driver uses dynamic conservative mode (setting 3) as the default value in which it dynamically adjusts the InterruptThrottleRate based on the incoming traffic. The valid range of the InterruptThrottleRate is the following:
  • 0 for off
  • 1 for dynamic
  • 3 for dynamic conservative (default)
  • 100-100000 for programming the adapter to send out a maximum of that many interrupts per second, even if more packets have come in
Consider setting the InterruptThrottleRate to a dynamic or static value when the default dynamic conservative mode results in inadequate performance. In most environments, setting the InterruptThrottleRate to dynamic should increase the interrupt rate and result in improved performance. However, in certain environments where even the dynamic mode does not result in an optimal interrupt moderation rate, setting the InterruptThrottleRate to a static value might be more desirable, yielding better performance.
 
To set the InterruptThrottleRate to dynamic, run the following command on the ESXi host:
 
 
vmkload_mod driver_name InterruptThrottleRate=value,value
 
For example:
 
vmkload_mod igb InterruptThrottleRate=8000,8000
 
Note: The value 1 sets the rate to dynamic. Each value represents a port. For example, the above command is for two ports. Additional values can added, separated by comas, depending on the number of available ports. For a four-port adapter, the command is:
 
vmkload_mod igb InterruptThrottleRate=8000,8000,8000,8000
 
Alternatively, use esxcfg-module to persist this parameter across ESXi reboots. For example, to set the parameter for an ESXi host with a dual-port adapter and a single-port adapter.

# esxcfg-module -s "InterruptThrottleRate=8000,8000,8000" igb

Verify the output of the command ethtool -c vmnicX.

Note: Make sure rx-usecs is set to 125 which indicates that the interrupt was changed recently.

Note:Setting InterruptThrottleRate to a static value will program the adapter to send out a maximum of that many interrupts per second, even if more packets have come in. Setting the value to a number lower than 3000 reduces the load on the system and can lower CPU utilization, but will increase latency and reduce throughput as the packets are not processed quickly. An InterruptThrottleRate value of 8000 was found to provide good performance for most traffic types, but lacked in small packet performance and latency. Consider using a larger InterruptThrottleRate value for network traffic that consists of small packets and has a requirement for extremely low latency.

For more information, see Linux* Base Driver for Intel(R) Ethernet Network Connection on the Intel Web site.

Additional Information

For translated versions of this article, see: