Knowledge Base

The VMware Knowledge Base provides support solutions, error messages and troubleshooting guides
 
Search the VMware Knowledge Base (KB)   View by Article ID
 

Understanding TCP Segmentation Offload (TSO) and Large Receive Offload (LRO) in a VMware environment (2055140)

Details

Using TSO and LRO on physical and virtual machine NICs improves the performance of ESX/ESXi hosts by reducing the CPU overhead for TCP/IP network operations. The host can use more CPU cycles to run applications.

If TSO is enabled on the transmission path, the NIC divides larger data chunks into TCP segments. If TSO is disabled, the CPU performs segmentation for TCP/IP.

Note: TSO is referred to as LSO (Large Segment Offload or Large Send Offload) in the latest VMXNET3 driver attributes. 

LRO reassembles incoming network packets into larger buffers and transfers the resulting larger but fewer packets to the network stack of the host or virtual machine. The CPU has to process fewer packets than when LRO is disabled, which reduces its utilization for networking.

To resolve potential issues related to performance in Linux virtual machines that have LRO enabled, see Poor TCP performance might occur in Linux virtual machines with LRO enabled (1027511).

Solution

Configuring TSO

  • Enabling or disabling hardware TSO on the host

    By default, a host is configured to use hardware TSO if its NICs support it.

    To toggle hardware TSO on the ESX/ESXi host, you can use the vSphere Web Client or run a console command on the host:

    • In the vSphere Web Client, on the Manage tab for the host, click Advanced System Settings and set the Net.UseHwTSO parameter to 1 to enable TSO, or to 0 to disable TSO.

    • In an ESX/ESXi Shell on the host, set the /Net/UseHwTSO advanced parameter to 1 to enable TSO, or to 0 to disable TSO by running a console command:

      On ESX/ESXi 4.x hosts:
      esxcfg-advcfg -s 1 /Net/UseHwTSO
      esxcfg-advcfg -s 0 /Net/UseHwTSO

      On ESXi 5.0 hosts and later:
      esxcli system settings advanced set -o /Net/UseHwTSO -i 1
      esxcli system settings advanced set -o /Net/UseHwTSO -i 0


  • Determining if hardware TSO is enabled on the host

    To determine if hardware TSO is enabled:

    1. In an ESX/ESXi Shell on the host, run the command:

      On ESX/ESXi 4.x hosts:
      esxcfg-advcfg -g /Net/UseHwTSO

      On ESXi 5.0 hosts and later:
      esxcli system settings advanced list -o /Net/UseHwTSO

    2. In the command output, verify that the Int Value property is equal to 1.

  • Enabling or disabling TSO on a physical NIC

    To enable TSO on a physical NIC in the ESX/ESXi Shell, run the command:

    ethtool -K vmnicX tso on

    Where X in vmnicX represents the number of NIC ports on the host.

    Note: By default, the ethtool command is not persistent across reboots of an ESXi host. To configure the ethtool command to be persistent across reboots, add an entry to the local.sh file (located at /etc/rc.local.d/) on the ESXi host. Create a backup copy of the local.sh file prior to making any changes. For more information, see Modifying the rc.local or local.sh file in ESXi/ESX to execute commands while booting (2043564).

  • Enabling TSO in a Linux guest operating system

    To enable TSO in the guest operating system of a Linux virtual machine, run this ethtool console command:

    ethtool -K ethY tso on

    Where Y in ethY is the sequence number of the NIC in the virtual machine. For more information, see below.

  • Enabling TSO in a Windows guest operating system

    To enable TSO on a Windows virtual machine, see Enabling TSO in a Windows virtual machine (1465).

Configuring LRO

  • Enabling hardware LRO for the VMXNET2 and VMXNET3 virtual machine adapters

    Hardware LRO is active for VMXNET2 and VMXNET3 virtual machine adapters by default.

    To enable or disable hardware LRO on the physical NIC for VMXNET2 and VMXNET3 adapters of virtual machines:

    • In the vSphere Web Client, on the Manage tab for the host, click Advanced System Settings and set the Net.Vmxnet2HwLRO and Net.Vmxnet3HwLRO parameters to 1 to enable hardware LRO, or to 0 to disable hardware LRO.

    • In an ESX/ESXi Shell on the host, run the esxcfg-advcfg -s (ESX/ESXi 4.x) or esxcli system settings advanced set (ESXi 5.0 and later) command to set the /Net/Vmxnet2HwLRO and /Net/Vmxnet3HwLRO parameters to 1 to enable LRO, or to 0 to disable LRO. For example, to enable hardware LRO for VMXNET3 adapters, run the command:

      On ESX/ESXi 4.x hosts:
      esxcfg-advcfg -s 1 /Net/Vmxnet3HwLRO

      On ESXi 5.0 hosts and later:
      esxcli system settings advanced set -o /Net/Vmxnet3HwLRO -i 1


  • Determining if hardware LRO is enabled on the host

    To determine if hardware LRO is enabled for VMXNET2 or VMXNET3 adapters, use the esxcfg-advcfg -g (ESX/ESXi 4.x) or esxcli system settings advanced list (ESXi 5.0 and later) command:

    1. In an ESX/ESXi Shell on the host, run the command for the /Net/Vmxnet2HwLRO or /Net/Vmxnet3HwLRO parameters:


      On ESX/ESXi 4.x hosts:
      esxcfg-advcfg -g /Net/Vmxnet2HwLRO
      esxcfg-advcfg -g /Net/Vmxnet3HwLRO

      On ESXi 5.0 hosts and later:
      esxcli system settings advanced list -o /Net/Vmxnet2HwLRO
      esxcli system settings advanced list -o /Net/Vmxnet3HwLRO

    2. In the command output, verify that the Int Value property is equal to 1.

  • Enabling software LRO in the VMkernel for VMXNET2 and VMXNET3 adapters

    To enable software LRO in the VMkernel for VMXNET2 and VMXNET3 adapters, set these parameters in the vSphere Web Client or in the ESX/ESXi Shell:

    • In the vSphere Web Client, set the Net.Vmxnet2SwLRO and Net.Vmxnet3SwLRO parameters to 1 to enable LRO, or to 0 to disable LRO.

    • In an ESX/ESXi Shell on the host, set the /Net/Vmxnet2SwLRO and /Net/Vmxnet3SwLRO parameters to 1 to enable LRO, or to 0 to disable LRO.

  • Enabling or disabling LRO for all VMkernel adapters on a host

    To enable or disable LRO for the default TCP/IP stack on the host, that is, for VMkernel adapters:

    • In the vSphere Web Client, on the Manage tab for the host, click Advanced System Settings and set Net.TcpipDefEnabled to 1 to enable LRO or to 0 to disable it.
    • In an ESX/ESXi Shell on the host, set the /Net/TcipDefEnabled to 1 to enable LRO or to 0 to disable it. For example, to disable LRO for the TCP/IP stack on the host:

      On ESX/ESXi 4.x hosts:
      esxcfg-advcfg -s 0 /Net/TcpipDefLROEnabled

      On ESXi 5.0 hosts and later:
      esxcli system settings advanced set -o /Net/TcpipDefLROEnabled -i 0

  • Determining if LRO is enabled for the VMkernel adapters on the host

    To determine if LRO is enabled for all VMkernel adapters on the host, use the esxcfg-advcfg -g (ESX/ESXi 4.x) or esxcli system settings advanced list (ESXi 5.0 and later) command:

    On ESX/ESXi 4.x hosts:
    esxcfg-advcfg -g /Net/TcpipDefLROEnabled

    On ESXi 5.0 hosts and later:
    esxcli system settings advanced list -o /Net/TcpipDefLROEnabled

  • Configuring LRO buffer size

    To configure the size of the LRO buffer in bytes, set the Net.VmxnetLROMaxLength parameter in the vSphere Web Client or the /Net/VmxnetLROMaxLength parameter in the esxcli system settings advanced set console command on the host.

  • Enabling or disabling LRO in a Linux guest operating system

    • To enable or disable LRO on a virtual machine that runs Linux kernel 2.6.24 and later and uses a VMXNET3 network interface, run this console command in the guest operating system:

      • To enable LRO:

        ethtool -K ethY lro on

      • To disable LRO:

        ethtool -K ethY lro off

      Where Y in ethY represents the sequence number of the NIC in the virtual machine.

    • For a virtual machine that runs an earlier version of the Linux kernel or uses a VMXNET2 network interface, run these console commands to enable or disable LRO:

      • To enable LRO:

        modprobe vmxnet3 disable_lro=0
        modprobe vmxnet disable_lro=0


      • To disable LRO:

        modprobe vmxnet3 disable_lro=1
        modprobe vmxnet disable_lro=1

    • To permanently disable LRO on these versions of Linux:

      1. Open the /etc/rc.local file in a text editor and add these two lines to the file for VMXNET or VMXNET3 driver:

        rmmod vmxnet
        modprobe vmxnet disable_lro=1


        Or

        rmmod vmxnet3
        modprobe vmxnet3 disable_lro=1


      2. Save the configuration file.

      3. Run both commands from the command line or reboot the virtual machine for the changes to take effect.

        Note: This is the appropriate way to disable LRO in RHEL 5.x.

  • Enabling or disabling LRO in a Windows guest operating system

    To enable or disable LRO in a Windows Server 2012 and later or Windows 8 and later virtual machine, see Enable or Disable LRO on a VMXNET3 Adapter on a Windows Virtual Machine and Enable LRO Globally on a Windows Virtual Machine.

    Note: On Windows, the LRO technology is also referred to as Receive Side Coalescing (RSC).

Segmentation and aggregation

  • Situation 1: Virtual machine communicating with a virtual machine on another hosts

    Location of TCP packet segmentation:

    Tx TSO Status TCP Packet Segmentation Location
    Enabled Transmitting physical NIC
    Disabled Guest operating system

    Notes:
    • If the physical NIC supports TSO and TSO is enabled, the NIC performs segmentation of application packets to TSO size. If the physical NIC does not support TSO, the VMkernel segments the application packets.
    • If TSO is disabled, the guest operating system of the transmitting virtual machine segments application data into packets with MTU size.

    Location of TCP packet aggregation:

    Rx LRO Status TCP Packet Aggregation Location
    Enabled Receiving physical NIC if it supports LRO
    Receiving virtual machine NIC backend if the physical NIC does not support LRO
    Disabled TCP packets are not aggregation

    Notes:
    • If the physical NIC on the receiving side supports hardware LRO, the NIC aggregates the arriving packets. If the NIC is not capable of LRO, the virtual machine NIC performs software aggregation.
    • If LRO is disabled for the receiving virtual machine, but hardware LRO is active for the network adapter type (VMXNET2 or VMXNET3), the receiving physical NIC aggregates the packets. The VMkernel on the receiving side then segments the aggregated packets into smaller chunks with TSO size, and forwards the chunks to the target virtual machine.

  • Situation 2: Virtual machine communicating with a virtual machine on the same host and virtual switch

    Note: TSO/LRO processing for traffic between two virtual machines on the same virtual switch is done by the VMkernel. Because the traffic never leaves the hypervisor kernel, and because hypervisor networking is very efficient, performance does not suffer over physical TSO/LRO.

    Location of TCP packet segmentation:

    Tx TSO Status TCP Segmentation Location
    Enabled Transmitting virtual machine NIC sends TSO size packets
    Disabled Guest operating system

    Notes:
    • If TSO is enabled for the transmitting virtual machine, the virtual machine NIC segments application data to packets that have TSO size.
    • If TSO is disabled for the transmitting virtual machine, the guest operating system segments application data to packets that have MTU size.

    Location of TCP packet aggregation:

    Rx LRO Status TCP Packet Aggregation Location
    Enabled Receiving virtual machine NIC
    Disabled TCP packets are not aggregated

    Notes:
    • If LRO is enabled for the receiving virtual machine, the virtual machine NIC aggregates application packets up to the configured buffer size, and sends the aggregated packets to the virtual machine.
    • If LRO is disabled, packets that have MTU size are directly transferred to the target virtual machine. If large packets arrive, the VMkernel segments transferred packets to TSO size and forwards them to the virtual machine.

Additional Information

For translated versions of this article, see:

Keywords

LRO, RSC, TSO, Large Receive Offload, TCP Segmentation Offload

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 12 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.
What can we do to improve this information? (4000 or fewer characters)
  • 12 Ratings
Actions
KB: