Understanding TCP Checksum Offloading (TCO) in a VMware Environment

Products

VMware vSphere ESXi

Issue/Introduction

The TCP header contains a 16-bit checksum field which is used to verify the integrity of the header and data. For performance reasons the checksum calculation on the transmit side and verification on the receive side may be offloaded from the operating system to the network adapter.

This article provides information on how the TCP Checksum Offload engine (TCO) works in a ESXi/ ESX virtual environment and where the offloading activity occurs. Typical commands to query, enable and disable TCO in an ESXi host login are given.

Environment

VMware ESXi 4.0.x Installable
VMware ESXi 4.1.x Embedded
VMware vSphere ESXi 5.1
VMware ESX Server 3.5.x
VMware ESXi 4.0.x Embedded
VMware ESXi 3.5.x Embedded
VMware ESX 4.1.x
VMware vSphere ESXi 5.0
VMware ESXi 4.1.x Installable
VMware ESXi 3.5.x Installable
VMware ESX 4.0.x
VMware vSphere ESXi 5.5

Resolution

Scenario 1

A virtual machine is communicating with a virtual machine on the same virtual switch on the same ESXi/ESX host.

Where is the Transmit (Tx) checksum calculation performed?

Tx vNIC
Rx vNIC	TCO Tx	Enabled	Disabled
	Enabled	Rx vNIC	Guest operating system
	Disabled	Rx vNIC	Guest operating system

Notes:

When TCO is disabled on the transmitting virtual machine virtual NIC, Tx checksum calculation is performed by the transmitting guest operating system.
When TCO is enabled on the transmitting virtual machines virtual NIC and enabled on the receiving virtual machine virtual NIC, then Tx checksum calculation is performed by the receiving virtual machines virtual NIC.
When TCO is enabled on the transmitting virtual machine virtual NIC and disabled on the receiving virtual machine virtual NIC, then Tx checksum calculation is performed by the receiving virtual machines virtual NIC but on the ESXi/ESX back-end.

Where is Receive (Rx) checksum verification performed?

Tx vNIC
Rx vNIC	TCO Rx	Enabled	Disabled
	Enabled	Rx vNIC	Rx vNIC
	Disabled	Guest operating system	Guest operating system

Notes:

When TCO is disabled on the virtual NIC of the receiving virtual machine, Rx checksum verification is always performed by the guest operating system receiving the traffic.
When TCO is enabled on the virtual NIC of the receiving virtual machine, Rx checksum verification is always performed by that virtual NIC.

Scenario 2

A virtual machine is communicating with a virtual machine on another virtual switch or another ESXi/ESX Host.

Where is Transmit (Tx) checksum calculation performed?

vNIC
Physical NIC	TCO Tx	Enabled	Disabled
	Enabled	Tx physical NIC	Guest operating system
	Disabled	VMkernel	Guest operating system

Notes:

When TCO is disabled on the virtual NIC of the transmitting virtual machine, the checksum calculation is always performed by the guest operating system regardless of whether TCO is enabled or not on the physical NIC.
When TCO is enabled on the virtual NIC and also on the physical NIC, the checksum calculation is performed on the physical NIC.
When TCO is enabled on the virtual NIC but not on the physical NIC, the checksum calculation is performed by the virtual machine kernel.

Where is Receive (Rx) checksum verification performed?

vNIC
Physical NIC	TCO Rx	Enabled	Disabled
	Enabled	Rx physical NIC	Rx physical NIC
	Disabled	Rx vNIC	Guest operating system

Notes:

When TCO is enabled on the physical NIC where the receiving virtual machine resides, checksum verification is always performed on that physical NIC regardless of virtual NIC TCO settings.
When TCO is enabled on the virtual NIC but disabled on the physical NIC, then Rx checksum verification is performed on the Rx virtual NIC back-end.
When TCO is disabled on both physical and virtual NIC the receiving guest operating system performs the Rx checksum verification.

Additional Information

Enable/Disable TCO in the VMware environment

Use the ethtool -k command to display the current state of TCO on an adapter. In a console or via SSH, log in to an ESXi host to display the TCO configuration on vmnic0:

ethtool -k vmnic0 2>&1 | egrep 'tx|rx'

You may see output similar to:

rx-checksumming: on
tx-checksumming: on

To enable TCO in an ESXi host, use the ethtool -K command and on. For example:

ethtool -K vmnic0 rx on
ethtool -K vmnic0 tx on

To disable TCO in an ESXi host, use the ethtool -K command and off. For example:

ethtool -K vmnic0 rx off
ethtool -K vmnic0 tx off

VMware 環境における TCP チェックサムオフロード (TCO) について