Search the VMware Knowledge Base (KB)
View by Article ID

After upgrading a virtual machine to hardware version 11, network dependent workloads experience performance degradation (2129176)

  • 55 Ratings

Symptoms

  • You notice performance degradation for some client/servers workloads.
  • Packets have up to a 0.5 second delay over the expected arrival in the application.

    This issue is observed under these conditions:

    • The guest operating system is Windows Server 2012,  Windows 8 or later.
    • The virtual machine is on hardware version 11/ ESXi 6.0 compatibility.
    • The virtual NIC is vmxnet3 and the driver version is 1.6.6.0 and later.
    • The Receive Side Coalescing (RSC) feature is enabled globally and on the vmxnet3 adapter.
    • This issue is more prevalent when:
      • Running Microsoft SQL/TDS based workloads
      • Using Jumbo Frames
      • The client and server are on two different hosts

Cause

Depending on different physical NIC and workload characteristics like chipset, coalescing settings and packet inter arrival rate, some packets offloaded by RSC might experience an additional delay. When aggregating a number of packets, ESXi will only keep the Push Flag (PSH flag) if it was set on the first packet to be coalesced. If the first packet does not have the PSH flag set but subsequent packets do, the final coalesced packet will not have it set and as a result, might not be delivered to the application immediately.

Resolution

This issue is resolved in ESXi 6.0 Update 2, available at VMware Downloads.

Note: Applying ESXi 6.0 Update 2 may cause a Purple Diagnostic screen.  For more information, see ESXi 6.0 Update 2 host fails with a purple diagnostic screen containing the error: Vmxnet3VMKDevRxWithLock and Vmxnet3VMKDevRx (2144968). This issue is resolved in ESXi 6.0 Patch Release ESXi600-201605001.  For more information, see VMware ESXi 6.0, Patch Release ESXi600-201605001 (2136186).

If you are unable to upgrade, you can work around any performance degradation by disabling RSC on the affected guest operating system.

To disable RSC on the virtual machine, perform one of these options:
  • Use the PowerShell command Disable-NetAdapterRsc.

    To disable for all interfaces:

    Disable-NetAdapterRsc *

    To disable for a specific interface:

    Disable-NetAdapterRsc -Name Ethernetx


  • Disable RSC globally using the netsh int tcp set global rsc=disabled command.

    Note: This command requires an elevated command prompt.
To verify that RSC is disabled on the virtual machine:
  • Use the PoweShell command Get-NetAdapterRsc. The IPv4OperationalState and IPv6OperationalState should both be set to FALSE.
  • Use the netsh int tcp show global command.  The Receive Segment Coalescing State should be set to disabled.
Note: This change is immediate and does not require rebooting the guest operating system.

Alternatively, you can disable LRO/RSC for all virtual machines on an ESXi host using:

esxcli system settings advanced set -o /Net/Vmxnet3SwLRO -i 0
esxcli system settings advanced set -o /Net/Vmxnet3HwLRO -i 0

Note: This will disable LRO for all virtual machines on the ESXi host. Virtual machines must be powered off and then powered on or migrated using vMotion to another hosts for the changes to take effect.


Additional Information

To be alerted when this article is updated, click Subscribe to Document in the Actions box.

For more information on the RSC feature, see the Enable or Disable LRO on a VMXNET3 Adapter on a Windows Virtual Machine section of the Management Network Resources guide.

See Also

Update History

03/16/2016 - Added the details of the ESXi 6.0 Update 2 release, which resolves this issue.

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 55 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.

What can we do to improve this information? (4000 or fewer characters)




Please enter the Captcha code before clicking Submit.
  • 55 Ratings
Actions
KB: