After upgrading a virtual machine to hardware version 11, network dependent workloads experience performance degradation
search cancel

After upgrading a virtual machine to hardware version 11, network dependent workloads experience performance degradation

book

Article ID: 317984

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:
  • You notice performance degradation for some client/servers workloads.
  • Packets have up to a 0.5 second delay over the expected arrival in the application.

    This issue is observed under these conditions:
     
    • The guest operating system is Windows Server 2012, Windows 8 or later.
    • The virtual machine is on hardware version 11/ ESXi 6.0 compatibility.
    • The virtual NIC is vmxnet3 and the driver version is 1.6.6.0 and later.
    • The Receive Side Coalescing (RSC) feature is enabled globally and on the vmxnet3 adapter.
    • This issue is more prevalent when:
      • Running Microsoft SQL/TDS based workloads
      • Using Jumbo Frames
      • The client and server are on two different hosts


Environment

VMware vSphere ESXi 6.0

Cause

Depending on different physical NIC and workload characteristics like chipset, coalescing settings and packet inter arrival rate, some packets offloaded by RSC might experience an additional delay. When aggregating a number of packets, ESXi will only keep the Push Flag (PSH flag) if it was set on the first packet to be coalesced. If the first packet does not have the PSH flag set but subsequent packets do, the final coalesced packet will not have it set and as a result, might not be delivered to the application immediately.

Resolution

This issue is resolved in ESXi 6.0 Update 2, available at VMware Downloads.
 
Note: Applying ESXi 6.0 Update 2 may cause a Purple Diagnostic screen. For more information, see ESXi 6.0 Update 2 host fails with a purple diagnostic screen containing the error: Vmxnet3VMKDevRxWithLock and Vmxnet3VMKDevRx (2144968). This issue is resolved in ESXi 6.0 Patch Release ESXi600-201605001. For more information, see VMware ESXi 6.0, Patch Release ESXi600-201605001 (2136186).
 
If you are unable to upgrade, you can work around any performance degradation by disabling RSC on the affected guest operating system.

To disable RSC on the virtual machine, perform one of these options:
  • Use the PowerShell command Disable-NetAdapterRsc.

    To disable for all interfaces:

    Disable-NetAdapterRsc *

    To disable for a specific interface:

    Disable-NetAdapterRsc -Name Ethernetx
     
  • Disable RSC globally using the netsh int tcp set global rsc=disabled command.

    Note: This command requires an elevated command prompt.
To verify that RSC is disabled on the virtual machine:
  • Use the PoweShell command Get-NetAdapterRsc. The IPv4OperationalState and IPv6OperationalState should both be set to FALSE.
  • Use the netsh int tcp show global command. The Receive Segment Coalescing State should be set to disabled.
Note: This change is immediate and does not require rebooting the guest operating system.
 
Alternatively, you can disable LRO/RSC for all virtual machines on an ESXi host using:
 
esxcli system settings advanced set -o /Net/Vmxnet3SwLRO -i 0
esxcli system settings advanced set -o /Net/Vmxnet3HwLRO -i 0
 
Note: This will disable LRO for all virtual machines on the ESXi host. Virtual machines must be powered off and then powered on or migrated using vMotion to another hosts for the changes to take effect.

 


Additional Information

To be alerted when this document is updated, click the Subscribe to Article link in the Actions box..
 
For more information on the RSC feature, see the Enable or Disable LRO on a VMXNET3 Adapter on a Windows Virtual Machine section of the Management Network Resources guide.VMware ESXi 6.0, Patch Release ESXi600-201605001
ESXi 6.0 Update 2 host fails with a PSOD error mentioning Vmxnet3VMKDevRxWithLock
仮想マシンをハードウェア バージョン 11 にアップグレードすると、ネットワークに依存するワークロードでパフォーマンスが低下する
将虚拟机升级到硬件版本 11 后，与网络相关的工作负载出现性能降级

Powered by Wolken Software