Hosts with virtual machines using the E1000/E1000E adapter have networking issues after upgrading to ESXi 5.1 U2
search cancel

Hosts with virtual machines using the E1000/E1000E adapter have networking issues after upgrading to ESXi 5.1 U2

book

Article ID: 301961

calendar_today

Updated On:

Products

VMware

Issue/Introduction

Symptoms:
  • After upgrading a host to ESXi 5.1 Update 2, virtual machines using the E1000/E1000E virtual adapter stop passing network traffic intermittently leading to the host being non-responsive
  • After powering up a virtual machine, the network adapter remains disconnected
  • While using vCloud Director, virtual machines spontaneously disconnect
  • When reconnecting a disconnected virtual machine, the other powered on and running virtual machines on the host disconnect
  • In the vmkernel.log file (located in /var/log) of the affected ESXi 5.1 U2 host, you see messages similar to:

    WARNING: Heap: 2677: Heap netGPHeap already at its maximum size. Cannot expand.
    WARNING: Heap: 3058: Heap_Align(netGPHeap, 35156/35156 bytes, 8 align) failed. caller: 0x41800915066c

    WARNING: E1000: 8817: failed to enable port 0x200002f on DvsPortset-0: Out of memory
    WARNING: Net: vm 367615: 4454: cannot enable port 0x200002f: Out of memory


  • In the vmware.log file on the affected virtual machine, you see entries similar to:

    vcpu-0| I120: VMM initialized.
    vcpu-0| W110: Restoring VMM
    vcpu-0| I120: Msg_Post: Error
    vcpu-0| I120: [msg.ethernet.e1000.openFailed] Failed to connect ethernet0.
    vcpu-0| I120: ----------------------------------------
    vcpu-0| I120: SVGA: Registering MemSpace at 0xd8000000(0xd8000000) and 0xd0800000(0x
    d0800000)

  • In the hostd.log file (located in /var/log) of the affected ESXi 5.1 U2 host, you see entries similar to:

    [7F7C#### verbose 'Vmsvc.vm:/vmfs/volumes/4f4eb4eb-xxxxxxxx-xxxx-0017a4770020/VM/VM.vmx'] Handling message _vmx5: Failed to connect virtual device Ethernet0.

  • In the hostd.log file, you see entries similar to:

    PktYYYY-MM-DDT<time>.905Z cpu24:12476)WARNING: Heap: 3058: Heap_Align(netGPHeap, 35156/35156 bytes, 8 align) failed. caller: 0x41800e15066c<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth PktYYYY-MM-DDT<time>.979Z cpu25:12478)WARNING: Heap: 3058: Heap_Align(netGPHeap, 35156/35156 bytes, 8 align) failed. caller: 0x41800e15066c<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth
    Pkt2014-03-05T10:34:19.058Z cpu14:9175)WARNING: Heap: 3058: Heap_Align(netGPHeap,35156/35156 bytes, 8 align) failed. caller: 0x41800e15066c<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Et</time></time>


    Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
Note: The issue described in this article is specific to hosts running ESXi 5.1 Update 2. If you are running an older version of ESXi, your issue is not related to this article. Search for your symptoms in the Knowledge Base or file a Support Request. For more information, see Filing a Support Request in Customer Connect (2006985).

Note: You may also get a similar error from a Nexus 1000v or VMware VDS even if you just list the vswitch(s) with the esxcfg-vswitch -l command. For more information, see Error message is displayed when a large number of dvPorts are in use in VMware ESXi 5.1.x (2034073).


Cause

This issue occurs when there is a memory leak in the E1000 VMkernel back end in ESXi 5.1 Update 2.

Resolution

This issue is resolved in VMware ESXi 5.1 Patch Release ESXi510-201402001 (Build 1612806). You can download the patch at http://www.vmware.com/patch/download/.

For more information on the patch, see VMware ESXi 5.1, Patch Release ESXi510-201402001 (2072652).

To work around this issue when you are unable to apply the patch, use one of these methods:
  • Disable the TSO Zero Copy and TxZero Copy offloading features:

    Notes:
    • This procedure does not reclaim memory lost to the leak. To recover lost memory, reboot the host after applying this workaround.
    • In most use-case scenarios, this procedure will have a minimal effect on performance.

    1. If the affected ESXi host is non-responsive, reboot the host.

    2. Log in to the ESXi host as the root user via SSH. For more information on enabling SSH, see Using Tech Support Mode in ESXi 4.1 and ESXi 5.x (1017910).

    3. Disable the TSO Zero Copy and TxZero Copy offloading features by running these commands:

      esxcli system settings advanced set -i 0 -o /Net/E1000TxTsoZeroCopy
      esxcli system settings advanced set -i 0 -o /Net/E1000TxZeroCopy


    4. Disconnect, then reconnect the NIC to allow the changes take effect:
      1. Right-click the affected virtual machine and click Edit Settings.
      2. Click the virtual NIC and deselect Connected.
      3. Click OK.
      4. Select Connected again and click OK.

  • Use a different virtual NIC, such as VMXNET3. For more information, see Network Virtual Machine Configuration in the vSphere 5.1 documentation.

    Note: This procedure does not reclaim memory lost to the leak. To recover lost memory, reboot the host after applying this workaround.


Additional Information



TCP and UDP Ports required to access VMware vCenter Server, VMware ESXi and ESX hosts, and other network components
Using Tech Support Mode in ESXi 4.1, ESXi 5.x, and ESXi 6.x
How to file a Support Request in Customer Connect
Error message is displayed when a large number of dvPorts are in use in VMware ESXi 5.1.x
VMware ESXi 5.1, Patch Release ESXi510-201402001
E1000/E1000E アダプタを使用する仮想マシンのあるホストを ESXi 5.1 U2 にアップグレードするとネットワーク上の問題が発生する
所含虚拟机使用 E1000/E1000E 适配器的主机在升级到 ESXi 5.1 U2 之后存在网络连接问题