ESXi/ESX host loses network connectivity after adding new NICs or an upgrade
search cancel

ESXi/ESX host loses network connectivity after adding new NICs or an upgrade

book

Article ID: 324538

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:
  • The ESX\ESXi host loses connectivity after applying a BIOS, firmware, or patch upgrade
  • The ESX\ESXi host loses connectivity after adding new NICs


Environment

VMware ESXi 4.1.x Embedded
VMware vSphere ESXi 5.1
VMware ESX 4.0.x
VMware ESXi 3.5.x Installable
VMware ESXi 4.1.x Installable
VMware vSphere ESXi 7.0.0
VMware ESXi 4.0.x Embedded
VMware vSphere ESXi 5.5
VMware ESXi 3.5.x Embedded
VMware vSphere ESXi 6.0
VMware vSphere ESXi 5.0
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.5
VMware ESX 4.1.x
VMware ESXi 4.0.x Installable

Cause

This issue is caused by NIC enumeration that occurs after applying a BIOS, firmware, patch upgrade or adding new NIC’s.

Virtual switches which provide network communication to the outside world are configured based on the VMNIC names. If all of the NICs are renamed, the vSwitches route packets to interfaces that no longer exist.

The PCI ID to VMNIC numbering relationship is determined at boot time and is automatically entered into the esx.conf file located at /etc/vmware/ for persistence. The ESX/ESXi host first scans the seg number, then the bus number, the slot number, and finally the function number. This order ensures that ports on the same multi-port NIC are numbered sequentially.

When ESXi/ESX is installed, the VMNIC order is sequential. This changes over time as NICs are removed and other NICS are added. This results in VMNIC ordering that is undesirable and not in sync with the naming convention on other ESX/ESXi hosts.

To view the VMNIC assignment connect to the host with an SSH session. For more information, see Using ESXi Shell in ESXi 5.x (2004746).

Open the esx.conf file located at /etc/vmware/ with a text editor. For information, see Editing configuration files in VMware ESXi and ESX (1017022) .

The file displays similar to:

/device/000:002:00.0/vmkname = "vmnic0"
/device/000:002:02.0/vmkname = "vmnic1"
/device/000:002:03.0/vmkname = "vmnic2"
/device/000:002:04.0/vmkname = "vmnic3"

 

 

Run esxcfg-nics –l to list out the currently installed NIC’s.
You see the output similar to:

 

vmnic0 0000:02:00.00 e1000 Up 1000Mbps Full xx:xx:xx:xx:xx:xw 1500 Intel Corporation 82545EM Gigabit Ethernet Controller (Copper)
vmnic1 0000:02:02.00 e1000 Up 1000Mbps Full xx:xx:xx:xx:xx:xx 1500 Intel Corporation 82545EM Gigabit Ethernet Controller (Copper)
vmnic2 0000:02:03.00 e1000 Up 1000Mbps Full xx:xx:xx:xx:xx:xy 1500 Intel Corporation 82545EM Gigabit Ethernet Controller (Copper)
vmnic3 0000:02:04.00 e1000 Up 1000Mbps Full xx:xx:xx:xx:xx:xz 1500 Intel Corporation 82545EM Gigabit Ethernet Controller (Copper)

 

 

When a NIC device is removed from a server, esx.conf entry can only be removed at boot time. This means, to reuse a device name, boot the server, without that device cannot delete its entry and free up the device name. On the next boot, the name is once again available for allocation.

 

Scenarios that cause VMNIC numbering to change:

  • A server is shut down and a NIC is moved to another PCI slot.
  • A server is shut down and a NIC is removed and replaced with another NIC in a different PCI slot.

Scenario that will maintain VMNIC numbering order:

  • A server is shut down and a NIC replaced with another NIC in the same PCI slot.
  • A server is shut down and a NIC removed. The server is booted and shut down again. A NIC is then added in the same or a new PCI slot.

Resolution

To reorder the numbering on a host which has an undesired VMNIC numbering order on ESXi/ESX host versions 4.0 through 5.1:

  1. Place the ESXi/ESX host into maintenance mode.
  2. Open the esx.conf file located at /etc/vmware/ with a text editor. Find all lines similar to:

    /device/000:002:01.0/vmkname = "vmnic0"
    /device/000:002:03.0/vmkname = "vmnic1"


    Edit these lines to rearrange the relation between PCI addresses (numbers similar to 000:002:00.0) and vmnic names. For example, to swap vmnic0 and vmnic1 in the above example, you would change the lines to:

    /device/000:002:01.0/vmkname = "vmnic1"
    /device/000:002:03.0/vmkname = "vmnic0"


     
  3. Find all lines similar to:

    /net/pnic/child[0000]/mac = "00:0c:29:6c:4c:f8"
    /net/pnic/child[0000]/name = "vmnic0"
    /net/pnic/child[0000]/virtualMac = "00:50:56:5c:4c:f8"
    /net/pnic/child[0001]/mac = "00:0c:29:6c:4c:02"
    /net/pnic/child[0001]/name = "vmnic1"
    /net/pnic/child[0001]/virtualMac = "00:50:56:5c:4c:02"


    Edit the names here in exactly the same way you did in step 2, so that the file still has the correct relation between physical NICs and their MAC addresses. For example, if you swapped vmnic0 and vmnic1 in step 2, you would change the lines to:

    /net/pnic/child[0000]/mac = "00:0c:29:6c:4c:f8"
    /net/pnic/child[0000]/name = "vmnic1"
    /net/pnic/child[0000]/virtualMac = "00:50:56:5c:4c:f8"
    /net/pnic/child[0001]/mac = "00:0c:29:6c:4c:02"
    /net/pnic/child[0001]/name = "vmnic0"
    /net/pnic/child[0001]/virtualMac = "00:50:56:5c:4c:02"


     
  4. Find all lines similar to:

    /net/vmkernelnic/child[0000]/macFromPnic = "vmnic0"
    /net/vmkernelnic/child[0000]/mac = "00:0c:29:6c:4c:f8"


    If you changed any of the NICs that are listed in a macFromPnic line, change the corresponding mac line to have the correct MAC address for the new NIC. For example, if you swapped vmnic0 and vmnic1 in step 2, you would change the lines to:

    /net/vmkernelnic/child[0000]/macFromPnic = "vmnic0"
    /net/vmkernelnic/child[0000]/mac = "00:0c:29:6c:4c:02"


     
  5. Save the file and exit from the text editor.
  6. Shut down and reboot the ESXi/ESX host.
  7. Adjust the assignment of the renamed vmnic devices to the virtual switches if necessary.
  8. Exit maintenance mode on the ESXi/ESX host.

Note:

This article does not apply to the ESXi 5.5 and later versions. Instead, see How VMware ESXi 5.5 and 6.0 determine the order in which names are assigned to devices (2091560).


Additional Information

 
 

Why do host NICs sometimes change their ordering after an ESX installation?
Editing configuration files in VMware ESXi and ESX
Using ESXi Shell in ESXi 5.x and 6.x
How VMware ESXi determines the order in which names are assigned to devices
ESXi/ESX ホストに新たに NIC を追加するか、アップグレードすると、ネットワーク接続が切断される
在添加新网卡或升级后,ESXi/ESX 主机失去网络连接