Troubleshoot duplicate IP without shutting down known device

Products

VMware vSphere ESXi

Issue/Introduction

This article is intended to assist with how to find duplicate devices without shutting down the known source of the IP address that is being duplicated.

Symptoms:

We often find duplicate IP addresses by shutting down the device we know has that duplicated IP address. However, sometimes shutting down the known device can be impractical due to production needs or other factors.

Resolution

Please note that any MAC addresses or UUIDs for any devices listed below are randomly generated and so not known to be associated with any actual devices:

Finding the MAC address of the device with duplicate IP:

On the ESXi host reporting the duplicate IP address, examine vmkernel.log to find the MAC address of the device running the duplicate IP, per Troubleshooting duplicate Mac or IP addresses in VMware vCenter Server. That KB assumes the duplicate IP is found on another ESXi host. However, it may be on a different device than another ESXi host - and may even be on the same ESXi host reporting the conflict.
Alternatively, on that same affected ESXi host, examine the hostd.log to find the MAC address of the device running the duplicate IP. In the following example, the hostd.log is showing the duplicate IP message highlighted in green, and the MAC address of the other device with the IP is highlighted in yellow:

2024-01-07T00:21:06.712Z info hostd[2111961] [Originator@6876 sub=Hostsvc.VmkVprobSource] VmkVprobSource::Post event: (vim.event.EventEx) {
--> key = 218,
--> chainId = -1,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> host = (vim.event.HostEventArgument) {
--> name = "MyVM",
--> host = 'vim.HostSystem:ha-host'
--> },
--> eventTypeId = "esx.problem.net.vmknic.ip.duplicate",
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> key = "1",
--> value = "10.121.82.189"
--> },
--> (vmodl.KeyAnyValue) {
--> key = "2",
--> value = "vmk2"
--> },
--> (vmodl.KeyAnyValue) {
--> key = "3",
--> value = "b4:f7:c8:56:07:d8"
--> }
--> ],
--> objectId = "ha-host",
--> objectType = "vim.HostSystem",
--> }

Rule out the device with the MAC address being on the same ESXi host:

Use ESXi host commands to ensure the IP is not duplicated on the same host that is giving the error. Run the following commands on the affected ESXi host console command line:
1. To see if vmks on the host are using that address:

# esxcfg-vmknic -l |grep "b4:f7:c8:56:07:d8"

To see if vmnics on the host are using that address:

# /usr/lib/vmware/vm-support/bin/nicinfo.sh |grep "b4:f7:c8:56:07:d8"

We can also run the following especially to see virtual switch and VM MAC addresses:

# python /usr/lib/vmware/vm-support/bin/dump-cswitch-info.py |grep "b4:f7:c8:56:07:d8"

Once the MAC address has been discovered, and you have found the device with that it either exists on the current ESXi host, or on another device, one or the other devices using that IP address must be changed.

The process for changing the IP for a device varies depending on what sort of device it is, and the methods are too numerous to list. For any vSphere-related device, please search https://docs.vmware.com/ or https://kb.vmware.com/s/ for the correct method for the device.

For any non-vSphere device, please consult the device vendor for direction as to how to change the IP.

Additional Information

Impact/Risks:
All commands listed in this article are informational only and are not expected to be capable of having any impact on the vSphere system or any of its components.