Troubleshooting vMotion fails with network errors

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

You see errors similar to:

Example 1:

The VMotion failed because the ESX hosts were not able to connect over the VMotion network. Please check your VMotion network settings and physical network configuration.
VMotion [184329483:1276605211167987] failed to create connection with remote host <10.252.164.14>: The ESX hosts failed to connect over the VMotion network
Migration [184329483:1276605211167987] failed to connect to remote host <10.252.164.14>: Timeout

Example 2:

The VMotion failed because the ESX hosts were not able to connect over the VMotion network. Please check your VMotion network settings and physical network configuration.
VMotion [-1408237366:1279683851917265] failed to create connection with remote host <172.16.1.204>: The ESX hosts failed to connect over the VMotion network
Migration [-1408237366:1279683851917265] failed to connect to remote host <172.16.1.204>: Connection refused
In the /var/log/hostd.log file, you see entries similar to:

[27703B90 verbose 'Vmsvc.vm:/vmfs/volumes/5538de90-249141cc-a57c-40f2e9638530/TEMP-TEMP/TEMP-TEMP.vmx'] Handling message _vmx1: Migration [a991333:1451374787315531] failed to connect to remote host from host : Timeout

Environment

VMware vSphere ESXi 5.5
VMware vSphere ESXi 5.1

Resolution

To resolve this issue:

Check for IP address conflicts on the vMotion network.

Note: Each host in the cluster should have a vMotion vmkernel, assigned a unique IP address.
Check for packet loss over the vMotion network. Try having the source host ping (vmkping) the destination host's vMotion vmknic IP address for the duration of the vMotion.

For example:

vmkping -I vmkX <IP of destination host vMotion vmk>

Where vmkX is the vmkernel number of the vMotion vmkernel on the source host.

For more information, see Testing VMkernel network connectivity with the vmkping command (1003728).
Check the driver/firmware of the physical adapter used for vMotion.

Note: If the driver or firmware is not up to date, the vMotions can fail with a "timeout" or "waiting for data" error. For more information, see Determining Network/Storage firmware and driver version in ESXi (1027206).
Ensure the ESXi firewall ruleset for vMotion is enabled and that there are no other conflicting rules blocking the traffic.

Note: For information on configuring the ESXi firewall, see the VMware documentation page ESXi Firewall Configuration

Check for potential interaction with firewall hardware or software that prevents connectivity between the source and the destination TCP port 8000.

Additional Information

For more information on vMotions and troubleshooting vMotion issues, see Understanding and troubleshooting vMotion (1003734).

Other examples:

Migration [184329483:1276605211167987] failed to connect to remote host <10.252.164.14>: Timeout.

This error indicates that remote host did not accept the connection within the allowed time limit.
Migration [-1408237366:1279683851917265] failed to connect to remote host <172.16.1.204>: Connection refused.

This error indicates that the remote host is explicitly not listening on the vMotion port.

Additional Articles

Impact/Risks:
Network misconfiguration can cause random vMotion failures. Retrying the vMotion operation may be successful, but VMware recommends that you follow this article to isolate and correct the problem.