SVMotion progress gets stuck at 23% and then times out with error: Cannot connect Host
Article ID: 316409
Updated On:
Products
VMware vSphere ESXi
Issue/Introduction
- Perform Storage vMotion from one datastore in a cluster to another datastore in a different cluster within the same vCenter
Symptoms:
- SVMotion Migration task times out with Error: Cannot Connect Host.
- Observation : The VMs that are failing to SVMotion have snapshots associated to them. the VMs with no snapshots were able to migrate.
Environment
VMware vSphere 6.x
Cause
vpxa.log
"Failed to connect to server" i.e. destination server
"Failed to connect to server" i.e. destination server
2019-10-17T08:15:33.967Z warning vpxa[FFCAEAE0] [Originator@6876 sub=Libs opID=k1kc4bzp-ac2q-h5:70030089-fb-01-bb] [NFC ERROR] NfcNewAuthdConnectionEx: Failed to connect to peer. Error: Failed to connect to server 172.17.2.x:902
2019-10-17T08:15:33.967Z error vpxa[FFCAEAE0] [Originator@6876 sub=NfcManager opID=k1kc4bzp-ac2q-h5:70030089-fb-01-bb] [NfcClient] Unable to connect to NFC server: Failed to connect to server 172.17.2.x:902
2019-10-17T08:15:33.967Z error vpxa[FFCAEAE0] [Originator@6876 sub=NfcManager opID=k1kc4bzp-ac2q-h5:70030089-fb-01-bb] [NfcWorker] Error encountered while opening clients for copy spec:
--> N3Vim5Fault16HostConnectFault9ExceptionE(vim.fault.HostConnectFault)
2019-10-17T08:15:33.967Z error vpxa[FFCAEAE0] [Originator@6876 sub=NfcManager opID=k1kc4bzp-ac2q-h5:70030089-fb-01-bb] [NfcManagerImpl] Copy operation failed with error: vim.fault.HostConnectFault
2019-10-17T08:15:33.967Z verbose vpxa[FFCAEAE0] [Originator@6876 sub=PropertyProvider opID=k1kc4bzp-ac2q-h5:70030089-fb-01-bb] RecordOp ASSIGN: info.state, task-3432. Applied change to temp map.
2019-10-17T08:15:33.968Z verbose vpxa[FFCAEAE0] [Originator@6876 sub=PropertyProvider opID=k1kc4bzp-ac2q-h5:70030089-fb-01-bb] RecordOp ASSIGN: info.cancelable, task-3432. Applied change to temp map.
2019-10-17T08:15:33.968Z verbose vpxa[FFCAEAE0] [Originator@6876 sub=PropertyProvider opID=k1kc4bzp-ac2q-h5:70030089-fb-01-bb] RecordOp ASSIGN: info.error, task-3432. Applied change to temp map.
2019-10-17T08:15:33.968Z verbose vpxa[FFCAEAE0] [Originator@6876 sub=PropertyProvider opID=k1kc4bzp-ac2q-h5:70030089-fb-01-bb] [CommitChangesAndNotify] Updating cached values
2019-10-17T08:15:33.968Z info vpxa[FFCAEAE0] [Originator@6876 sub=vpxLro opID=k1kc4bzp-ac2q-h5:70030089-fb-01-bb] [VpxLRO] -- FINISH task-3432
2019-10-17T08:15:33.968Z info vpxa[FFCAEAE0] [Originator@6876 sub=Default opID=k1kc4bzp-ac2q-h5:70030089-fb-01-bb] [VpxLRO] -- ERROR task-3432 -- nfcManager -- nfc.NfcManager.copy: vim.fault.HostConnectFault:
nfcd.log:
Noted the following events in nfcd.log in continuous manner
2019-10-16T08:18:57Z nfcd[34831]: Failed to switch netstack to the vSphereProvisioning stack. Using default management network stack.
2019-10-17T08:15:33.967Z error vpxa[FFCAEAE0] [Originator@6876 sub=NfcManager opID=k1kc4bzp-ac2q-h5:70030089-fb-01-bb] [NfcClient] Unable to connect to NFC server: Failed to connect to server 172.17.2.x:902
2019-10-17T08:15:33.967Z error vpxa[FFCAEAE0] [Originator@6876 sub=NfcManager opID=k1kc4bzp-ac2q-h5:70030089-fb-01-bb] [NfcWorker] Error encountered while opening clients for copy spec:
--> N3Vim5Fault16HostConnectFault9ExceptionE(vim.fault.HostConnectFault)
2019-10-17T08:15:33.967Z error vpxa[FFCAEAE0] [Originator@6876 sub=NfcManager opID=k1kc4bzp-ac2q-h5:70030089-fb-01-bb] [NfcManagerImpl] Copy operation failed with error: vim.fault.HostConnectFault
2019-10-17T08:15:33.967Z verbose vpxa[FFCAEAE0] [Originator@6876 sub=PropertyProvider opID=k1kc4bzp-ac2q-h5:70030089-fb-01-bb] RecordOp ASSIGN: info.state, task-3432. Applied change to temp map.
2019-10-17T08:15:33.968Z verbose vpxa[FFCAEAE0] [Originator@6876 sub=PropertyProvider opID=k1kc4bzp-ac2q-h5:70030089-fb-01-bb] RecordOp ASSIGN: info.cancelable, task-3432. Applied change to temp map.
2019-10-17T08:15:33.968Z verbose vpxa[FFCAEAE0] [Originator@6876 sub=PropertyProvider opID=k1kc4bzp-ac2q-h5:70030089-fb-01-bb] RecordOp ASSIGN: info.error, task-3432. Applied change to temp map.
2019-10-17T08:15:33.968Z verbose vpxa[FFCAEAE0] [Originator@6876 sub=PropertyProvider opID=k1kc4bzp-ac2q-h5:70030089-fb-01-bb] [CommitChangesAndNotify] Updating cached values
2019-10-17T08:15:33.968Z info vpxa[FFCAEAE0] [Originator@6876 sub=vpxLro opID=k1kc4bzp-ac2q-h5:70030089-fb-01-bb] [VpxLRO] -- FINISH task-3432
2019-10-17T08:15:33.968Z info vpxa[FFCAEAE0] [Originator@6876 sub=Default opID=k1kc4bzp-ac2q-h5:70030089-fb-01-bb] [VpxLRO] -- ERROR task-3432 -- nfcManager -- nfc.NfcManager.copy: vim.fault.HostConnectFault:
nfcd.log:
Noted the following events in nfcd.log in continuous manner
2019-10-16T08:18:57Z nfcd[34831]: Failed to switch netstack to the vSphereProvisioning stack. Using default management network stack.
Resolution
- Enable vSphere provisioning on vmkernel ports used for vMotion on both source and destination server.
- Go to Host > Configure > VMkernel Adapters under Networking > Select the vmkernel port used for vMotion > Edit
Additional Information
- Please refer to Vmware Document page :
https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.networking.doc/GUID-D4191320-209E-4CB5-A709-C8741E713348.html
Supports the traffic for virtual machine cold migration, cloning, and snapshot migration. You can use the provisioning TCP/IP to handle Network File Copy (NFC) traffic during long-distance vMotion. NFC provides a file-specific FTP service for vSphere. ESXi uses NFC for copying and moving data between datastores. VMkernel adapters configured with the provisioning TCP/IP stack handle the traffic from cloning the virtual disks of the migrated virtual machines in long-distance vMotion. By using the provisioning TCP/IP stack, you can isolate the traffic from the cloning operations on a separate gateway. After you configure a VMkernel adapter with the provisioning TCP/IP stack, all adapters on the default TCP/IP stack are disabled for the Provisioning traffic.
https://docs.vmware.com/en/VMware-vSphere/6.0/com.vmware.vsphere.vcenterhost.doc/GUID-AC418F24-8E7D-40B6-9D07-A9DB504C7F1B.html
Use the provisioning TCP/IP stack to isolate traffic for cold migration, VM clones, and snapshots, and to assign a dedicated default gateway, routing table, and DNS configuration for this traffic. To enable the Provisioning TCP/IP stack, assign it a new VMkernel adapter.
By using a separate TCP/IP stack, you can handle vMotion and cold migration traffic according to the topology of the network and as required for your organization:
Supports the traffic for virtual machine cold migration, cloning, and snapshot migration. You can use the provisioning TCP/IP to handle Network File Copy (NFC) traffic during long-distance vMotion. NFC provides a file-specific FTP service for vSphere. ESXi uses NFC for copying and moving data between datastores. VMkernel adapters configured with the provisioning TCP/IP stack handle the traffic from cloning the virtual disks of the migrated virtual machines in long-distance vMotion. By using the provisioning TCP/IP stack, you can isolate the traffic from the cloning operations on a separate gateway. After you configure a VMkernel adapter with the provisioning TCP/IP stack, all adapters on the default TCP/IP stack are disabled for the Provisioning traffic.
https://docs.vmware.com/en/VMware-vSphere/6.0/com.vmware.vsphere.vcenterhost.doc/GUID-AC418F24-8E7D-40B6-9D07-A9DB504C7F1B.html
Use the provisioning TCP/IP stack to isolate traffic for cold migration, VM clones, and snapshots, and to assign a dedicated default gateway, routing table, and DNS configuration for this traffic. To enable the Provisioning TCP/IP stack, assign it a new VMkernel adapter.
By using a separate TCP/IP stack, you can handle vMotion and cold migration traffic according to the topology of the network and as required for your organization:
- Route the traffic for migration of powered on or powered off virtual machines by using a default gateway that is different from the gateway assigned to the default stack on the host.
- By using a separate default gateway, you can use DHCP for IP address assignment to the VMkernel adapters for migration in a flexible way.
- Assign a separate set of buffers and sockets.
- Avoid routing table conflicts that might otherwise appear when many features are using a common TCP/IP stack.
- Isolate traffic to improve security.
Impact/Risks:
NA
Feedback
Yes
No
Powered by
