After upgrading to ESXi 7.0 U2a, the Communication Channels show Down in the Host Preparation tab for NSX Data Center for vSphere 6.4.x
search cancel

After upgrading to ESXi 7.0 U2a, the Communication Channels show Down in the Host Preparation tab for NSX Data Center for vSphere 6.4.x

book

Article ID: 318520

calendar_today

Updated On:

Products

VMware NSX Networking VMware vSphere ESXi

Issue/Introduction

Symptoms:
  • ESXi is at 7.0 U2a
  • Under Network and Security > Installation and Upgrade > Host Preparation, the hosts show their Communication Channels as Down
  • Running esxcli network ip connection list | grep 1234 shows no connections to the NSX Controllers (or fewer than there should be)
  • Running esxcli network ip connection list | head -n2 && esxcli network ip connection list | grep 5671 shows connections to the NSX Manager, with one connection having a large value in the Recv Q field
  • The netcpa daemon is having trouble connecting to vsfwd on port 15002:
$ grep rmqClient /var/run/log/netcpa.log
2021-05-05T17:04:00UTC rmqClient Send returned -1 Err No 32, While Sending Len 66
2021-05-05T17:04:00UTC rmqClient Error Sending Data on Idx 19 Status -1005
2021-05-05T17:04:00UTC rmqClient Unable to Send Data for Id 19 , error 32, ret -1005
2021-05-05T17:04:05UTC rmqClient Send returned -1 Err No 32, While Sending Len 66
2021-05-05T17:04:05UTC rmqClient Error Sending Data on Idx 19 Status -1005
2021-05-05T17:04:05UTC rmqClient Unable to Send Data for Id 19 , error 32, ret -1005
2021-05-05T17:04:10UTC rmqClient Send returned -1 Err No 32, While Sending Len 66
2021-05-05T17:04:10UTC rmqClient Error Sending Data on Idx 19 Status -1005
2021-05-05T17:04:10UTC rmqClient Unable to Send Data for Id 19 , error 32, ret -1005
2021-05-05T17:04:15UTC rmqClient Send returned -1 Err No 32, While Sending Len 66
2021-05-05T17:04:15UTC rmqClient Error Sending Data on Idx 19 Status -1005
2021-05-05T17:04:15UTC rmqClient Unable to Send Data for Id 19 , error 32, ret -1005
2021-05-05T17:04:20UTC rmqClient Send returned -1 Err No 32, While Sending Len 66
2021-05-05T17:04:20UTC rmqClient Error Sending Data on Idx 19 Status -1005
2021-05-05T17:04:20UTC rmqClient Unable to Send Data for Id 19 , error 32, ret -1005
2021-05-05T17:04:25UTC rmqClient Send returned -1 Err No 32, While Sending Len 66
2021-05-05T17:04:25UTC rmqClient Error Sending Data on Idx 19 Status -1005
2021-05-05T17:04:25UTC rmqClient Unable to Send Data for Id 19 , error 32, ret -1005
2021-05-05T17:04:30UTC rmqClient Send returned -1 Err No 32, While Sending Len 66
2021-05-05T17:04:30UTC rmqClient Error Sending Data on Idx 19 Status -1005
2021-05-05T17:04:30UTC rmqClient Unable to Send Data for Id 19 , error 32, ret -1005
2021-05-05T17:04:35UTC rmqClient Connect Returned  -1
2021-05-05T17:04:40UTC rmqClient Connect Returned  -1

$ esxcli network ip connection list | grep 15002
tcp        66       0  127.0.0.1:15002   127.0.0.1:25572     ESTABLISHED   2488670  newreno  vsfwd
tcp         0       0  127.0.0.1:25572   127.0.0.1:15002     ESTABLISHED   2465681  newreno  netcpa
tcp        79       0  127.0.0.1:15002   127.0.0.1:24691     CLOSE_WAIT    2488670  newreno  vsfwd
tcp         0       0  127.0.0.1:24691   127.0.0.1:15002     FIN_WAIT_2    2488678  newreno
tcp         0       0  127.0.0.1:15002   127.0.0.1:31347     CLOSE_WAIT    2488670  newreno  vsfwd
tcp         0       0  127.0.0.1:31347   127.0.0.1:15002     FIN_WAIT_2    2465681  newreno  netcpa
tcp         0       0  127.0.0.1:15002   0.0.0.0:0           LISTEN        2488670  newreno  vsfwd


Environment

VMware vSphere ESXi 7.0.x

Cause

This is caused by the vsfwd process coming into a deadlock situation.
The cause of the deadlock is a child process trying to save the password never returned from a system call.

Resolution

Currently there is no resolution for VMware ESXi 7.0u2a. 

Please upgrade to VMware ESXi 7.0u2c or later to prevent this issue.

Workaround:
The current workaround is to re-install ESXi.

Powered by Wolken Software