-- Looking at the /var/log/nsxproxy.log in the ESXi host, log messages similar to below will be seen, which show the host lost connectivity to its master controller
2018-12-01T05:23:32Z nsx-proxy[69012]: [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-net" tid="69018" level="INFO"] StreamConnection[80 Closing to ssl://10.208.214.45:1235 sid:80] Closing (reason: Remotely)
2018-12-01T05:23:32Z nsx-proxy[69012]: [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-net" tid="69018" level="INFO"] StreamConnection[80 Closed to ssl://10.208.214.45:1235 sid:-1] Closed (reason: Remotely, error: 2-End of file)
2018-12-01T05:23:32Z nsx-proxy[69012]: [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-rpc" tid="69018" level="INFO"] RpcConnection[80 Connected to ssl://10.208.214.45:1235] Closing (network error)
2018-12-01T05:23:32Z nsx-proxy[69012]: [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-rpc" tid="69018" level="INFO"] RpcTransport[2] Unregistering remote endpoint ids '918c4331-98cd-4945-8322-ae92390a26f1' @ RpcConnection[80 Closed to ssl://10.208.214.45:1235]
2018-12-01T05:23:32Z nsx-proxy[69012]: [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-rpc" tid="69018" level="INFO"] RpcTransport[2]::RemoteServiceManager Unregistering remote service 'vmware.nsx.h
andshake.VersionMastershipHandshake' @ RpcConnection[80 Closed to ssl://10.208.214.45:1235]
2018-12-01T05:23:32Z nsx-proxy[69012]: [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-rpc" tid="69018" level="INFO"] RpcConnection[80 Closed to ssl://10.208.214.45:1235] Notifying channels on connection down (network error)
2018-12-01T05:23:32Z nsx-proxy[69012]: [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-rpc" tid="69029" level="ERROR" errorCode="RPC101"] RpcCall:Client:ServerStreaming[vmware.nsx.handshake.VersionMastershipHandshake/GetMaster, 0x0001, LOCAL_DONE] Channel is down
2018-12-01T05:23:32Z nsx-proxy[69012]: [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-rpc" tid="69018" level="INFO"] ConnectionKeeper[15 ssl://10.208.214.45:1235] resetting connection, will reconnect
2018-12-01T05:23:32Z nsx-proxy[69012]: [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-rpc" tid="69029" level="ERROR" errorCode="RPC102"] RpcCall:Client:ServerStreaming[vmware.nsx.handshake.VersionMastershipHandshake/GetMaster, 0x0001, LOCAL_DONE] Is in error state (COMMUNICATION_ERROR, status is not reported to the Server)
2018-12-01T05:23:32Z nsx-proxy[69012]: [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-rpc" tid="69018" level="INFO"] ConnectionKeeper[15 ssl://10.208.214.45:1235] closing and releasing connection cid:80
2018-12-01T05:23:32Z nsx-proxy[69012]: [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" tid="69029" level="INFO"] VersionMastershipHandshakeClient: received MasterResponse CallStatus COMMUNICATION_ERROR
2018-12-01T05:23:32Z nsx-proxy[69012]: [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-rpc" tid="69018" level="INFO"] RpcConnection[80 Closed to ssl://10.208.214.45:1235] Closing (closed by user)
2018-12-01T05:23:32Z nsx-proxy[69012]: [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-rpc" tid="69018" level="WARN"] RpcConnection[80 Closed to ssl://10.208.214.45:1235] is already closed
2018-12-01T05:23:32Z nsx-proxy[69012]: [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" tid="69029" level="INFO"] VersionMastershipHandshakeClient: connection down, detected
-- In the same log file, the Transport Node disconnected from its controller. Note that the IP address of the controller is in decimal format i.e { ipv4: 181458477 } which translates to 10.208.214.45 in dotted decimal format.
2018-12-01T05:23:32Z nsx-proxy[69012]: [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" tid="69029" level="INFO"] Write ccp session message to nestdb ccp_id { left: 10487831512535091525 right: 94493069112221
46801 } ip { ipv4: 181458477 } server_port: 1235 fqdn: "" state: DISCONNECTED master: true
-- 24 hours after the Transport Node disconnected from the controller, the controller deletes the Transport Node data from its local database.
2018-12-02 05:23:51.729 UTC INFO [pool-16-thread-1] [c.v.n.c.r.LcpReplicatorImpl] - [nsx@6876 comp="nsx-controller" subcomp="replication"] TN data was cleared (expired, removed or inMaintenance): 48051aaf-6cec-478b-a2c7-34a630743105
-- After several days, the transport node and controller communication is re-established
2018-12-07T01:57:09Z nsx-proxy[69012]: [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-net" tid="69018" level="INFO"] StreamConnection[24262 Connected to ssl://10.208.214.45:1235 sid:24262] Connected from ssl-tcp://10.208.214.142:47322 to server with certificate with sha256 fingerprint '4aa22f69b93189889f74abf441c07c26f010ad1ad1275c6cc52a97f9b268f187'
2018-12-07T01:57:09Z nsx-proxy[69012]: [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-rpc" tid="69018" level="INFO"] ConnectionKeeper[3469 ssl://10.208.214.45:1235] established connection cid:24262
2018-12-07T01:57:09Z nsx-proxy[69012]: [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" tid="69029" level="INFO"] VersionMastershipHandshakeClient: connected {net: {endpoint: ssl://10.208.214.45:1235, local-ep:
ssl-tcp://10.208.214.142:47322, peer-ep: ssl://10.208.214.45:1235, peer-auth: {certificate sha256 '4aa22f69b93189889f74abf441c07c26f010ad1ad1275c6cc52a97f9b268f187'}}}
2018-12-07T01:57:09Z nsx-proxy[69012]: [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-rpc" tid="69018" level="INFO"] RpcTransport[2] Registering remote endpoint ids '918c4331-98cd-4945-8322-ae92390a26f1' @ RpcConnection[24262 Connected to ssl://10.208.214.45:1235]
2018-12-07T01:57:09Z nsx-proxy[69012]: [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-rpc" tid="69018" level="INFO"] RpcTransport[2]::RemoteServiceManager Registering remote service 'vmware.nsx.handshake.VersionMastershipHandshake' @ RpcConnection[24262 Connected to ssl://10.208.214.45:1235]
2018-12-07T01:57:09Z nsx-proxy[69012]: [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" tid="69029" level="INFO"] VersionMastershipHandshakeClient: Connected
2018-12-07T01:57:09Z nsx-proxy[69012]: [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" tid="69029" level="INFO"] VersionMastershipHandshakeClient: created stub successfully
2018-12-07T01:57:09Z nsx-proxy[69012]: [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" tid="69029" level="INFO"] VersionMastershipHandshakeClient: version rpc sending transport_node { left: 5189583486363387787 right: 11729401642891882757 } supported_version { major: 2 minor: 3 } supported_version { major: 2 minor: 1 }
2018-12-07T01:57:09Z nsx-proxy[69012]: [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" tid="69029" level="INFO"] VersionMastershipHandshakeClient: received VersionResponse supported_version { major: 2 minor: 3 }
2018-12-07T01:57:09Z nsx-proxy[69012]: [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" tid="69029" level="WARN"] ChannelDetection new channel reconnect
2018-12-07T01:57:09Z nsx-proxy[69012]: [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" tid="69029" level="INFO"] VersionMastershipHandshakeClient: mastership rpc query sending transport_node { left: 5189583486363387787 right: 11729401642891882757 }
2018-12-07T01:57:09Z nsx-proxy[69012]: [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" tid="69029" level="INFO"] VersionMastershipHandshakeClient: received CallStatus SUCCESS
2018-12-07T01:57:09Z nsx-proxy[69012]: [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" tid="69029" level="INFO"] VersionMastershipHandshakeClient: received MasterResponse { master_node { left: 10487831512535091525 right: 9449306911222146801 } }
-- Transport Node connects to its master controller
2018-12-07T01:57:09Z nsx-proxy[69012]: [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" tid="69029" level="INFO"] Write ccp session message to nestdb ccp_id { left: 10487831512535091525 right: 9449306911222146801 } ip { ipv4: 181458477 } server_port: 1235 fqdn: "" state: CONNECTED master: true
-- In Controller nsx-ccp logs, see the communication channel re-establishing
2018-12-07 01:56:38.813 UTC INFO [nsx-rpc:CCP-918c4331-98cd-4945-8322-ae92390a26f1:user-executor-0] [c.v.n.h.VersionMastershipServiceImpl] - [nsx@6876 comp="nsx-controller" subcomp="handshake-server"] Versi
on handshake succeeded for 48051aaf-6cec-478b-a2c7-34a630743105
2018-12-07 01:56:38.815 UTC INFO [nsx-rpc:CCP-918c4331-98cd-4945-8322-ae92390a26f1:user-executor-3] [c.v.n.h.VersionMastershipServiceImpl] - [nsx@6876 comp="nsx-controller" subcomp="handshake-server"] Open
mastership for 48051aaf-6cec-478b-a2c7-34a630743105, controller is myself 918c4331-98cd-4945-8322-ae92390a26f1
2018-12-07 01:56:38.815 UTC INFO [nsx-rpc:CCP-918c4331-98cd-4945-8322-ae92390a26f1:user-executor-3] [c.v.n.r.c.NsxRpcChannel] - [nsx@6876 comp="nsx-controller" subcomp="ccp"] ClientChannel(b550f158-9659-4c8
e-8707-cff279fce1e9) doOpenStream OutboundMessage(streamId=null payloadSize=0 streamControl=service: "vmware.nsx.nestdb.NestDb"open_stream: truepriority: 64 frame=)
-- An empty dataset is pushed to the Transport Node by its master controller
2018-12-08 01:58:51.734 UTC INFO [Curator-PathChildrenCache-0] [c.v.n.c.r.LcpReplicatorImpl] - [nsx@6876 comp="nsx-controller" subcomp="replication"] Data set becomes empty, transportNodeId=[48051aaf-6cec-478b-a2c7-34a630743105], process=[Store update].
-- In the host vmkernel.log file, you would see messages such as below, indicating that the port is in a blocked state
2018-12-07T01:57:10.790Z cpu1:66668)NetPort: 3203: blocking traffic on DV port 86762494-05a4-40c2-9da3-6419b5fa1b62
2018-12-07T01:57:10.790Z cpu1:66668)kcp: KCP_DeletePort:459: [nsx@6876 comp="nsx-esx" subcomp="kcp"]Port 67108875 is cleared and blocked
--
2018-12-07T01:57:10.791Z cpu1:66668)NetPort: 3203: blocking traffic on DV port d331e025-0983-4825-8693-24279742c6ce
2018-12-07T01:57:10.791Z cpu1:66668)kcp: KCP_DeletePort:459: [nsx@6876 comp="nsx-esx" subcomp="kcp"]Port 67108870 is cleared and blocked
--
2018-12-07T01:57:10.792Z cpu1:66668)NetPort: 3203: blocking traffic on DV port 53588a39-fabc-4bce-86b1-29477b28a83c
2018-12-07T01:57:10.792Z cpu1:66668)kcp: KCP_DeletePort:459: [nsx@6876 comp="nsx-esx" subcomp="kcp"]Port 67108871 is cleared and blocked
--