• ESXi hosts disconnects from vCenter Server after upgrading to vSphere 7.0 U3x (any 7.0 U3 updates)

• Connecting the host to vCenter Server fails with the following error message:

"A general system error occurred: Timed out waiting for vpxa to start."
 

• Trying to start vpxa manually using the  command /etc/init.d/vpxa start also fails

• In the vpxa agent log /var/run/log/vpxa.log you find the following error:

2022-02-24T17:30:18.247Z error vpxa[2105502] [Originator@6876 sub=vmomi.soapStub[1]] Error deserializating SOAP response body; <<io_obj p:0x000000ae2f01dfb8, h:14, <TCP '127.0.0.1 : 18946'>, <TCP '127.0.0.1 : 8307'>>, /sdk>, method: waitForUpdates, e:
--> Error returned by expat parser: not well-formed (invalid token)
-->
--> while parsing serialized value of type string
--> at line 19, column 1132139
-->
--> while parsing property "physicalLocation" of static type ArrayOfString
-->
--> while parsing serialized DataObject of type vim.host.ScsiDisk
--> at line 19, column 1119057
-->
--> while parsing property "scsiLun" of static type ArrayOfScsiLun
-->
--> while parsing serialized DataObject of type vim.host.StorageDeviceInfo
--> error parsing Any with xsiType HostStorageDeviceInfo
--> at line 19, column 1067318
-->
--> while parsing property "val" of static type anyType
-->
--> while parsing serialized DataObject of type vmodl.query.PropertyCollector.Change
--> at line 19, column 1067262
-->
--> while parsing property "changeSet" of static type ArrayOfPropertyChange
-->
--> while parsing serialized DataObject of type vmodl.query.PropertyCollector.ObjectUpdate
--> at line 19, column 1047023
-->
--> while parsing property "objectSet" of static type ArrayOfObjectUpdate
-->
--> while parsing serialized DataObject of type vmodl.query.PropertyCollector.FilterUpdate
--> at line 7, column 52326
-->
--> while parsing property "filterSet" of static type ArrayOfPropertyFilterUpdate
-->
--> while parsing serialized DataObject of type vmodl.query.PropertyCollector.UpdateSet
--> at line 7, column 50
-->
--> while parsing return value of type vmodl.query.PropertyCollector.UpdateSet, version [N5Vmomi15InternalVersionE:0x000000ae2ea451d0]
--> at line 7, column 0
-->
--> while parsing SOAP body
--> at line 6, column 0
-->
--> while parsing SOAP envelope
--> at line 2, column 0
-->
--> while parsing HTTP response for method waitForUpdates
--> on object of type vmodl.query.PropertyCollector
--> at line 1, column 0
2022-02-24T17:30:18.247Z warning vpxa[2105502] [Originator@6876 sub=IO.Http] Closing response processing in unexpected state; <io_obj p:0x000000ae2f01dfb8, h:14, <TCP '127.0.0.1 : 18946'>, <TCP '127.0.0.1 : 8307'>>, state: 3
2022-02-24T17:30:18.248Z error vpxa[2105502] [Originator@6876 sub=VpxaCnxHostd opID=WFU-41a7] Fatal error from StartFetchingUpdates(): N5Vmomi4Soap24InvalidResponseExceptionE(
--> Error returned by expat parser: not well-formed (invalid token)
-->
--> while parsing serialized value of type string
--> at line 19, column 1132139
-->
--> while parsing property "physicalLocation" of static type ArrayOfString
-->
--> while parsing serialized DataObject of type vim.host.ScsiDisk
--> at line 19, column 1119057
-->
--> while parsing property "scsiLun" of static type ArrayOfScsiLun
-->
--> while parsing serialized DataObject of type vim.host.StorageDeviceInfo
--> error parsing Any with xsiType HostStorageDeviceInfo
--> at line 19, column 1067318
-->
--> while parsing property "val" of static type anyType
-->
--> while parsing serialized DataObject of type vmodl.query.PropertyCollector.Change
--> at line 19, column 1067262
-->
--> while parsing property "changeSet" of static type ArrayOfPropertyChange
-->
--> while parsing serialized DataObject of type vmodl.query.PropertyCollector.ObjectUpdate
--> at line 19, column 1047023
-->
--> while parsing property "objectSet" of static type ArrayOfObjectUpdate
-->
--> while parsing serialized DataObject of type vmodl.query.PropertyCollector.FilterUpdate
--> at line 7, column 52326
-->
--> while parsing property "filterSet" of static type ArrayOfPropertyFilterUpdate
-->
--> while parsing serialized DataObject of type vmodl.query.PropertyCollector.UpdateSet
--> at line 7, column 50
-->
--> while parsing return value of type vmodl.query.PropertyCollector.UpdateSet, version [N5Vmomi15InternalVersionE:0x000000ae2ea451d0]
--> at line 7, column 0
-->
--> while parsing SOAP body
--> at line 6, column 0
-->
--> while parsing SOAP envelope
--> at line 2, column 0
-->
--> while parsing HTTP response for method waitForUpdates
--> on object of type vmodl.query.PropertyCollector
--> at line 1, column 0)
--> [context]zKq7AVICAgAAACzgJAEIdnB4YQAAdUMXbGlidm1hY29yZS5zbwABhOIYbGlidm1vbWkuc28AAejtGACcxS0AJAEuANIOPwI7fQBsaWJwdGhyZWFkLnNvLjAAA72nDmxpYmMuc28uNgA=[/context]
2022-02-24T17:30:18.252Z error vpxa[2105502] [Originator@6876 sub=vpxaInvtHostCnx opID=WFU-41a7] Can't connect to hostd. Shutting down...

This is a known issue in ESXi 7.0 Update 3 and has been fixed in ESXi 7.0 Update 3d.
Should you still experience this issue after the ESXi has been updated to 7.0 U3d, please open a support request.

To work around the issue, please take the following steps:

IMPORTANT NOTE:
Workaround steps mentioned below are NOT APPLICABLE for ESXi Hosts in vLCM Image Clusters as removing the VIB from these hosts will result in Host out of compliance with the Cluster Image, sample out of compliance screenshot in Related Information of this KB.

1. Set the ESXi host in maintenance mode

2. Enable SSH or DCUI Shell on the ESXi host

3. Login to the host using SSH Client or through DCUI Shell 

4. Remove the VIB "lsuv2-lsiv2-drivers-plugin" by executing the following command

# esxcli software vib remove -n lsuv2-lsiv2-drivers-plugin

Sample screenshot:

 

5. Reboot the host

6. Exit the host from maintenance mode
    

Impact of Removing this Plugin: This plugin (lsuv2-lsiv2-drivers-plugin) helps to get the device info (PHYSICAL/RAID level), disk location (slot number) and turn on/off the disk LED on the hosts. Other than this, there is no other impact in removing the specified VIB.

When using the workaround steps on ESXi hosts managed by vSphere Lifecycle Manager (vLCM) with a Cluster Image, the removal will result in the ESXi no longer being compliant with the image, as can be seen in the sample screenshot pasted below:

In this configuration, please do not apply the workaround, but instead update the ESXi to 7.0 Update 3d where the problem is fixed.