After update to ESXi 7.0 Update 3 vSphere HA fails to enable

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

After patching ESXi to 7.0 Update 3 - HA can no longer be successfully enabled
When trying to enable HA, the installation of the HA agent (FDM) starts, and the status switches into "Waiting for cluster election to complete", only to fail and start over again
The HA status for the ESXi host is stuck at "Uninitialized"
In some cases the HA enablement task fails with "Cannot find vSphere HA master agent"
Migrating or powering on VMs in an HA enabled cluster fail

In /var/run/log/esxupdate.log on the ESXi, you find the following entries:

2021-10-13T14:01:17Z esxupdate: 2140874: root: ERROR: Traceback (most recent call last):
2021-10-13T14:01:17Z esxupdate: 2140874: root: ERROR:   File "/usr/lib/vmware/esxcli-software", line 773, in <module>
2021-10-13T14:01:17Z esxupdate: 2140874: root: ERROR:     main()
2021-10-13T14:01:17Z esxupdate: 2140874: root: ERROR:   File "/usr/lib/vmware/esxcli-software", line 764, in main
2021-10-13T14:01:17Z esxupdate: 2140874: root: ERROR:     ret = CMDTABLE[command](options)
2021-10-13T14:01:17Z esxupdate: 2140874: root: ERROR:   File "/usr/lib/vmware/esxcli-software", line 601, in VibInstallCmd
2021-10-13T14:01:17Z esxupdate: 2140874: root: ERROR:     res = t.InstallVibsFromSources(viburls, [], nameids,
2021-10-13T14:01:17Z esxupdate: 2140874: root: ERROR:   File "/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 965, in InstallVibsFromSources
2021-10-13T14:01:17Z esxupdate: 2140874: root: ERROR:     inst, removed, exitstate = self._installVibs(curprofile,
2021-10-13T14:01:17Z esxupdate: 2140874: root: ERROR:   File "/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 1207, in _installVibs
2021-10-13T14:01:17Z esxupdate: 2140874: root: ERROR:     hasConfigDowngrade = checkFdmConfigDowngrade(curProfile, newProfile)
2021-10-13T14:01:17Z esxupdate: 2140874: root: ERROR:   File "/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 1122, in checkFdmConfigDowngrade
2021-10-13T14:01:17Z esxupdate: 2140874: root: ERROR:     compDowngrades = curProfile.GetCompsDowngradeInfo(newProfile)
2021-10-13T14:01:17Z esxupdate: 2140874: root: ERROR:   File "/lib64/python3.8/site-packages/vmware/esximage/ImageProfile.py", line 2416, in GetCompsDowngradeInfo
2021-10-13T14:01:17Z esxupdate: 2140874: root: ERROR:     curComp = self.components.GetComponent(name)
2021-10-13T14:01:17Z esxupdate: 2140874: root: ERROR:   File "/lib64/python3.8/site-packages/vmware/esximage/Bulletin.py", line 1276, in GetComponent
2021-10-13T14:01:17Z esxupdate: 2140874: root: ERROR:     raise ValueError('Expected 1 component, found %u'
2021-10-13T14:01:17Z esxupdate: 2140874: root: ERROR: ValueError: Expected 1 component, found 2

Environment

VMware vSphere ESXi 7.0.3

Cause

This issue occurs when both the i40en and i40enu VIBs for the inbox Intel network driver are installed on the host
The issue was also reported after upgrading to vCenter 7.0 Update 3b, released on 12th November 2021, which is no longer available.
In vSphere 7.0 Update 2, the driver was renamed from i40en to i40enu. Starting with vSphere 7.0 Update 3, the inbox i40enu network driver for ESXi changes back to i40en

Resolution

This issue is resolved in ESXi 7.0 Update 3c.

Workaround:
To work around the issue, remove the conflicting i40enu driver

Disable vSphere HA on the cluster with the affected hosts.
Put the ESXi in maintenance mode, then run the following command to remove the obsolete i40enu VIB:
```
# esxcli software vib remove --vibname=i40enu
```
Once the VIB has been removed, reboot the ESXi host.
When this has been done for all hosts in the cluster, HA can be enabled again.

Additional Information

This issue is can also lead to other problems, e.g. failures when attempting to patch ESXi using Lifecycle Manager, which are described in KB :- https://kb.vmware.com/s/article/85982
Please read and subscribe to referenced KB article for further information.