vSphere Lifecycle Manager can show non-compliant status after rollup Remediation, new cluster creation or transition from Baseline-managed cluster to Image-managed cluster for few upgrade paths due to i40en/i40enu driver name change.
search cancel

vSphere Lifecycle Manager can show non-compliant status after rollup Remediation, new cluster creation or transition from Baseline-managed cluster to Image-managed cluster for few upgrade paths due to i40en/i40enu driver name change.

book

Article ID: 318834

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • This is an awareness article where upgrade from 7.0U2a to 7.0U3 using rollup patch baseline will fail
    Environments 
  • Environments upgraded host to 7.0 U3 from 7.0U2 (using rollup patch baseline), will not be able to import image (HostSeeding) from that host to desired image based cluster.
  • During Baseline-Image managed cluster (where hosts are upgraded from 70u2 to 70u3 using rollup baseline upgrade) to Desired-Image managed cluster transition, there is one additional remediation to be performed


Symptoms:
  • Rollup upgrade of ESXi hosts with patch baselines from ESXi 7.0 Update 2a to ESXi 7.0 Update 3 fails
  • "VMware vSphere Update Manager had an unknown error"
  • On vCenter UI the error message reported :- "host returned esxupdate code -1"

    esxupdate.log reported this exception:
    2021-08-27T14:11:56Z esxupdate: 2102510: Ramdisk: INFO: Unmounting manual tardisk /tardisks.noauto/weaselin-2102510
    2021-08-27T14:11:56Z esxupdate: 2102510: esxupdate: ERROR: An esxupdate error exception was caught:
    2021-08-27T14:11:56Z esxupdate: 2102510: esxupdate: ERROR: Traceback (most recent call last):
    2021-08-27T14:11:56Z esxupdate: 2102510: esxupdate: ERROR:   File "/usr/sbin/esxupdate", line 216, in main
    2021-08-27T14:11:56Z esxupdate: 2102510: esxupdate: ERROR:     cmd.Run()
    2021-08-27T14:11:56Z esxupdate: 2102510: esxupdate: ERROR:   File /lib64/python3.8/site-packages/vmware/esx5update/Cmdline.py, line 153, in Run
    2021-08-27T14:11:56Z esxupdate: 2102510: esxupdate: ERROR:   File /tmp/esx-update-2102510/lib64/python3.8/site-packages/vmware/esximage/Transaction.py, line 962, in InstallVibsFromSources
    2021-08-27T14:11:56Z esxupdate: 2102510: esxupdate: ERROR:   File /tmp/esx-update-2102510/lib64/python3.8/site-packages/vmware/esximage/Transaction.py, line 928, in checkBaseImageUpgrade
    2021-08-27T14:11:56Z esxupdate: 2102510: esxupdate: ERROR: esximage.Errors.InstallationError: This upgrade transaction would skip ESXi Base Image VIB(s) VMW_bootbank_i40en_1.11.1.31-1vmw.703.0.0.49537716, which could cause failures post upgrade. Please use an alternative upgrade method that would install the above VIB(s). For example, use an image profile instead of VIBs to perform the upgrade.
    2021-08-27T14:11:56Z esxupdate: 2102510: esxupdate: DEBUG: <<<
  • Import Image from an upgraded host (where upgrade was done from 70u2, using 70u3 rollup patch baseline) may fail with the following error
  • "An error occurred while extracting image info on the host" ESXi  /var/log/lifecycle.log will show below message 
2021-08-18T13:39:24Z lifecycle: 2101447: HostSeeding:767 ERROR Software info extract errors: The following Components have been removed or downgraded on the host: Intel-i40en                                      
2021-08-18T13:39:24Z lifecycle: 2101447: imagemanagerctl:554 ERROR Get Software Info Failed: The following Components have been removed or downgraded on the host: Intel-i40en
  • Baseline-Image managed cluster to Desired-Image managed cluster transition would result into non-compliant state.
  • ISO-based upgrade to ESXi 7.0 Update 3 fails
/var/log/vua-*.log reported this exception:
ERROR:root:Failed to perform image profile update: Expected 1 component, found 2
...
ValueError: Expected 1 component, found 2
  • On vCenter UI error is reported: "Cannot run upgrade script on host". 
  • One of /var/log/vua-*.log on host shows error: 
--> ERROR:root:Failed to perform image profile update: Expected 1 component, found 2 
--> Traceback (most recent call last): 
--> File "/tmp/vuaScript-QtZ4DU/prep.py", line 145, in <module> 
--> sys.exit(main()) 
--> File "/tmp/vuaScript-QtZ4DU/prep.py", line 135, in main 
--> res = t.InstallVibsFromDeployDir(pathToISO) 
--> File "/tmp/vuaScript-QtZ4DU/esximage.zip/vmware/esximage/Transaction.py", line 1703, in InstallVibsFromDeployDir 
--> File "/tmp/vuaScript-QtZ4DU/esximage.zip/vmware/esximage/Transaction.py", line 1207, in _installVibs 
--> File "/tmp/vuaScript-QtZ4DU/esximage.zip/vmware/esximage/Transaction.py", line 1122, in checkFdmConfigDowngrade 
--> File "/tmp/vuaScript-QtZ4DU/esximage.zip/vmware/esximage/ImageProfile.py", line 2416, in GetCompsDowngradeInfo 
--> File "/tmp/vuaScript-QtZ4DU/esximage.zip/vmware/esximage/Bulletin.py", line 1276, in GetComponent 
--> ValueError: Expected 1 component, found 2 
  • On 7.0 Update 2c patched via patch baseline from vendor Image 7.0 Update 2a, HA cannot be enabled/configured 
    • On vCenter UI error is reported: "Sphere HA agent for this host has an error: The vSphere HA agent is not reachable from the vCenter Server" 
    • /var/run/log/esxupdate.log of impacted host shows: 
2021-10-07T23:18:23Z esxupdate: 2110025: esxupdate: ERROR: An unexpected exception was caught: 
2021-10-07T23:18:23Z esxupdate: 2110025: esxupdate: ERROR: Traceback (most recent call last): 
2021-10-07T23:18:23Z esxupdate: 2110025: esxupdate: ERROR: File "/usr/sbin/esxupdate", line 216, in main 
2021-10-07T23:18:23Z esxupdate: 2110025: esxupdate: ERROR: cmd.Run() 
2021-10-07T23:18:23Z esxupdate: 2110025: esxupdate: ERROR: File "/lib64/python3.8/site-packages/vmware/esx5update/Cmdline.py", line 153, in Run 
2021-10-07T23:18:23Z esxupdate: 2110025: esxupdate: ERROR: File "/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 965, in InstallVibsFromSources 
2021-10-07T23:18:23Z esxupdate: 2110025: esxupdate: ERROR: inst, removed, exitstate = self._installVibs(curprofile, 
2021-10-07T23:18:23Z esxupdate: 2110025: esxupdate: ERROR: File "/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 1207, in _installVibs 
2021-10-07T23:18:23Z esxupdate: 2110025: esxupdate: ERROR: hasConfigDowngrade = checkFdmConfigDowngrade(curProfile, newProfile) 
2021-10-07T23:18:23Z esxupdate: 2110025: esxupdate: ERROR: File "/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 1122, in checkFdmConfigDowngrade 
2021-10-07T23:18:23Z esxupdate: 2110025: esxupdate: ERROR: compDowngrades = curProfile.GetCompsDowngradeInfo(newProfile) 
2021-10-07T23:18:23Z esxupdate: 2110025: esxupdate: ERROR: File "/lib64/python3.8/site-packages/vmware/esximage/ImageProfile.py", line 2416, in GetCompsDowngradeInfo 
2021-10-07T23:18:23Z esxupdate: 2110025: esxupdate: ERROR: curComp = self.components.GetComponent(name) 
2021-10-07T23:18:23Z esxupdate: 2110025: esxupdate: ERROR: File "/lib64/python3.8/site-packages/vmware/esximage/Bulletin.py", line 1276, in GetComponent 
2021-10-07T23:18:23Z esxupdate: 2110025: esxupdate: ERROR: raise ValueError('Expected 1 component, found %u' 
2021-10-07T23:18:23Z esxupdate: 2110025: esxupdate: ERROR: ValueError: Expected 1 component, found 2 
  • Remediate patch baseline to 7.0 Update 2c/2d or subsequent Update 2 patch failed 
    • On vCenter UI the error message reported is: "host returned esxupdate code -1" 
    • /var/run/log/esxupdate.log of impacted host reported this exception: 
2021-10-13T16:10:15Z esxupdate: 2106354: esxupdate: ERROR: An esxupdate error exception was caught: 
2021-10-13T16:10:15Z esxupdate: 2106354: esxupdate: ERROR: Traceback (most recent call last): 
2021-10-13T16:10:15Z esxupdate: 2106354: esxupdate: ERROR: File "/usr/sbin/esxupdate", line 238, in main 
2021-10-13T16:10:15Z esxupdate: 2106354: esxupdate: ERROR: cmd.Run() 
2021-10-13T16:10:15Z esxupdate: 2106354: esxupdate: ERROR: File "/lib64/python3.8/site-packages/vmware/esx5update/Cmdline.py", line 153, in Run 
2021-10-13T16:10:15Z esxupdate: 2106354: esxupdate: ERROR: File "/tmp/esx-update-2106354/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 957, in InstallVibsFromSources 
2021-10-13T16:10:15Z esxupdate: 2106354: esxupdate: ERROR: File "/tmp/esx-update-2106354/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 923, in checkBaseImageUpgrade 
2021-10-13T16:10:15Z esxupdate: 2106354: esxupdate: ERROR: esximage.Errors.InstallationError: This upgrade transaction would skip ESXi Base Image VIB(s) VMW_bootbank_i40enu_1.8.1.137-1vmw.702.0.20.18426014, which could cause failures post upgrade. Please use an alternative upgrade method that would install the above VIB(s). For example, use an image profile instead of VIBs to perform the upgrade
 
 

 


Environment

VMware vSphere ESXi 7.0.2
VMware vSphere ESXi 7.0.3
VMware vSphere 7.0.x
VMware vSphere ESXi 7.0.0

Cause

  • Starting with vSphere 7.0 Update 3, the inbox i40enu network driver for ESXi changes name back to i40en.
  • The i40en driver was renamed to i40enu in vSphere 7.0 Update 2
  • The name change impacted rollup upgrade path, where on impacted workflow i40enu would not be replaced by i40en, which causes errors for import image (host seeding) and Baseline Image managed cluster to Desired Image managed cluster transition workflows

Resolution

This issue is resolved in ESXi 7.0 Update 3c.
The issue was also resolved in ESXi 7.0 U3b (No longer available).

Workaround:
Please follow the steps mentioned in any of below methods to resolve the issue :

Pre-Check:
  • Prior to upgrade, check if i40enu vib is present on the host
# esxcli software vib list | grep i40enu 
  • If i40enu vib is present, check if there are two Intel-i40en components exist on the host  

# esxcli software component get | grep Intel-i40en -A15
Workaround - 1:

  • Run both the steps from pre-check section above,
  • If two Intel-i40en components are found on ESXi host, one with i40enu vib, use following workaround steps on the impacted hosts
    • Remove i40enu driver vib with the below command 
      • #esxcli software vib remove -n i40enu 
    • Reboot the host 
      • #reboot
    • Above workaround will help in resolving: Enabling HA issue explained in the KB :- 86191
    • Now you can attempt enabling HA and it should succeed
    • Upgrade issue to 7.0 Update 3
      • Now you can proceed with upgrade to 7.0 Update 3 and it should succeed
    • Reboot the host once the upgrade to 7.0 U3 is completed with the below command  
      • #reboot
 Workaround - 2:
  • Run both the steps from pre-check section above,
  • If only one Intel-i40en component found with i40enu vib and ESXi host is currently running ESXi 7.0 Update 2a build, use one of the following alternative ways to upgrade to U3, such as
    • ISO upgrade baseline,  
    • esxcli profile, or  
    • Migrate to image-managed cluster and perform upgrade
  • Alternately 
    • Use patch baseline to apply 7.0 Update 2c first, before upgrading to 7.0 Update 3

 Workaround - 3:

  • Run both the steps from pre-check section above,
  • If you are running into upgrade failures during upgrade attempt to 7.0 Update 2c/ 7.0 Update 2d, it is recommended to upgrade to 7.0 Update 3 directly using ISO upgrade baseline
    • If you are attempting to upgrade using patching baseline/rollup upgrade, you need to apply workaround-1 or workaround-2 based on the i40enu vib presence and number of Intel-i40en components found with pre-check steps
If are running into upgrade issues due to intel-nvme-vmd/iavmd drivers, please refer KB 85701 for the relevant resolution/workaround 

IMPORTANT NOTE:
Post upgrade to 7.0 Update 3, do not apply 7.0U2x based baselines or the “critical host patches” baseline until fix for this issue is made available in future patches. Otherwise, you may hit a similar issue again