ESXi upgrade failure due to expired VIB Certificate
search cancel

ESXi upgrade failure due to expired VIB Certificate

book

Article ID: 318791

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:
  • On an esxcli upgrade from pre ESXi 65U2 to ESXi 7.0 and later, you see the error:
Could not find a trusted signer.

For example: 
esxcli software profile update -d /vmfs/volumes/datastore1/VMware-ESXi-7.0.0-15843807-depot.zip -p ESXi-7.0.0-15843807-standard
[InstallationError]
'VMware_bootbank_lsuv2-hpv2-hpsa-plugin_1.0.0-2vmw.700.1.0.15843807', 'Could not find a trusted signer.') vibs = VMware_bootbank_lsuv2-hpv2-hpsa-plugin_1.0.0-2vmw.700.1.0.15843807
 Please refer to the log file for more details.
  • For the VUM based upgrade from pre ESXi 65U2 to ESXi 7.0 and later, you see the error:
Cannot deploy host upgrade agent.

An ESXi fails if upgraded from:
  • Versions starting 6.0 GA (Build: 2494585) but before 6.0 ESXi600-201807001(Build: 9239799) or Versions starting 6.5 GA  (Build: 4564106) but before 6.5 U2 (Build: 8294253) to ESXi 6.5 (OR) ESXi 6.7
  1. Upgrading using the esxcli command fails with the error similar to:
    For example:
    esxcli software  profile update -d <depot location> -p <profile name>
    [InstallationError]
    ('<vib-name>', 'Could not find a trusted signer.')
    vibs = <vib-name>
    Please refer to the log file for more details.
  • In the /var/log/esxupdate.log file, you see entries similar to:
<YYYY-MM-DD>T<time> esxupdate: 78526: root: ERROR: Traceback (most recent call last):
<YYYY-MM-DD>T<time> esxupdate: 78526: root: ERROR:   File "/build/mts/release/bora-4564106/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmware/esximage/Vib.py", line 1570, in VerifySignature
<YYYY-MM-DD>T<time> esxupdate: 78526: root: ERROR: VibSign.PKCS7CertError: Could not find a trusted signer.
  1. ESXi upgrade using Update Manager fails with an error in the vSphere Web Client similar to:
    Notification
    Task Name: Remediate entity
    Target: < Host IP or FQDN >
    Status: Cannot execute upgrade script on host     
    • In the /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server-log4cpp.log file in the vCenter Server, you see entries similar to:
[<YYYY-MM-DD>T<time> 'HU-Upgrader' 140215538296576 ERROR] [upgraderImpl, 445] Script execution failed on host: 10.160.28.250).....
[2<YYYY-MM-DD>T<time> 'AgentDeploy' 140215538296576 INFO] [agentDeploy, 247] Agent installed
[<YYYY-MM-DD>T<time> 'SingleHostUpgradeRemediateTask.SingleHostUpgradeRemediateTask{36}' 140215538296576 ERROR] [singleHostUpgradeRemediateTask, 333] Error running check scripts on host: <host IP address>, host Id: host-9, error: Fault cause: integrity.fault.HostUpgradeRunScriptFailure
  2. Attempting to enable vSphere HA on a vSphere Cluster utilizing Secure Boot fails with “Operation Timed Out”.
In the esxupdate.log file on the ESXi hosts you were attempting to enable vSphere HA on, you see entries similar to:
2020-04-09T03:39:10Z esxupdate: 157884: esxupdate: ERROR: Traceback (most recent call last):^@
2020-04-09T03:39:10Z esxupdate: 157884: esxupdate: ERROR: File "/build/mts/release/bora-7388607/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmware/esximage/Vib.py", line 1570, in VerifySignature^@
2020-04-09T03:39:10Z esxupdate: 157884: esxupdate: ERROR: VibSign.PKyesterdayCS7CertError: Could not find a trusted signer.^@
.....
2020-04-09T03:39:10Z esxupdate: 157884: esxupdate: ERROR: File "/build/mts/release/bora-7388607/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmware/esximage/Vib.py", line 1576, in VerifySignature^@
2020-04-09T03:39:10Z esxupdate: 157884: esxupdate: ERROR: vmware.esximage.Errors.VibSigInvalidError: ('VMware_bootbank_vmware-fdm_6.7.0-15973156'', 'Could not find a trusted signer.')^@
  1. Attempting to re-configure the TLS (Transport Layer Security) settings on the ESXi host fails with the error similar to:  
    root@vcenter-server [ /usr/lib/vmware-TlsReconfigurator/EsxTlsReconfigurator ]# ./reconfigureEsx vCenterHost -h < Host IP or FQDN > -u [email protected]  -p TLSv1.2
ESXi Transport Layer Security reconfigurator, version=6.7.0, build=<vcenter build number>
For more information refer to the following article:https://kb.vmware.com/kb/2147469
Log file: "/var/log/vmware/vSphere-TlsReconfigurator/EsxTlsReconfigurator.log".
Connecting to vCenter Server at: "localhost".
Password:
Validating product version at: "localhost".
Validating ESXi host: "< Host IP or FQDN >".
Reconfiguring ESXi host: "< Host IP or FQDN >" of version "6.5"
Updating ESXi host "< Host IP or FQDN >" advanced option "UserVars.ESXiVPsDisabledProtocols" from "sslv3" to "sslv3,tlsv1,tlsv1.1"
Removing the <sslOptions> tag (if exists) from the reverse HTTP proxy configuration file on ESXi host: "< Host IP or FQDN >".
Reconfiguration FAILED for ESXi host "< Host IP or FQDN >": Cannot install the vCenter Server agent service. Cannot verify the installer signature.
  2. ESXi baseline patching using Update Manager fails with esxupdate error in the vSphere Web Client similar to:  
The host returns esxupdate error code:15. The package manager transaction is not successful. Check the Update Manager log files and esxupdate log files for more details.
  • In the /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server-log4cpp.log file in the vCenter Server, you see entries similar to:
[2020-06-30 04:48:22:568 'HostUpdateDepotManager' 140368283825920 INFO] [installer20, 90] Processing the Install results for host: <host IP address> (Entity: host-27).
<esxupdate-response>
<version>1.50</version>
<error errorClass="InstallationError">
<errorCode>15</errorCode>
<errorDesc>The installation transaction failed.</errorDesc>
<vibs>VMware_bootbank_esx-ui_1.34.2-16361878</vibs>
<msg>('VMware_bootbank_esx-ui_1.34.2-16361878', 'Could not find a trusted signer.')</msg>
</error>
</esxupdate-response>
....
[2020-06-30 04:48:22:568 'HostUpdateDepotManager' 140368283825920 INFO] [installer20, 117] Exit code (Errors): 15, The installation transaction failed. (Host: 10.161.146.58 (Entity: host-27))
Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.


Environment

VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.0
VMware vSphere ESXi 7.0.0

Cause

  • ESXi introduced the new certificate for VIB signing into the trust store in 2018 to begin the transition from the old certificate (which expired on 31st December 2019)
  • ESXi hosts running releases before this transition period accept only the old certificate in VIB signature verification
  • As this transition period is now complete, VIB signatures of new releases are untrusted on releases of ESXi prior to this transition, and an upgrade or installation failure would occur

Resolution

  • Please refer to the workaround given below.


Workaround:
  • In preparation for this expiry event, VMware recommends all customers upgrade their existing ESXi hosts to a minimum of the builds (listed below), which were released after March 2018, and have the newer signing certificate in the trust store
  • Doing so prevents future installation and upgrades failures due to the signing certificate being untrusted
    • 6.0 ESXi600-201807001 (Build: 9239799) or later 
    • 6.5 U2 (Build: 8294253) or later
    • 6.7 GA (Build: 8169922) or later.
  • For more information on ESXi builds and their release dates, please refer to the KB Article - Build numbers and versions of VMware ESXi/ESX (2143832).
  • This article will be updated with more information in the future. Subscribe for the latest.


Additional Information



Powered by Wolken Software