Upgrading vCenter Server or ESXi 8.0 fails during precheck due to a weak certificate signature algorithm
search cancel

Upgrading vCenter Server or ESXi 8.0 fails during precheck due to a weak certificate signature algorithm

book

Article ID: 313460

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

Symptoms:
Upgrading to vCenter Server or ESXi 8.0 requires an additional precheck. The precheck ensures that vCenter Server and ESXi are not using certificates with weak signature algorithms, such as sha1WithRSAEncryption. Certificates with weak signature algorithms are no longer supported in vSphere 8.0 in most contexts.

Environment

VMware vCenter Server 8.0.0
VMware vSphere ESXi 8.0.0

Cause

vCenter Server upgrades may fail during precheck with an error stating, "Support for certificates with weak signature algorithms has been removed from vCenter Server 8.0." An example of the screenshot with a representative error is shown below:

image.png

ESXi upgrades may fail during precheck with an error string stating, "Support for certificates with weak signature algorithm SHA-1 has been removed in ESXi 8.0." Example screenshots with representative errors are show below.

image.png

image.png

image.png

 

Resolution

Replacing Certificates

The vCenter Server Machine Certificate

If the upgrade precheck failure message indicates that a problematic certificate is present in the VECS store "MACHINE_SSL_CERT", then the vCenter Server machine certificate must be replaced before upgrade can proceed. The vCenter Server machine certificate is the primary TLS certificate used by the vCenter Server HTTPS endpoint, which defaults to port 443. This is the certificate that is seen when connecting to the vCenter Server web UI.
Steps to replace the certificate:
  1. Replace the vCenter Server machine certificate, and the associated root and intermediate certificates with a certificate that contains a SHA-2 digital signature
  1. Push the new CA certificate(s) to the ESXi hosts using "Refresh CA Certificates" in the vSphere UI.
  1. Remove the old and now unused root certificate from vCenter Server using the `dir-cli trustedcert unpublish` command.

A vCenter Server Trusted Root Certificate

If the upgrade precheck failure message indicates that a problematic certificate is present in the VECS store "TRUSTED_ROOTS", then vCenter Server has configured trusted root or intermediate certificate that must be removed or replaced before upgrade can proceed. There may be a dependency on the problematic trusted root certificates and it's important to update the dependent services before removing or replacing the certificate. Also note that the certificates that are present in the VECS store "TRUSTED_ROOTS" are pushed to all connected ESXi hosts.

Steps to replace the certificate:
  1. Add a new certificate to the "TRUSTED_ROOTS" store that will replace the now unsupported certificate.
  1. Push the new CA certificate(s) to the ESXi hosts using "Refresh CA Certificates" in the vSphere UI.
  1. Ensure that any certificate signed by the unsupported certificate is removed from the vSphere environment. If a leaf certificate is found to be signed by the unsupported certificate, it must also be replaced. Consider the following cases.
  • The leaf certificate may be in use on an ESXi host.
  • The leaf certificate may be used by a VMware solution, such as vRA, vROps, SRM, NSX, etc.
  • The leaf certificate may be used by a partner solution, such as vVols or a backup solution.
Note: For vVols please re-register the VASA Provider in vCenter
  1. Once the unsupported certificate is no longer in use, it may be removed from the TRUSTED_ROOTS store.
  1. Push the CA certificate changes to the ESXi hosts using "Refresh CA Certificates" in the vSphere UI.

VMCA is Acting as a Subordinate CA

In a default vCenter Server deployment, VMCA is the root certificate authority (CA). However, VMCA can be configured a subordinate CA where an outside authority is the root and VMCA uses an intermediate certificate. If VMCA is configured as a subordinate CA and the root certificate uses a weak digital signature, then the root certificate will need to be replaced and all certificates reissued.
  1. Reset the VMCA certificate using one of the following methods:
- A new intermediate certificate may be configured with VMCA as a subordinate CA.
- A default VMCA root certificate may be configured.
- All vCenter Server certificates will be regenerated.
- A default VMCA root certificate may be configured.
- All vCenter Server certificates will be regenerated.
- Note that the REST API cannot be used to configure VMCA as a subordinate CA.
  1. Reissue all ESXi TLS certificates with the new VMCA certificate if they were signed with the previous VMCA certificate.
  1. Remove the old and now unused CA root certificate and intermediate certificates from the TRUSTED_ROOTS store of VECS.
  1. Push the CA certificate changes to the ESXi hosts using "Refresh CA Certificates" in the vSphere UI.

vCenter Server BACKUP_STORE Certificate

If the upgrade precheck failure message indicates that a problematic certificate is present in the VECS store "BACKUP_STORE", then the certificate can be safely removed using the `vecs-cli` command.

The ESXi Server TLS Certificate (rui.crt)

The ESXi TLS certificate is managed by vCenter Server by default, however administrators may choose to manually assign a certificate. If the current TLS certificate contains a weak digital certificate, then a new certificate must be issued. Note that this certificate is stored in a file name "rui.crt" which may be displayed in the upgrade precheck error messages.

The ESXi Server Certificate Store (castore.pem)

vCenter Server pushes its own Trusted Root certificates, "TRUSTED_ROOTS", to the ESXi Certificate Store. However, each ESXi host may have additional certificates added manually as well. If the ESXi Certificate Store contains a certificate with a weak digital signature then the certificate can be removed using esxcli.

The currently configured certificates can be listed with the following command.
$ esxcli system security certificatestore list
Certificates can be used with the following command.
$ esxcli system security certificatestore remove
Note that esxcli command can also be accessed using PowerCLI

Standalone Precheck Script

The `vsphere8_upgrade_certificate_checks.py` Python script verifies that vCenter Server and the connected ESXi hosts are not using certificates with a weak digital signature algorithm. This is a standalone version of the same prechecks that are preformed during vCenter Server and ESXi upgrades. It can be run manually before a planned upgrade maintenance window.

The script first checks if vCenter Server has any unsupported certificates in the VECS stores. It then iterates through all of the ESXi hosts in the inventory to perform similar checks.

To run the script perform the following steps:

  1. Download the `vsphere8_upgrade_certificate_checks.py` script attached to this KB.
  2. Transfer the script file to a temporary folder (e.g. `/tmp`) on vCenter Server (e.g. using `scp` or `WinSCP`).
  3. Login to the vCenter Server appliance using root credentials with an SSH client, or create a new RDP session if the vCenter Server is Windows based.
  4. Execute the script using the following command:
  • On Linux using shell: `python /tmp/vsphere8_upgrade_certificate_checks.py`
  • On Windows using cmd.exe: `"%VMWARE_PYTHON_BIN%" C:\Users\Administrator\Desktop\vsphere8_upgrade_certificate_checks.py`

If any certificates with a weak signature algorithm are found, the details are printed to the console window. These issues should be resolved before proceeding with upgrade. An example output with failures is shown below.

image.png

vVols (Virtual Volumes)

vVol VASA Provider uses Self Signed Certificate that are issued by 3rd party VASA providers which in certain cases are stored directly in VC trusted store. Follow the below mentioned steps:

  1. Verify if the certificate algorithm is of sha1WithRSAEncryption from “Certificate Info” section of VASA provider. If So, follow below steps to update Certificate to SHA256.
  2. Confirm the self-signed certificate using the following command:

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text

Alias : https://<VP_IP>:<PORT_NO>/version.xml
Signature Algorithm: sha1WithRSAEncryption
SAN: IP:VP_IP, DNS:VP_DNS
CA:False

  1. If CA field is 'False', then it’s a self-signed certificate and if the user is using vVols, then a new self signed certificate needs to be generated and installed on VASA provider which has at least SHA256 Signature.

Note – While installing the certificates the VASA provider may be unavailabile on ESXi host.

  1. Unregister/re-register the VASA provider in all VCs that are connected to this VASA provider.
  2. Run the root sync so all the connected hosts get the new trusted root.
  3. Proceed with the upgrade.

 


Attachments

vsphere8_upgrade_certificate_checks get_app