vCenter Server certificate validation error for external solutions in environments with Embedded Platform Services Controller
search cancel

vCenter Server certificate validation error for external solutions in environments with Embedded Platform Services Controller

book

Article ID: 322835

calendar_today

Updated On:

Products

VMware Aria Suite VMware Live Recovery VMware NSX Networking VMware vCenter Server VMware vSphere ESXi VMware Integrated OpenStack

Issue/Introduction

Some solutions, such as VMware vCenter Site Recovery Manager, VMware vSphere Replication, or VMware vCenter Support Assistant are always installed on a different machine than the vCenter Server system.

If you replace the machine SSL certificate of a vCenter Server system with an embedded Platform Services Controller, a connection error results when the solution attempts to connect to the vCenter Server system. The reason is that the vCenter Server system uses a new certificate, but the corresponding registration with the VMware Lookup Service is not updated. When solutions connect to vCenter Server, they use the service registration information, which includes the service URL and the sslTrust string. The sslTrust string is the Base 64 encoded certificate. By default, the old certificate remains part of the service registration even if you successfully replace the vCenter Server certificate.
 
This article explains how to resolve the issue in environments with an Embedded Platform Services Controller.
Warning (vSphere 6.0 only):
 
For Windows Servers running the Platform Services Controller, you must replace the lstoolutil.py file prior to running the ls_update_certs.py script.

To replace lstoolutil.py:
  1. Back up the existing "%VMWARE_CIS_HOME%"\VMware Identity Services\lstool\scripts\lstoolutil.py file. By default, file is location at C:\Program Files\VMware\vCenter Server\VMware Identity Services\lstool\scripts\.
  2. Download the attached 2121701_lstoolutil.py.zip file.
  3. Extract the lstoolutil.py file from the 2121701_lstoolutil.py.zip file in to "%VMWARE_CIS_HOME%"\VMware Identity Services\lstool\scripts\.

This issue is resolved in vSphere 6.0 Update 1.


Environment

VMware vCenter Server 6.7.x
VMware vCenter Server Appliance 6.0.x
VMware Integrated OpenStack 1.0.x
VMware vCenter Support Assistant 6.0.x
VMware vSphere Replication 6.0.x
VMware vCenter Server Appliance 6.7.x
VMware vCenter Server 6.5.x
VMware vCenter Server 6.0.x
VMware NSX for vSphere 6.1.x
VMware vCenter Site Recovery Manager 6.0.x
VMware vCenter Server Appliance 6.5.x

Resolution

This issue is resolved in vCenter Server 6.0 Update 1b, available at VMware Downloads. For more information, see VMware vCenter Server 6.0 Update 1b Release Notes.
 
Notes:
  • Installing vCenter Server 6.0 update 1b on a system that is affected does not resolve the issue until you replace the certificates again.
  • The update resolves the issue for certificate replacement with the Certificate Manager utility. The update does not resolve the issue for certificate replacement from the Services Controller UI.
 
To resolve this issue when using the Platform Services Controller UI to replace the certificates, run the ls_update_certs.py script on the Platform Services Controller. When you run the script, you pass in the old certificate and the new certificate.

Notes:
  • Run this script always on the Platform Services Controller.
  • To run the script, you need the thumbprint of the old vCenter Server certificate and you need the new certificate. You must upload these files to the Platform Services Controller before you run the script.
  • Ensure to back up your existing certificates before you run the script.
  • Run this script each time you replace a certificate.
The process includes several tasks:
 

Task 0: Validating the sslTrust Anchors for the vCenter Server with Embedded PSC

Validating the sslTrust Anchors from Command Line on the vCenter Server Appliance with Embedded PSC
  1. Log in to the External Platform Services Controller Appliance through SSH or console.
  2. Run this command to enable access the Bash shell:

    shell.set --enabled true
     
  3. Type shell and press Enter.
  4. Run this command to get the current sslTrust anchor stored for the Platform Services Controller:

    /usr/lib/vmidentity/tools/scripts/lstool.py list --url https://localhost/lookupservice/sdk --no-check-cert --ep-type com.vmware.cis.cs.identity.sso 2>/dev/null

    For example:

    Note: SSL trust was truncated for readability.

    Service Product: com.vmware.cis
    Service Type: cs.identity
    Service ID: 04608398-1493-4482-881b-b35961bf5141
    Site ID: vmware
    Owner ID: [email protected]
    Version: 2.0
    Endpoints:
    Type: com.vmware.cis.cs.identity.sso
    Protocol: wsTrust
    URL: https://psc.vmware.local/sts/STSService/vsphere.local
    SSL trust: MIIDWDCCAkCgAwIBAgIJANr+++MJ5+WxMA0GCSqGSIb3DQEBCwUAMGgxCzAJBgNV ... LqSKWg/apP1XlBV1VyC5LPZrH/rrq8+Naoj7i/P6HAzTwAAL+O10ggClaP8=
     
  5. Run this command to get the current SSL certificate used on port 443 on the Platform Services Controller:

    echo | openssl s_client -connect localhost:443

    For example:

    Note: The certificate was truncated for readability.

    CONNECTED(00000003)
    depth=3 /DC=local/DC=VMWARE/CN=VMWARE-WCA-CA-1
    verify return:1
    depth=2 /DC=local/DC=VMWARE/CN=VMWARE-WCAI-CA-1
    verify return:1
    depth=1 /C=US/DC=vsphere/DC=local/O=psc.vmware.local/CN=CA
    verify return:1
    depth=0 /CN=psc.vmware.local/C=US
    verify return:1
    ---
    Certificate chain
    0 s:/CN=psc.vmware.local/C=US
    i:/C=US/DC=vsphere/DC=local/O=psc.vmware.local/CN=CA
    ---
    Server certificate
    -----BEGIN CERTIFICATE-----
    MIIDWDCCAkCgAwIBAgIJANr+++MJ5+WxMA0GCSqGSIb3DQEBCwUAMGgxCzAJBgNV
    ...
    LqSKWg/apP1XlBV1VyC5LPZrH/rrq8+Naoj7i/P6HAzTwAAL+O10ggClaP8=
    -----END CERTIFICATE-----
     
  6. If you have more than one PSC in a vSphere domain, repeat this command using the FQDN on any remaining PSC nodes.

    echo | openssl s_client -connect psc2.vmware.local:443
     
  7. Using the output from the openssl s_client and the lstool.py, verify if the returned SSL certificates match for your vCenter Server with embedded Platform Services Controller. If they do match, you do not need to continue. If they do not match, proceed to Task 1: Retrieving the Old Certificate from the Managed Object Browser (MOB) to start updating the sslTrust anchors.
Validating the sslTrust Anchors from the Command Line on a Windows vCenter Server with Embedded PSC Installation
  1. Connect to the External Platform Services Controller with a Remote Desktop Session.
  2. Open an administrative command prompt.
  3. Run this command to get the current sslTrust anchor stored for the Platform Services Controller:

    "%VMWARE_PYTHON_BIN%" "%VMWARE_CIS_HOME%\VMware Identity Services\lstool\scripts\lstool.py" list --url https://localhost/lookupservice/sdk --no-check-cert --ep-type com.vmware.cis.cs.identity.sso 2> NULL

    For example:

    Note: SSL trust was truncated for readability.

    Service Product: com.vmware.cis
    Service Type: cs.identity
    Service ID: 04608398-1493-4482-881b-b35961bf5141
    Site ID: vmware
    Owner ID: [email protected]
    Version: 2.0
    Endpoints:
    Type: com.vmware.cis.cs.identity.sso
    Protocol: wsTrust
    URL: https://psc.vmware.local/sts/STSService/vsphere.local
    SSL trust: MIIDWDCCAkCgAwIBAgIJANr+++MJ5+WxMA0GCSqGSIb3DQEBCwUAMGgxCzAJBgNV ... LqSKWg/apP1XlBV1VyC5LPZrH/rrq8+Naoj7i/P6HAzTwAAL+O10ggClaP8=
     
  4. Run this command to get the current SSL certificate used on port 443 on the Platform Services Controller:

    "%VMWARE_OPENSSL_BIN%" s_client -connect localhost:443

    For example:

    Note: The certificate was truncated for readability.

    CONNECTED(00000003)
    depth=3 /DC=local/DC=VMWARE/CN=VMWARE-WCA-CA-1
    verify return:1
    depth=2 /DC=local/DC=VMWARE/CN=VMWARE-WCAI-CA-1
    verify return:1
    depth=1 /C=US/DC=vsphere/DC=local/O=psc.vmware.local/CN=CA
    verify return:1
    depth=0 /CN=psc.vmware.local/C=US
    verify return:1
    ---
    Certificate chain
    0 s:/CN=psc.vmware.local/C=US
    i:/C=US/DC=vsphere/DC=local/O=psc.vmware.local/CN=CA
    ---
    Server certificate
    -----BEGIN CERTIFICATE-----
    MIIDWDCCAkCgAwIBAgIJANr+++MJ5+WxMA0GCSqGSIb3DQEBCwUAMGgxCzAJBgNV
    ...
    LqSKWg/apP1XlBV1VyC5LPZrH/rrq8+Naoj7i/P6HAzTwAAL+O10ggClaP8=
    -----END CERTIFICATE-----
     
  5. Using the output from the openssl s_client and the lstool.py, verify if the outputted SSL certificates match for your vCenter Server with embedded Platform Services Controller. If they do match, you do not need to continue. If they do not match, proceed to Task 1: Retrieving the Old Certificate from the Managed Object Browser (MOB) to start updating the sslTrust anchors.

Task 1: Retrieving the Old Certificate

You can retrieve the old certificate using the Managed Object Browser (MOB) or from the backup store. Backup store contents changes after each certificate replacement operation; using the MOB is a more reliable option.
 
You can find certificate in the sslTrust field of the ArrayOfLookupServiceRegistrationInfo managed object by performing the following procedure:
  1. On the Platform Services controller, create a directory to store the old certificate. This article uses the following locations:
     
    Platform Services Controller Appliance/certificates
    Platform Services Controller on WindowsC:\certificates\
  2. To open the MOB, go to https://vc_with_embedded_psc.example.com/lookupservice/mob?moid=ServiceRegistration&method=List in a browser.
  3. Log in with the [email protected] username and password when prompted. If you are using a custom vCenter Single Sign-On domain, use that username and password.
  4. In the filterCriteria text field, leave only the tags <filterCriteria></filterCriteria> and click Invoke Method. The ArrayOfLookupServiceRegistrationInfo object is displayed.
  5. Search (Ctlr+F) for vc1.example.com on the page.
  6. Find the value of the corresponding sslTrust field. The content of that field is the Base64 encoded string of the old certificate. Any of the occurrences of vc1.example.com and Base64 encoded strings is acceptable.

    Use the following example as a model (The actual string was shortened significantly to improve legibility.).
     
    sslTrustArrayofStringMIIDfjCCAmag...
    urlanyURIhttps://vcenter.vmware.local:443/sdk
  7. Copy the Base64 encoded string to a file and save the file as old_machine.txt.
  8. Open old_machine.txt in a text editor.
  9. Append -----BEGIN CERTIFICATE----- to the beginning of the text string, and append -----END CERTIFICATE----- to the end of the text string. Add a carriage return after the 64th character of each line of the contents copied from the sslTrust field.

    For Example:

    -----BEGIN CERTIFICATE-----
    LIIDeDCCAmCgAwIBAgIJAP7kGwWSSd0yMA0GCSqGSIb3DQEBCwUAMGgxCzAJBgNV
    PAMMAkNBMRcwFQYKCZImiZPyLGQBGRYHdnNwaGVyZTEVMBMGCgmSJomT8ixkARkW
    QWxvY2FsMQswCQYDVQQGEwJVUzEcMBoGA1UECgwTaG9tZXBzYy5mcml0ei5sb2Nh
    NDAeFw0xNTA4MTAwMDMwMjZaFw0yNTA4MDQwMDMwMjVaMCsxHDAaBgNVBAMME2hv
    HWVwc2MuZnJpdHoubG9jYWwxCzAJBgNVBAYTAlVTMIIBIjANBgkqhkiG9w0BAQEF
    LAOCAQ8AMIIBCgKCAQEAzuf/uVMLwlkUKsMXsUPigqZdrXKzEOEzOQ04q8YgVvDX
    w7MAPSTMZzeUsI6P+/4doZU14zAQTl/6dnbwYg65p9mv7CVJb4QgAJH9xFD+33Ab
    aQX7za/bWPgyxsPtccnn+si8QQDx9mMZbDzF0gjdARvpKWwVv4lln8iZ8wUahyC7
    bxnzc5/oWo4Z3DTruHMnvadHRZWzZTn8YeID06R2g8Yu5c50wXbAvNj3TE4x0Qyv
    fUbABXvv2EdYC5tb3g++L6A6tuWYgl+dr4KJ1G5gLvliECAsWsMwtQXq5nH65JdV
    XvRUVIlajC9OavGkd+ziT3yRibJBu2NJrLQp7ehgmQIDAQABo2IwYDAeBgNVHREE
    FzAVghNob21lcHNjLmZyaXR6LmxvY2FsMB0GA1UdDgQWBBSaRwv8djR7+qg7Wk3A
    zib3C3ArljAfBgNVHSMEGDAWgBRkYn4wsyRye8o14OoE3AOTMus6rzANBgkqhkiG
    9w0BAQsFAAOCAQEAU3X/ZEDXO8yDRJkjrQH0acxoc76QRDv+3s6yCpPFU8HmqU1E
    LmoDq67rHoKZw5ziBR/lGHn5oVHYYuJRFdO/b8NO1t2MnedhAaenqmAr4v0FzH6K
    UCgiLq8+ZMPFBz3qFu2i0I8mG6Yy0ud9T4wWUabgZ1C3sDNkQ+NLHXKVxNrPwgQd
    3KyrNpXgBQ0+ZWY3xvvdW5yOwnWkeAeqnGRYvzifG9M6DK/YMP1S/akAJvXSgEkJ
    PEJ3vlvSRy7l2lvU19upt4O/BAk3ZJ+X5uFtv/4GMdbEVZBCmNDS7Y85NorISiQf
    AVy/R2wjP4rNWDfN9DMCcwfPvw/0nFwrpr+0Cg==
    -----END CERTIFICATE-----
     
  10. Save old_machine.txt as old_machine.crt.
  11. Move or upload the file to the Platform Services Controller to the location specified in Step 1.
     
    Platform Services Controller Appliance/certificates/old_machine.crt
    Platform Services Controller on WindowsC:\certificates\old_machine.crt
You can now extract the thumbprint from this file.

Task 2: Extracting the Thumbprint from the Old Certificate

You can extract the thumbprint from the command line or by using a certificate viewer tool. After you extract the certificate, you can upload it to the Platform Services Controller.
 
Extracting the Thumbprint from the Command Line on the Appliance
  1. Log in to the External Platform Services Controller Appliance via SSH.
  2. Run this command to enable access to the Bash shell:

    shell.set --enabled true
     
  3. Enter shell and press Enter.
  4. Run this command to get the thumbprint:

    openssl x509 -in /certificates/old_machine.crt -noout -sha1 -fingerprint
You see output similar to:

SHA1 Fingerprint=13:1E:60:93:E4:E6:59:31:55:EB:74:51:67:2A:99:F8:3F:04:83:88

The thumbprint, is the sequence of numbers and letters that follow the equal sign.


Extracting the Thumbprint from the Command Line on a Windows Installation
  1. Make a remote desktop connection to the External Platform Services Controller.
  2. Open an administrative command prompt.
  3. Run this command to get the thumbprint:

    "%VMWARE_OPENSSL_BIN%" x509 -in c:\certificates\old_machine.crt -noout -sha1 -fingerprint


    You see output similar to:

    SHA1 Fingerprint=13:1E:60:93:E4:E6:59:31:55:EB:74:51:67:2A:99:F8:3F:04:83:88

    The thumbprint is the sequence of numbers and letters that follow the equal sign.

Extracting the Thumbprint Using a Certificate Viewer Tool
 
You can extract the thumbprint by performing these steps:
  1. Open the file with a certificate viewer tool. In Windows, double-click the file to open it in Windows Certificate Viewer.
  2. Get the SHA1 Thumbprint string. In Windows Certificate Viewer, select the SHA1 Thumbprint field.
  3. Copy the thumbprint string into a text editor and replace the spaces with colons.

    Note: With some text editors, invisible characters are added at the beginning. Delete the first character of the thumbprint and any associated spaces, then type, not paste, the character.

Proceed to Task 3 to retrieve the new certificate.

Task 3: Retrieving the New Certificate

If you did not archive the new certificate, you can retrieve it using vecs-cli:

Retrieving the New Certificate on the vCenter Server Appliance

  1. Log in to the vCenter Server system through console or  and SSH session.
  2. Run this command to enable access to the Bash shell:

    shell.set --enabled true 
     
  3. Type shell and press Enter.
  4. Run this command to view the new certificate:

    /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT 
     
  5. Export the certificate to a file with this command:

    /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /certificates/new_machine.crt 

Retrieving the New Certificate on the vCenter Server on Windows Installation

  1. Make a remote desktop connection to the vCenter Server system.
  2. Open an administrative command prompt.
  3. Run this command to view the new certificate:

    "%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry list --store MACHINE_SSL_CERT --text |more 
     
  4. Export the certificate to a file with this command:

    "%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output c:\certificates\new_machine.crt 
     
  5. Move or upload the certificate to the Platform Services controller via WinSCP or another SCP client.
Proceed to Task 4 to execute the script with the information you gathered in Task 1-3

Task 4: Running the ls_update_certs.py Script

Run the ls_update_certs.py script on the Platform Services Controller after replacing the vCenter Server certificate. To successfully run the script, you must have both the thumbprint of the old vCenter Server certificate and the new vCenter Server certificate.
 
Warning: You cannot undo the actions of this script. Perform a backup or a snapshot of the virtual machine so you can recover if problems result.
 
Note: On Windows systems, enclose the password in double quotes.

Running ls_update_cert on the Appliance

The ls_update_certs script is located at /usr/lib/vmidentity/tools/scripts/ls_update_certs.py.

  1. Log in to the External Platform Services Controller Appliance through console or an SSH session.
  2. Run this command to enable access the Bash shell:

    shell.set --enabled true
     
  3. Enter shell and press Enter.
  4. Change directories to /usr/lib/vmidentity/tools/scripts/ with the following command:

    cd /usr/lib/vmidentity/tools/scripts/
     
  5. Run this command:

    python ls_update_certs.py --url Lookup_Service_FQDN_of_Platform_Services_Controller --fingerprint Old_Certificate_Fingerprint_from_Task_2 --certfile New_Certificate_Path_from_Task_3 --user [email protected] --password 'Password'

    For example (do not copy the fingerprint used in this example):

    python ls_update_certs.py --url https://psc.vmware.com/lookupservice/sdk --fingerprint 13:1E:60:93:E4:E6:59:31:55:EB:74:51:67:2A:99:F8:3F:04:83:88 --certfile /certificates/new_machine.crt --user [email protected] --password 'Password'

Running ls_update_cert on a Platform Services Controller Windows Installation

  1. Connect to the External Platform Services Controller with a Remote Desktop session and administrator permissions.
  2. Open an administrative command prompt.
  3. Change directories to C:\Program Files\VMware\vCenter Server\VMware Identity Services\lstool\scripts\ with this command:

    cd C:\Program Files\VMware\vCenter Server\VMware Identity Services\lstool\scripts\
     
  4. Run this command:

    "%VMWARE_PYTHON_BIN%" ls_update_certs.py --url Lookup_Service_FQDN_of_Platform_Services_Controller --fingerprint Old_Certificate_Fingerprint_from_Task_2 --certfile New_Certificate_Path_from_Task_3 --user Username --password Password

    For example (do not copy the fingerprint used in this example):

    "%VMWARE_PYTHON_BIN%" ls_update_certs.py --url https://psc.vmware.com/lookupservice/sdk --fingerprint 13:1E:60:93:E4:E6:59:31:55:EB:74:51:67:2A:99:F8:3F:04:83:88 --certfile c:\certificates\new_machine.crt --user [email protected] --password Password

Additional Information

Syntax for ls_update_cert

Run the script using the following syntax:

python ls_update_certs.py --url LS_URL --fingerprint OLD_CERT_SHA1_HASH --certfile NEW_CERT_PEM_FILEPATH --user USER --password PASSWORD
 

 

LS_URLLookup service URL. On the External Platform Services controller, use the following URL as a model:
https://external_platform_services_controller_FQDN.example.com/lookupservice/sdk
OLD_CERT_SHA1_HASH
Thumbprint of the certificate that vCenter Server or Platform Services Controller used before certificate replacement acquired in Task 2.
 
First you retrieve the old certificate:
  • If possible, download the current certificate from the vCenter Server system before you perform certificate replacement. For more information, see How to download and install vCenter Server root certificates to avoid Web Browser certificate warnings (2108294).
  • Otherwise, you can follow the process in Task 1: Retrieving the Old Certificate from the Managed Object Brower
  • If you only performed one certificate replacement operation, you can instead use the process in Retrieving the Old Certificate from the Managed Object Brower, listed under Additional Information.
Then you extract the thumbprint from the old certificate.
Note: VMware does not recommends to find the old vCenter Server certificate in the filesystem.
NEW_CERT_PEM_FILEPATH
PEM encoded file of the new vCenter Server machine SSL certificate acquired in Task 3.
 
Use the file that you just passed in as part of certificate replacement. If you no longer have that file, use the process in Retrieving the New Certificate.
 
Note: Attempting to find the new vCenter Server certificate in the filesystem is not recommended.
USER and PASSWORDUser with administrator privileges for vCenter Single Sign-On.

 

Retrieving the Old Certificate from the BACKUP_STORE

If you are using the vSphere Certificate Manager utility, you can retrieve the old machine SSL certificate from the BACKUP_STORE inside VECS.
 
Note: The backup store only keeps the last certificate. If you performed multiple replacement actions, you can instead retrieve the certificate from the MOB, as discussed below.
 
On the vCenter Server Appliance, you retrieve the old certificate thumbprint as follows:
  1. Run this command and look for the Machine_Cert entry to verify it is the previous certificate:

    /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store BACKUP_STORE --text
     
  2. Notice bkp___Machine_Cert under Machine_Cert.
  3. Run this command to output the Machine_Cert from the BACKUP_Store.

    /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store BACKUP_STORE --alias bkp___MACHINE_CERT --output /certificates/old_machine.crt
     
  4. Run this command to output the thumbprint:

    openssl certificate -fingerprint< -sha1 -noout /certificates/old_machine.crt -in x509>

    You see output similar to:

    SHA1 Fingerprint=11:41:9F:D8:CF:27:6B:EA:F7:49:20:3E:D7:90:8C:F6:A0:62:E1:31

On a vCenter Server Windows system, you retrieve the old certificate as follows:

  1. Run this command and look for the Machine_Cert entry to verify it is the previous certificate:

    "%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry list --store BACKUP_STORE –text | more
     
  2. To output the Machine_Cert from the BACKUP_Store, run this command:

    "%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store BACKUP_STORE --alias bkp___MACHINE_CERT --output c:\certificates\old_machine.crt
     
  3. Run this command to output the thumbprint:

    openssl certificate -fingerprint< -sha1 -noout c:\certificates\old_machine.crt -in x509>

    You see output similar to:

    SHA1 Fingerprint=11:41:9F:D8:CF:27:6B:EA:F7:49:20:3E:D7:90:8C:F6:A0:62:E1:31

Translations

For translated versions of this article, see:

Additional Information

Attachments

2121689_lstoolutil.zip get_app
Powered by Wolken Software