Recovering from expired SSL Certificates in VMware vCenter Server 5.5
search cancel

Recovering from expired SSL Certificates in VMware vCenter Server 5.5

book

Article ID: 335949

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

This article provides steps to recover when connecting to SSO fails with cert error.


Symptoms:
After the SSL Certificates expire, you experience these symptoms:
  • Unable to log in to vCenter Server using the vSphere Web Client.
  • You see the error:
Cannot connect to vCenter Single Sign On server https://vc.domain.com:7444/ims/STSService?wsdl. The SSL certificate cannot be verified.
  • The VMware VirtualCenter Server service is unable to start.
  • In the C:\ProgramData\VMware\VMware VirtualCenter\Logs\ vpxd.log file, you see entries similar to:
<YYYY-MM-DD>T<time> [03992 error 'HttpConnectionPool-000001'] [ConnectComplete] Connect failed to <cs p:0000000008165ed0, TCP:vc.domain.com:7444>; cnx: (null), error: class Vmacore::Ssl::SSLVerifyException(SSL Exception: Verification parameters:</time>
--> PeerThumbprint: 39:D4:04:4A:FB:AC:8E:05:EC:45:22:81:3F:45:28:44:4C:C7:25:DF
--> ExpectedThumbprint:
--> ExpectedPeerName: vc.domain.com
--> The remote host certificate has these problems:
-->
--> * certificate has expired)
<YYYY-MM-DD>T<time></time> [03884 error '[SSO][SsoFactory_CreateFacade]'] Unable to create SSO facade: SSL Exception: Verification parameters:
--> PeerThumbprint: 39:D4:04:4A:FB:AC:8E:05:EC:45:22:81:3F:45:28:44:4C:C7:25:DF
--> ExpectedThumbprint:
--> ExpectedPeerName: vc.domain.com
--> The remote host certificate has these problems:
-->
--> * certificate has expired.
<YYYY-MM-DD>T<time></time> [03884 error 'vpxdvpxdMain'] [Vpxd::ServerApp::Init] Init failed: Vpx::Common::Sso::SsoFactory_CreateFacade(sslContext, ssoFacadeConstPtr)
--> Backtrace:
--> backtrace[00] rip 000000018018b86a
--> backtrace[01] rip 0000000180102ac8
--> backtrace[02] rip 0000000180103f9e
--> backtrace[03] rip 000000018008d22b
--> backtrace[04] rip 00000000004e5bdc
--> backtrace[05] rip 0000000000506652
--> backtrace[06] rip 00007ff71e14f001
--> backtrace[07] rip 00007ff71e148e1c
--> backtrace[08] rip 00007ff71e36d8db
--> backtrace[09] rip 00007ffe927381d5
--> backtrace[10] rip 00007ffe927b16ad
--> backtrace[11] rip 00007ffe92a94409
-->
<YYYY-MM-DD>T<time></time> [03884 warning 'VpxProfiler'] ServerApp::Init [TotalTime] took 5015 ms
<YYYY-MM-DD>T<time></time> [03884 error 'Default'] Failed to intialize VMware VirtualCenter. Shutting down...
<YYYY-MM-DD>T<time></time> [03884 info 'vpxdvpxdSupportManager'] Wrote uptime information

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.


Environment

VMware vCenter Server 5.5.x

Resolution

This article assumes that you have already prepared new and valid SSL Certificates for all vCenter Server 5.5 components.
 
If you have not yet prepared new and valid SSL Certificates, see Generating certificates for use with the VMware SSL Certificate Automation Tool (2044696).
 
Note: This article uses examples based on the new and valid certificates being stored in the C:\Certs\Service\ directory structure.
 
You may need to alter the commands to suit your environment.
 
Before proceeding you need to manually create the rui.pfx files for the vCenter Server, vSphere Web Client, and the Log Browser service as VMware do not leverage the VMware SSL Certificate Automation Tool for these services.
 
  1. Open an elevated command prompt as an Administrator.
  2. Change directory to the location of the OpenSSL Binaries. VMware use the OpenSSL binaries installed to the Inventory Service Installation Directory.

    cd "C:\Program Files\VMware\Infrastructure\Inventory Service\bin"
     
  3. Create a PFX File by running the OpenSSL command:

    openssl pkcs12 -export -in C:\Certs\<Service>\chain.pem -inkey C:\Certs\<Service>\rui.key -name "rui" -passout pass:testpassword -out C:\Certs\<Service>\rui.pfx

    Notes:
    • Repeat the preceding command until you have created a rui.pfx file for vCenter Server, vSphere Web Client and the Log Browser service.
    • The password of testpassword should not be changed.
       
  4. Set the JAVA and PATH environment variables by running these two commands:

    SET JAVA_HOME=C:\Program Files\Common Files\VMware\VMware vCenter Server - Java Components
    SET PATH=%PATH%;C:\Program Files\VMware\Infrastructure\VMware\CIS\vmware-sso;%JAVA_HOME%\bin

     
  5. Launch the vCenter SSL Automation Tool, ssl-updater.bat file, and run these tasks:
     
    1. Update the Single Sign-On SSL Certificate
    2. Update the Inventory Service Trust to Single Sign-On
    3. Update the Inventory Service SSL Certificate
    4. Update the vCenter Server Trust to Single Sign-On

      Note: Do not close the SSL Automation Tool at this time, you can return the the tool later.
       
  6. Place the new vCenter Server service certificates at C:\ProgramData\VMware\Virtual Center\SSL\:

    mkdir "C:\ProgramData\VMware\VMware VirtualCenter\SSL\old"
    move "C:\ProgramData\VMware\VMware VirtualCenter\SSL\rui*"
    "C:\ProgramData\VMware\VMware VirtualCenter\SSL\old"
    copy C:\Certs\vCenterServer\rui.* "C:\ProgramData\VMware\VMware VirtualCenter\SSL\"

     
  7. Rehash the vCenter Server service database password by running this command:

    cd "C:\Program Files\VMware\Infrastructure\VirtualCenter Server\"
    vpxd.exe -p


    Note: When prompted enter the password for the account that vCenter Server uses to communicate with the vCenter Server Database.
     
  8. List the services registered to Single Sign-On by running this command:

    ssolscli listServices https://vc55.domain.com:7444/lookupservice/sdk

    Service 6
    -----------
    serviceId={715F8796-C93B-4F8D-ABD0-7B4EE6CDA9B3}:26
    serviceName=vCenterService
    type=urn:vc
    endpoints={[url=https://vc51.domain.com:443/sdk,protocol=vmomi]}
    version=5.1
    description=vCenter Server
    ownerId=vCenterServer_XXXX.XX.XX_XXXXXX@System-Domain
    productId=<null>
    viSite={715F8796-C93B-4F8D-ABD0-7B4EE6CDA9B3}

     
  9. Check and note the ownerID for the vCenter Server Service:

    vCenterServer_XXXX.XX.XX_XXXXXX

    Note: Do not include ownerId= or @vsphere.local.
     
  10. Unregister vCenter Server serviceID from Single Sign-On by running this command:

    ssolscli unregisterService -d https://vc55.domain.com:7444/lookupservice/sdk -u [email protected] -p VMware123$ -si "C:\ProgramData\VMware\VMware VirtualCenter\LS_ServiceID.prop"
     
  11. Unregister vCenter Server SolutionUser from Single Sign-On by running this command:

    ssolscli unregisterSolution -d https://vc55.domain.com:7444/lookupservice/sdk -u [email protected] -p VMware123$ -su vCenterServer_XXXXXXXX
     
  12. Re-register vCenter Server back to Single Sign-On by running this command:

    Unzip sso_svccfg.zip located at "C:\Program Files\VMware\Infrastructure\VirtualCenter Server\ssoregtool\"

    cd "C:\Program Files\VMware\Infrastructure\VirtualCenter Server\ssoregtool\sso_svccfg"

    repoint.cmd configure-vc --lookup-server https://vc55.domain.com:7444/lookupservice/sdk --user [email protected] --password VMware123$ --openssl-path "C:\Program Files\VMware\Infrastructure\Inventory Service\bin/"


    Note: The command completes but reports that the VMware VirtualCenter Server service could not be restarted. This is expected at this point. Continue with the next step.
     
  13. The repoint.cmd command blanks the certificate and privatekey fields in the vpxd.cfg file. Repopulate the vpxd.cfg file with the correct paths.

    copy "C:\ProgramData\VMware\VMware VirtualCenter\vpxd.cfg" "C:\ProgramData\VMware\VMware VirtualCenter\vpxd.cfg.backup"

    notepad "C:\ProgramData\VMware\VMware VirtualCenter\vpxd.cfg"

    Find the <certificate> and <privateKey> tags as below
    <solutionUser>
    <certificate>null</certificate>
    <name>vCenterServer_XXXX.XX.XX_XXXXXX</name>
    <privateKey>null</privateKey>
    </solutionUser>
    Replace "null" with the correct paths to the vCenter Server rui.crt and rui.key
    <solutionUser>
    <certificate>C:\ProgramData\VMware\VMware VirtualCenter\ssl\rui.crt</certificate>
    <name>vCenterServer_XXXX.XX.XX_XXXXXX</name>
    <privateKey>C:\ProgramData\VMware\VMware VirtualCenter\ssl\rui.key</privateKey>
    </solutionUser>


    Note: If the preceding tags do not exist, add them.
     
  14. Start the VMware VirtualCenter Server service by running this command:

    net start vpxd
     
  15. Return the vCenter SSL Automation Tool, ssl-updater.bat file, and then run these tasks:
     
    1. Update the vCenter Server Trust to Inventory Service
    2. Update the Inventory Service Trust to vCenter Server
    3. Update the vCenter Orchestrator Trust to Single Sign-On
    4. Update the vCenter Orchestrator Trust to vCenter Server
    5. Update the vCenter Orchestrator SSL Certificate

      Note: Orchestrator tasks are optional depending on whether you use the component or not.
       
  16. List the services registered to Single Sign-On by running this command:

    ssolscli listServices https://vc55.domain.com:7444/lookupservice/sdk

    Identify the Services for both Log Browser and vSphere Web Client

    Service 5

    -----------
    serviceId=
    Default-First-Site:f0c6df23-47bb-47de-ab4f-2e3de4f65bcf
    serviceName=VMware Log Browser
    type=urn:logbrowser:logbrowser
    endpoints={[url=https://vc55.domain.com:12443/vmwb/logbrowser,protocol=unknown],[url=https://vc55.domain.com:12443/authentication/authtoken,protocol=unknown]}version=1.0.2175565
    description=Enables browsing vSphere log files within the VMware Web Client
    ownerId= WebClient_XXXX.XX.XX_XXXXXX
    productId=
    viSite=Default-First-Site

    Service 6
    -----------

    serviceId=
    Default-First-Site:37a10eec-7d36-415a-9266-507b5dee824c
    serviceName=VMware vSphere Web Client
    type=urn:com.vmware.vsphere.client
    endpoints={[url=https://vc55.domain.com:9443/vsphere-client,protocol=vmomi]}
    version=5.5
    description=VMware vSphere Web Client Service
    ownerId=
    WebClient_XXXX.XX.XX_XXXXXX
    productId=
    viSite=Default-First-Site

     
  17. Check and note the ownerID for the VMware vSphere Web Client Service:

    WebClient_XXXX.XX.XX_XXXXXX
     
  18. Create service_id files for both the Log Browser and vSphere Web Client by running these commands:

    echo Default-First-Site:f0c6df23-47bb-47de-ab4f-2e3de4f65bcf >> logbrowser_id
    echo Default-First-Site:37a10eec-7d36-415a-9266-507b5dee824c >> webclient_id

     
  19. Unregister Log Browser serviceID from Single Sign-On by running this command:

    ssolscli unregisterService -d https://vc55.domain.com:7444/lookupservice/sdk -u [email protected] -p VMware123$ -si logbrowser_id
     
  20. Unregister vSphere Web Client serviceID from Single Sign-On by running this command:

    ssolscli unregisterService -d https://vc55.domain.com:7444/lookupservice/sdk -u [email protected] -p VMware123$ -si webclient_id
     
  21. Unregister vSphere Web Client SolutionUser from Single Sign-On by running this command:

    ssolscli unregisterSolution -d https://vc55.domain.com:7444/lookupservice/sdk -u [email protected] -p VMware123$ -su WebClient_XXXX.XX.XX_XXXXXX

    Note: There is only 1 Solution User for both the Web Client and Log Browser services.
     
  22. Copy the new Log Browser and vSphere Web Client certificates to their respective locations:

    mkdir "C:\ProgramData\VMware\vSphere Web Client\ssl\old"
    move "C:\ProgramData\VMware\vSphere Web Client\ssl\rui*"
    "C:\ProgramData\VMware\vSphere Web Client\ssl\old"
    Copy "C:\Certs\vCenterWebClient\rui*" "C:\ProgramData\VMware\vSphere Web Client\ssl\"

    mkdir "C:\Program Files\VMware\Infrastructure\vSphereWebClient\logbrowser\conf\old"

    move "C:\Program Files\VMware\Infrastructure\vSphereWebClient\logbrowser\conf\rui*" "C:\Program Files\VMware\Infrastructure\vSphereWebClient\logbrowser\conf\old"

    copy "C:\Certs\vCenterLogBrowser\rui*" "C:\Program Files\VMware\Infrastructure\vSphereWebClient\logbrowser\conf\"


     
  23. Re-register the Log Browser and vSphere Web Client back to Single Sign-On:

    cd C:\Program Files\VMware\Infrastructure\vSphereWebClient\scripts

    client-repoint.bat https://vc55.domain.com:7444/lookupservice/sdk "[email protected]" "VMware123$"

     
  24. Open a Web Browser to these URLs and verify the certificate presented:

    Single Sign-on https://vc55.domain.com:7444/lookupservice/sdk
    Inventory Service https://vc55.domain.com:10443
    vCenter Server https://vc55.domain.com:443
    vRealize Orchestrator https://vc55.domain.com:8281

    Note: This service may not be running or not in use.

    Log Browser https://vc55.domain.com:12443
    vSphere Web Client https://vc55.domain.com:9443
 
 
 
 


Additional Information

How to regenerate vSphere 6.x certificates using self-signed VMCA 
This article is specifically for vCenter Server 5.5.
To resolve this issue in vCenter Server 5.1, see Recovering from expired SSL Certificates in vCenter Server 5.1 (2097692).
Implementing CA signed SSL certificates with vSphere 5.x
Generating certificates for use with the VMware SSL Certificate Automation Tool
Deploying and using the SSL Certificate Automation Tool 5.5
Recovering from expired SSL Certificates in VMware vCenter Server 5.1
VMware vCenter Server 5.5 において、SSL 証明書が期限切れになった場合の修復方法
Logging in to vSphere web client fails with error: The login request has expired due to a clock synchronization issue between vSphere Web Client and vCenter Single Sign-On server
vSphere Web Client へのログインが次のエラーで失敗する: The login request has expired due to a clock synchronization issue between vSphere Web Client and vCenter Single Sign-On server
从 VMware vCenter Server 5.5 中过期的 SSL 证书进行恢复