Recovering expired SSL Certificates in VMware vCenter Server Appliance 5.5.x
search cancel

Recovering expired SSL Certificates in VMware vCenter Server Appliance 5.5.x

book

Article ID: 338184

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:
When the vCenter Server SSL Certificates expire, you experience these symptoms:
  • Cannot log in to vCenter Server using the vSphere Web Client
  • Logging in to vCenter Server using the vSphere Web Client fails
  • You see this error:
Cannot connect to vCenter Single Sign On server https://vc.domain.com:7444/ims/STSService?wsdl. The SSL certificate cannot be verified.

  • The VMware VirtualCenter Server service cannot start.
  • In the vpxd.log file, located at /var/log/vmware/vpx/vpxd.log, you see entries similar to:
<YYYY-DD-MM>< TIME> [03992 error 'HttpConnectionPool-000001'] [ConnectComplete] Connect failed to <cs p:0000000008165ed0, TCP:vc.domain.com:7444>; cnx: (null), error: class Vmacore::Ssl::SSLVerifyException(SSL Exception: Verification parameters:
--> PeerThumbprint: 39:D4:04:4A:FB:AC:8E:05:EC:45:22:81:3F:45:28:44:4C:C7:25:DF
--> ExpectedThumbprint:
--> ExpectedPeerName: vc.domain.com
--> The remote host certificate has these problems:
-->
--> * certificate has expired)
<YYYY-DD-MM>< TIME> [03884 error '[SSO][SsoFactory_CreateFacade]'] Unable to create SSO facade: SSL Exception: Verification parameters:
--> PeerThumbprint: 39:D4:04:4A:FB:AC:8E:05:EC:45:22:81:3F:45:28:44:4C:C7:25:DF
--> ExpectedThumbprint:
--> ExpectedPeerName: vc.domain.com
--> The remote host certificate has these problems:
-->
--> * certificate has expired.
<YYYY-DD-MM>< TIME> [03884 error 'vpxdvpxdMain'] [Vpxd::ServerApp::Init] Init failed: Vpx::Common::Sso::SsoFactory_CreateFacade(sslContext, ssoFacadeConstPtr)
--> Backtrace:
--> backtrace[00] rip 000000018018b86a
--> backtrace[01] rip 0000000180102ac8
--> backtrace[02] rip 0000000180103f9e
--> backtrace[03] rip 000000018008d22b
--> backtrace[04] rip 00000000004e5bdc
--> backtrace[05] rip 0000000000506652
--> backtrace[06] rip 00007ff71e14f001
--> backtrace[07] rip 00007ff71e148e1c
--> backtrace[08] rip 00007ff71e36d8db
--> backtrace[09] rip 00007ffe927381d5
--> backtrace[10] rip 00007ffe927b16ad
--> backtrace[11] rip 00007ffe92a94409
-->
<YYYY-DD-MM>< TIME> [03884 warning 'VpxProfiler'] ServerApp::Init [TotalTime] took 5015 ms
<YYYY-DD-MM>< TIME> [03884 error 'Default'] Failed to intialize VMware VirtualCenter. Shutting down...
<YYYY-DD-MM>< TIME> [03884 info 'vpxdvpxdSupportManager'] Wrote uptime information


Environment

VMware vCenter Server Appliance 5.5.x

Resolution

Notes:
  • This article assumes that you have prepared new and valid SSL Certificates. If you have not yet prepared new certificates, then see the Generating the certificate requests and Getting the certificates sections of the Configuring Certificate Authority (CA) signed certificates for vCenter Server Appliance 5.5 (2057223).
  • This article uses examples based on the new certificates being stored in these directories:
    • /ssl/vpxd/
    • /ssl/inventory/
    • /ssl/logbrowser/
    • /ssl/autodeploy/

  • Before continuing, ensure that you have a valid working backup of the vCenter Server Appliance. Take a snapshot of the vCenter Server Appliance virtual machine before proceeding.
To resolve this issue, recover the expired SSL certificates in VMware vCenter Server Appliance 5.5.x:

To recover the expired SSL certificates:
  1. Run this command to back up the vpxd.cfg file:

    cp /etc/vmware-vpx/vpxd.cfg /etc/vmware-vpx/vpxd.cfg.backup
  2. Edit the /etc/vmware-vpx/vpxd.cfg file using vi editor.

    To edit the vpxd.cfg file:

    1. Open the vi /etc/vmware-vpx/vpxd.cfg file using vi editor.
    2. Locate this line:

      <enabled>true</enabled>

    3. Press i to change to INSERT mode.
    4. Change this line to:

      <enabled>false<enabled>

    5. Press ESC :wq ENTER to save the file and quit.

  3. Replace the Single Sign-on and VMware VirtualCenter Server service certificate:

    To replace the Single Sign-on and VMware VirtualCenter Server service certificate:

    1. Stop the VMware VirtualCenter Server service and the vCenter Single Sign-On service using these commands:

      service vmware-stsd stop
      service vmware-vpxd stop

    2. Create the chain.pem file for VMware VirtualCenter Server service by running this command:

      cat /ssl/vpxd/rui.crt /ssl/vpxd/cachain.pem > /ssl/vpxd/chain.pem
    3. Replace the SSL certificate by running this command:

      /usr/sbin/vpxd_servicecfg certificate change /ssl/vpxd/chain.pem /ssl/vpxd/rui.key

      Wait until you receive this response:

      VC_CFG_RESULT = 0

  4. Revert the changes to the vpxd.cfg file performed in Step 2.

    To edit the vpxd.cfg file:

    1. Open the vi /etc/vmware-vpx/vpxd.cfg file using vi editor.
    2. Locate this line:

      <enabled>false</enabled>

    3. Press i to change to INSERT mode.
    4. Change this line to:

      <enabled>true<enabled>

    5. Press ESC :wq ENTER to save the file and quit.

  5. Run this command to un-register the VMware VirtualCenter Server serviceID from Single Sign-On:

    /usr/lib/vmware-sso/bin/vi_regtool unregisterService -d https://vcenter_FQDN:7444/lookupservice/sdk -u [email protected] -p PASSWORD -si /etc/vmware-vpx/ls-service-id
  6. Run this command to check and note the vCenter Server Solution User:

    grep name /etc/vmware-vpx/vpxd.cfg
    <name>vpxd-vcva55.domain.com-7032d8b5-63a2-4e8d-b025-a07f8852a75f</name>
  7. Run this command to un-register the vCenter Server Solution User from Single Sign-On:

    /usr/lib/vmware-sso/bin/vi_regtool unregisterSolution -d https://vcenter_FQDN:7444/lookupservice/sdk -u [email protected] -p PASSWORD -su vpxd-vcva55.domain.com-e8b409a5-40da-4353-8546-48eaf7608045
  8. Run this command to re-register vCenter Server with Single Sign-On:

    /etc/vmware-sso/register-hooks.d/01-vcenter --mode install --ls-server https://vcenter_FQDN:7444/lookupservice/sdk --user [email protected] --password PASSWORD --option sso-deployment-type=embedded --option ls-certificate-thumbprint=null --option vc-admin-principal=root --option vc-admin-is-group=false
  9. Run this command to check and note the Inventory Service Solution User:

    grep dataservice.sso.solutionUser /usr/lib/vmware-vpx/inventoryservice/lib/server/config/dataservice.properties
    dataservice.sso.solutionUser=inventory-service-162e5b30-59e9-4f9d-82ac-2718e186287f
  10. Run this command to un-register the Inventory Service Solution User:

    /usr/lib/vmware-sso/bin/vi_regtool unregisterSolution -d https://vcenter_FQDN:7444/lookupservice/sdk -u [email protected] -p PASSWORD -su inventory-service-162e5b30-59e9-4f9d-82ac-2718e186287f
  11. Replace the Inventory Service certificate.

    To replace the Inventory Service certificate:

    1. Create the chain.pem file for the Inventory Service by running this command:

      cat /ssl/inventory/rui.crt /ssl/inventory/cachain.pem > /ssl/inventory/chain.pem
    2. Create the *.pfx file by running this command:

      openssl pkcs12 -export -out /ssl/inventory/rui.pfx -in /ssl/inventory/chain.pem -inkey /ssl/inventory/rui.key -name rui -passout pass:testpassword
    3. Run this command to copy the rui.key, rui.crt, and rui.pfx files to the /usr/lib/vmware-vpx/inventoryservice/ssl directory:

      cp /ssl/inventory/rui.* /usr/lib/vmware-vpx/inventoryservice/ssl/.

  12. Change the permissions on these files by running these commands:

    chmod 400 /usr/lib/vmware-vpx/inventoryservice/ssl/rui.key /usr/lib/vmware-vpx/inventoryservice/ssl/rui.pfx
    chmod 644 /usr/lib/vmware-vpx/inventoryservice/ssl/rui.crt
  13. Run this command to re-register the Inventory Service with Single Sign-On:

    /etc/vmware-sso/register-hooks.d/02-inventoryservice --mode install --ls-server https://vcenter_FQDN:7444/lookupservice/sdk --user [email protected] --password PASSWORD
  14. Run this command to force a re-register of the Inventory Service with the vCenter Server on next restart of the VMware VirtualCenter Server service:

    rm /var/vmware/vpxd/inventoryservice_registered
  15. Run this command to restart the VMware VirtualCenter Server service:

    service vmware-vpxd restart
  16. Run this command to list the services registered to Single Sign-On and check and note the Log Browser ServiceID and Solution User (ownerID):

    /usr/lib/vmware-sso/bin/vi_regtool listServices https://vcenter_FQDN:7444/lookupservice/sdk

    Service 6
    -----------
    serviceId=local:b23d652b-eed8-4737-9638-2867abd9fd0a
    serviceName=VMware Log Browser
    type=urn:logbrowser:logbrowser
    endpoints={[url=https://vcva55.domain.com:12443/vmwb/logbrowser,protocol=unknown],[url=https://vcva55.domain.com:12443/authentication/authtoken,protocol=unknown]}
    version=1.0.2175565
    description=Enables browsing vSphere log files within the VMware Web Client
    ownerId=logbrowser-vcva55.domain.com-23bc85a0-894c-435b-a3e8-19be1e371e4c
    productId=
    viSite=local
    Return code is: Success

  17. Run this command to create a ServiceID File for the Log Browser service:

    echo local:b23d652b-eed8-4737-9638-2867abd9fd0a >> /tmp/logbrowser_id

  18. Run this command to un-register the Log Browser ServiceID from Single Sign-On:

    /usr/lib/vmware-sso/bin/vi_regtool unregisterService -d https://vcenter_FQDN:7444/lookupservice/sdk -u [email protected] -p PASSWORD -si /tmp/logbrowser_id

  19. Run this command to un-register the Log Browser Solution User from Single Sign-On:

    /usr/lib/vmware-sso/bin/vi_regtool unregisterSolution -d https://vcenter_FQDN:7444/lookupservice/sdk -u [email protected] -p PASSWORD -su logbrowser-vcva55.domain.com-23bc85a0-894c-435b-a3e8-19be1e371e4c

  20. Replace the Log Browser certificate.

    To replace the Log Browser certificate:

    1. Create the chain.pem file for VMware Log Browser Service by running this command:

      cat /ssl/logbrowser/rui.crt /ssl/logbrowser/cachain.pem > /ssl/logbrowser/chain.pem

    2. Create the *.pfx file by running this command:

      openssl pkcs12 -export -out /ssl/logbrowser/rui.pfx -in /ssl/logbrowser/chain.pem -inkey /ssl/logbrowser/rui.key -name rui -passout pass:testpassword

    3. Run this command to copy the rui.key, rui.crt, and rui.pfx files to the /usr/lib/vmware-logbrowser/conf/ directory:

      cp rui.* /usr/lib/vmware-logbrowser/conf/

    4. Change the permissions on these files by running these commands:

      chmod 400 /usr/lib/vmware-logbrowser/conf/rui.key /usr/lib/vmware-logbrowser/conf/rui.pfx
      chmod 644 /usr/lib/vmware-logbrowser/conf/rui.crt

  21. Run this command to re-register the Log Browser with Single Sign-On:

    /etc/vmware-sso/register-hooks.d/09-vmware-logbrowser --mode install --ls-server https://vcenter_FQDN:7444/lookupservice/sdk --user [email protected] --password PASSWORD

  22. Run this command to restart the Log Browser service:

    service vmware-logbrowser restart

  23. Run this command to restart the vSphere Web Client service:

    service vsphere-client restart

    Note: Steps 24 to 27 are optional and do not need to be completed unless you are using Auto Deploy.

  24. Edit the /etc/vmware-rbd/autodeploy-setup.xml file:

    Change <serviceAddress>127.0.0.1</serviceAddress> so that it states the FQDN of the vCenter Server Appliance <serviceAddress>vcenter_FQDN</serviceAddress>

  25. Replace the Auto Deploy certificate.

    To replace the Auto Deploy certificate:

    1. Run these commands to copy and rename the rui.key, rui.crt files to /etc/vmware-rbd/ssl/:

      cp /ssl/autodeploy/rui.crt /etc/vmware-rbd/ssl/waiter.crt
      cp /ssl/autodeploy/rui.key /etc/vmware-rbd/ssl/waiter.key

    2. Change the permissions on these files by running these commands:

      chmod 644 /etc/vmware-rbd/ssl/waiter.crt
      chmod 400 /etc/vmware-rbd/ssl/waiter.key
      chown deploy:deploy /etc/vmware-rbd/ssl/waiter.crt /etc/vmware-rbd/ssl/waiter.key

  26. Run this command to re-register the Auto Deploy Service with the VMware VirtualCenter Server service:

    service vmware-rbd-watchdog stop
    rm /var/vmware/vpxd/autodeploy_registered
    service vmware-vpxd restart

    Note: The autodeploy_registered file may not exist.

  27. Run this command to reboot the vCenter Server Appliance:

    reboot

  28. After the vCenter Server Appliance is fully booted, open a Web Browser to these URLs and verify the certificate presented:

    Single Sign-on - https://vcenter_FQDN:7444
    Inventory Service - https://vcenter_FQDN:10443
    vCenter Server - https://vcenter_FQDN:443
    Log Browser - https://vcenter_FQDN:12443
    vSphere Web Client - https://vcenter_FQDN:9443
    VAMI - https://vcenter_fqdn:5480/


Additional Information

Configuring Certificate Authority (CA) signed certificates for vCenter Server Appliance 5.5

Powered by Wolken Software