WCP Service fails to start after replacing vCenter Server certificates

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

The following messages appear in /var/log/vmware/wcp/wcpsvc.log

Failed to get EAM agencies. Err ServerFaultCode: EAM is still loading from database. Please try again later.

The following messages appear in /var/log/vmware/eam/eam.log.log

Failed to login to vCenter as extension. vCenter has probably not loaded the EAM extension.xml yet.: Cannot complete login due to an incorrect user name or password.

The SSL Certificates on the vCenter Appliance were recently replaced.

Environment

VMware vCenter Server 7.0.x

Cause

This issue is due to the certificate manager utility being unable to automatically update the EAM certificate when solution user certificates are updated. WCP requires EAM to be functional in order to start.

You do not need to be licensed for or using WCP/vSphere 7 with Kubernetes in order to be susceptible to this issue.

Resolution

VMware is aware of this issue and working to fix it in a future release.

Workaround:
To workaround this issue, run the updateExtensionCertInVC.py script as outlined below.

Log in to the vCenter Server Appliance using SSH.
Run this command to enable access the Bash shell:

shell.set --enabled true
Type shell and press Enter.
Run this command to retrieve the vpxd-extension solution user certificate and key:

mkdir /certificate

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.key
Run this command to update the extension's certificate with vCenter Server.

python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s localhost -u Administrator@domain.local

Note: If this produces the error "Hostname mismatch, certificate is not valid for 'localhost'", change 'localhost' to the FQDN or IP of the vCenter. The process is checking this value against the SAN entries of the certificate.

Note: The default user and domain is [email protected]. If this was changed during configuration, change the domain to match your environment. When prompted, type in the Administrator@domain.local password.

Restart EAM and start the rest of the services with these commands:

service-control --stop vmware-eam

service-control --start --all

Note: Please refer to the Related Information in this KB if you are receiving the error "certificate verify failed: Hostname mismatch, certificate is not valid for 'sdkTunnel'" during extension's certificate update (step 5).

Additional Information

This issue is also known to cause problems with NSX solutions connected to vCenter. See Deploying NSX VIBs fails on the ESXi host after changing the vCenter Server certificates for more information.
In certain situations, you might receive the error "certificate verify failed: Hostname mismatch, certificate is not valid for 'sdkTunnel'". This error can be safely ignored if you are getting the error after the message "Successfully updated certificate for "com.vmware.vim.eam" extension" as this message confirms that Extension certificate updated successfully with vCenter Server.

Password to connect to VC server for user="[email protected]":
2023-01-25T16:13:12.974Z Updating certificate for "com.vmware.vim.eam" extension
2023-01-25T16:13:13.116Z Successfully updated certificate for "com.vmware.vim.eam" extension
Traceback (most recent call last):
File "/usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py", line 175, in <module>
update_extension_cert_in_VC()
File "/usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py", line 163, in update_extension_cert_in_VC
sessionMgr = si.content.sessionManager
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 589, in __call__
return self.f(*args, **kwargs)
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 394, in _InvokeAccessor
return self._stub.InvokeAccessor(self, info)
File "/usr/lib/vmware/site-packages/pyVmomi/StubAdapterAccessorImpl.py", line 43, in InvokeAccessor
return self.InvokeMethod(mo, info, (prop, ))
File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1525, in InvokeMethod
conn.request('POST', self.path, req, headers)
File "/usr/lib/python3.7/http/client.py", line 1281, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.7/http/client.py", line 1327, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.7/http/client.py", line 1276, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.7/http/client.py", line 1036, in _send_output
self.send(msg)
File "/usr/lib/python3.7/http/client.py", line 976, in send
self.connect()
File "/usr/lib/python3.7/http/client.py", line 1451, in connect
server_hostname=server_hostname)
File "/usr/lib/python3.7/ssl.py", line 423, in wrap_socket
session=session
File "/usr/lib/python3.7/ssl.py", line 870, in _create
self.do_handshake()
File "/usr/lib/python3.7/ssl.py", line 1139, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'sdkTunnel'. (_ssl.c:1076)