After replacing the VMware vCenter Server certificates in VMware vSphere 6.0, the VMware vSphere Auto Deploy solution user fails to log in

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:
After replacing certificates on VMware vCenter Server, you experience these symptoms:

In the /var/log/vmware/sca/sca.log or C:\ProgramData\VMware\vCenterServer\logs\sca\sca .log files for the vSphere Auto Deploy service (rbd ), you see entries similar to:

2015-07-01T05:58:17.523-04:00 [pool-5-thread-21 WARN com.vmware.sca.health.HealthStatusRequest] requestHealthStatusFromEndpoint: Failed to request health status (service:'rbd', url:https://vCenter_Server_FQDN:6502/vmw/rbd/healthStatus)

javax.net.ssl.SSLException: hostname in certificate didn't match: <vCenter.vmware.local> != <vpxd-extension>

2015-05-04T07:59:29.815 [37068]ERROR:rbd_watchdog_windows:caught exception in thread Feedback
Traceback (most recent call last):
File "rbd_watchdog_windows.pyc", line 50, in infiniteLoop
File "rbd_watchdog_windows.pyc", line 64, in feedbackServer
File "rbd\waiter\feedback.pyc", line 52, in __init__
File "pyVmomi\VmomiSupport.pyc", line 543, in <lambda>
File "pyVmomi\VmomiSupport.pyc", line 352, in _InvokeMethod
File "pyVmomi\SoapAdapter.pyc", line 1270, in InvokeMethod
vim.fault.InvalidLogin: (vim.fault.InvalidLogin) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
msg = "Can not make a connection because the username or password is incorrect.",
faultCause = <unset>,
faultMessage = (vmodl.LocalizableMessage) []
}
2015-05-04T07:59:31.487 [36744]ERROR:rbd_watchdog_windows:caught exception in thread VC-Monitor
Traceback (most recent call last):
File "rbd_watchdog_windows.pyc", line 50, in infiniteLoop
File "rbd_watchdog_windows.pyc", line 58, in vcMonitor
File "rbd\waiter\vc_monitor.pyc", line 48, in __init__
File "pyVmomi\VmomiSupport.pyc", line 543, in <lambda>
File "pyVmomi\VmomiSupport.pyc", line 352, in _InvokeMethod
File "pyVmomi\SoapAdapter.pyc", line 1270, in InvokeMethod
vim.fault.InvalidLogin: (vim.fault.InvalidLogin) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
msg = "Can not make a connection because the username or password is incorrect.",
faultCause = <unset>,
faultMessage = (vmodl.LocalizableMessage) []
}
2015-05-04T07:59:34.838 [37068]INFO:rbd_watchdog_windows:starting Feedback
2015-05-04T07:59:34.838 [37068]INFO:vc_servers:client SSL material -- C:\ProgramData\VMware\vCenterServer\data\autodeploy\ssl\waiter.key, C:\ProgramData\VMware\vCenterServer\data\autodeploy\ssl\waiter.crt
2015-05-04T07:59:36.733 [36744]INFO:rbd_watchdog_windows:starting VC-Monitor
2015-05-04T07:59:36.733 [36744]INFO:vc_servers:client SSL material -- C:\ProgramData\VMware\vCenterServer\data\autodeploy\ssl\waiter.key, C:\ProgramData\VMware\vCenterServer\data\autodeploy\ssl\waiter.crt
2015-05-04T07:59:37.862 [37068]ERROR:rbd_watchdog_windows:caught exception in thread Feedback
Traceback (most recent call last):
File "rbd_watchdog_windows.pyc", line 50, in infiniteLoop
File "rbd_watchdog_windows.pyc", line 64, in feedbackServer
File "rbd\waiter\feedback.pyc", line 52, in __init__
File "pyVmomi\VmomiSupport.pyc", line 543, in <lambda>
File "pyVmomi\VmomiSupport.pyc", line 352, in _InvokeMethod
File "pyVmomi\SoapAdapter.pyc", line 1270, in InvokeMethod
vim.fault.InvalidLogin: (vim.fault.InvalidLogin) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
msg = "Can not make a connection because the username or password is incorrect.",
faultCause = <unset>,
faultMessage = (vmodl.LocalizableMessage) []
}
2015-05-04T07:59:39.763 [36744]ERROR:rbd_watchdog_windows:caught exception in thread VC-Monitor
Traceback (most recent call last):
File "rbd_watchdog_windows.pyc", line 50, in infiniteLoop
File "rbd_watchdog_windows.pyc", line 58, in vcMonitor
File "rbd\waiter\vc_monitor.pyc", line 48, in __init__
File "pyVmomi\VmomiSupport.pyc", line 543, in <lambda>
File "pyVmomi\VmomiSupport.pyc", line 352, in _InvokeMethod
File "pyVmomi\SoapAdapter.pyc", line 1270, in InvokeMethod
vim.fault.InvalidLogin: (vim.fault.InvalidLogin) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
msg = "Can not make a connection because the username or password is incorrect.",
faultCause = <unset>,
faultMessage = (vmodl.LocalizableMessage) []
}
In Health Messages under System Configuration > Auto Deploy > Summary > Auto Deploy Summary tab, you see these errors:

Failed to request health status from URI https://vCenter_Server_FQDN:6502/vmw/rbd/healthStatus.
AutoDeploy Service is not running. Enable AutoDeploy and refresh.

Environment

VMware vSphere 6.5.x
VMware vSphere 6.x
VMware vSphere 6.7.x

Cause

This issue occurs when the VMware vSphere Auto Deploy (rbd ) service is not aware of the new certificate after replacing the solution user certificates on VMware vCenter Server.

Resolution

This issue is resolved in VMware vCenter Server 6.0 U1b, available at VMware Downloads.

You can work around this issue by updating the extension's certificate with vCenter Server.

To update the extension's certificate in vCenter Server for Windows:

Connect to vCenter Server as an administrative user through a console or remote desktop session.
Open an elevated command prompt.
Run this command to retrieve the vpxd-extension solution user certificate and key:

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output c:\certificates\vpxd-extension.crt

"%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output c:\certificates\vpxd-extension.key
Navigate to C:\Program Files\VMware\vCenter Server\vpxd\scripts:

cd C:\Program Files\VMware\vCenter Server\vpxd\scripts

Note: The path listed is for a default install of vCenter Server. If you have customized the install location of vCenter Server, change the directory accordingly.
Run this command to update the extension's certificate with vCenter Server:
For RBD service:
python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.rbd -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s vcsa_FQDN -u [email protected]

For ImageBuilder Service (applicable 6.5.x and onward):
python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.imagebuilder -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s vcsa_FQDN -u [email protected]

6. When prompted, type the [email protected] password.
Note: If you have customized the vCenter Single Sign-On domain, change the domain
name accordingly.

7. root@mb1esxvc [ ~ ]# service-control --restart rbd

root@mb1esxvc [ ~ ]#
root@mb1esxvc [ ~ ]#
root@mb1esxvc [ ~ ]# service-control --restart imagebuilder
root@mb1esxvc [ ~ ]#

To update the extension's certificate in the vCenter Server Appliance:

Log in to the vCenter Server Appliance as root through SSH.
Run this command to enable access the Bash shell:

shell.set --enabled true
Type shell and press Enter.
Run this command to retrieve the vpxd-extension solution user certificate and key:

mkdir /certificate

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.crt

/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.key
Run this command to update the extension's certificate with vCenter Server

For RBD service:
python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.rbd -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s vcsa_FQDN -u [email protected]

For ImageBuilder Service (applicable 6.5.x and onward):

python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.imagebuilder -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s vcsa_FQDN -u [email protected]

When prompted, type the [email protected] password.

Note: If you have customized the vCenter Single Sign-On domain, change the domain name accordingly.

6. root@mb1esxvc [ ~ ]# service-control --restart vmware-imagebuilder

Successfully restarted service imagebuilder
root@mb1esxvc [ ~ ]#
root@mb1esxvc [ ~ ]#
root@mb1esxvc [ ~ ]# service-control --restart vmware-rbd-watchdog
Successfully restarted service rbd
root@mb1esxvc [ ~ ]#

Additional Information

在 VMware vSphere 6.0 中替换 VMware vCenter Server 证书后，VMware vSphere Auto Deploy 解决方案用户登录失败
VMware vSphere 6.0 で VMware vCenter Server 証明書を置き換えると VMware vSphere Auto Deploy ソリューションユーザーがログインに失敗する
Error:"Failed to register Auto Deploy" while Upgrading Windows vCenter Server 6.5 to 6.7 (55076)