"Failed to login to vCenter as extension, Cannot complete login due to an incorrect user name or password", ESX Agent Manager (com.vmware.vim.eam) solution user fails to log in after replacing the vCenter Server certificates in vCenter Server 6.x/7.x/8.x
search cancel

"Failed to login to vCenter as extension, Cannot complete login due to an incorrect user name or password", ESX Agent Manager (com.vmware.vim.eam) solution user fails to log in after replacing the vCenter Server certificates in vCenter Server 6.x/7.x/8.x

book

Article ID: 318255

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:
After replacing certificates on vCenter Server, you experience these symptoms:
  • ESX Agent Manager solution user fails to log in.
  • In the /var/log/vmware/eam/eam.log (on VCSA) file or the C:\ProgramData\VMware\vCenterServer\logs\eam\eam.log (on Windows) file for the ESX Agent Manager (EAM), you see entries similar to:
YYYY-MM-DDTHH:MM:SS.MSZ | INFO | eam-0 | VcConnection.java | 167 | Connecting to vCenter as com.vmware.vim.eam extension
YYYY-MM-DDTHH:MM:SS.MSZ | INFO | eam-0 | VcConnection.java | 603 | Connecting to https://vCenter_Server_FQDN:8089/sdk/vimService via vCenter proxy http://localhost:80
YYYY-MM-DDTHH:MM:SS.MSZ
| DEBUG | http-bio-0.0.0.0-15005-exec-1 | AllowAllSamlTokenPolicy.java | 24 | HealtStatus request's token subject name: machine-7502fb4c-3521-48c7-93ed-3d1865e0fff1, subject domain: vsphere.local
YYYY-MM-DDTHH:MM:SS.MSZ
| ERROR | eam-0 | VcConnection.java | 179 | Failed to login to vCenter as extension. vCenter has probably not loaded the EAM extension.xml yet.: Cannot complete login due to an incorrect user name or password.
YYYY-MM-DDTHH:MM:SS.MSZ | WARN | eam-0 | VcListener.java | 114 | Trying to recover from error
(vim.fault.InvalidLogin) {
faultCause = null,
faultMessage = null
}
at sun.reflect.GeneratedConstructorAccessor82.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at java.lang.Class.newInstance(Unknown Source)
at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:173)
at com.vmware.vim.vmomi.core.types.impl.DefaultDataObjectFactory.newDataObject(DefaultDataObjectFactory.java:26)
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.ComplexStackContext.<init>(ComplexStackContext.java:31)
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.parse(UnmarshallerImpl.java:141)
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.unmarshall(UnmarshallerImpl.java:102)
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:89)
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:84)
at com.vmware.vim.vmomi.client.common.impl.SoapFaultStackContext.setValue(SoapFaultStackContext.java:41)
at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.unmarshal(ResponseUnmarshaller.java:112)
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.unmarshalResponse(ResponseImpl.java:273)
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:230)
at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:144)
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:51)
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:186)
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:77)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:581)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:562)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:348)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:308)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:182)
at com.sun.proxy.$Proxy48.loginExtensionByCertificate(Unknown Source)
at com.vmware.eam.vc.VcConnection.connectEam(VcConnection.java:171)
at com.vmware.eam.vc.VcListener.login(VcListener.java:149)
at com.vmware.eam.vc.VcListener.main(VcListener.java:129)
at com.vmware.eam.vc.VcListener.call(VcListener.java:111)
at com.vmware.eam.vc.VcListener.call(VcListener.java:60)
at com.vmware.eam.async.impl.AuditedJob.call(AuditedJob.java:35)
at com.vmware.eam.async.impl.FutureRunnable.run(FutureRunnable.java:52)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
YYYY-MM-DDTHH:MM:SS.MSZ | INFO | eam-0 | VcListener.java | 121 | Retrying in 10

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
  • Unable to deploy VIBs to your ESXi hosts from NSX for vSphere or vCloud Networking and Security.
  • vCenter Server experiences high CPU usage.


Environment

VMware vCenter Server 6.7.x
VMware vCenter Server 6.0.x
VMware vCenter Server 7.0.x
VMware vCenter Server 8.0.x
VMware vCenter Server Appliance 6.0.x
VMware vCenter Server Appliance 6.7.x
VMware vCenter Server 6.5.x
VMware vCenter Server Appliance 6.5.x

Cause

This issue can happen if there is a mismatch between vpxd-extension certificate stored in VECS and the certificate information stored in vCenter Server Database for EAM extension.

Resolution

To resolve the issue, update the extension's certificate with vCenter Server by following below steps.

Notes:
  • Certificate Manager automatically updates the certificate for extensions like com.vmware.vim.eam (EAM), com.vmware.rbd (Autodeploy) from vCenter Server 6.0 U1b onwards. However, below steps can be tried on any vCenter Server 6.x or 7.x versions if you are seeing the error messages mentioned in Symptoms section
  • Please refer to the Related Information in this KB if you are receiving the error "certificate verify failed: Hostname mismatch, certificate is not valid for 'sdkTunnel'during Extension certificate registration

Process to update the extension's certificate:

On vCenter Server for Windows:
  1. Connect to vCenter Server through a Console or Remote desktop session.
  2. Open an elevated command prompt.
  3. Create a temporary directory named c:\certificate.
  4. Run this command to retrieve the vpxd-extension solution user certificate and key:

    "%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output c:\certificate\vpxd-extension.crt

    "%VMWARE_CIS_HOME%"\vmafdd\vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output c:\certificate\vpxd-extension.key

     
  5. Navigate to C:\Program Files\VMware\vCenter Server\vpxd\scripts:

    cd C:\Program Files\VMware\vCenter Server\vpxd\scripts

    Note: The path listed is for a default install of vCenter Server. If you have customized the install location of vCenter Server, change the directory accordingly.
     
  6. Run this command to update the extension's certificate with vCenter Server:

    "%VMWARE_PYTHON_BIN%" updateExtensionCertInVC.py -e com.vmware.vim.eam -c C:\Certificate\vpxd-extension.crt -k C:\Certificate\vpxd-extension.key -s <PNID/FQDN of vCenter Server> -u Administrator@vsphere.local

    Note: The default user and domain is [email protected]. If this was changed during configuration, change the domain to match your environment.
 
  1. When prompted, enter the Administrator@vsphere.local password.
  2. Navigate to the C:\Program Files\VMware\vCenter Server\bin folder:

    cd C:\Program Files\VMware\vCenter Server\bin
     
  3. Restart the VMware ESX Manager service with these commands:

    For more information on managing service in the vCenter Server Appliance, see Stopping, starting, or restarting VMware vCenter Server Appliance 6.0 services (2109887).
    • service-control --stop EsxAgentManager
    • service-control --start EsxAgentManager
 
On the vCenter Server Appliance:
  1. Log in to the vCenter Server Appliance using SSH.
  2. Run this command to enable access the Bash shell (Note: this is applicable only on vCenter Server 6.0):

    shell.set --enabled true
     
  3. Type shell and press Enter.
  4. Run this command to retrieve the vpxd-extension solution user certificate and key:

    mkdir /certificate

    /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.crt

    /usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.key

     
  5. Run this command to update the extension's certificate with vCenter Server.

    python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s <PNID/FQDN of vCenter Server> -u Administrator@vsphere.local

    Note: The default user and domain is [email protected]. If this was changed during configuration, change the domain to match your environment. When prompted, type in the Ad[email protected] password.
     
  6. Restart the VMware ESX Manager service with these commands:
     
    • service-control --stop vmware-eam
    • service-control --start vmware-eam
       
    For more information on managing service in the vCenter Server, see Stopping, starting, or restarting VMware vCenter Server 6.0 services (2109881).


Additional Information

  • In certain situations, you might receive the error "certificate verify failed: Hostname mismatch, certificate is not valid for 'sdkTunnel'". This error can be safely ignored if you are getting the error after the message "Successfully updated certificate for "com.vmware.vim.eam" extension" as this message confirms that Extension certificate updated successfully with vCenter Server.
root@hostname [ ~ ]# python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s vcsa1.test.com -u [email protected]
Password to connect to VC server for user="[email protected]":
2021-03-11T22:31:22.517Z  Updating certificate for "com.vmware.vim.eam" extension
2021-03-11T22:31:22.649Z  Successfully updated certificate for "com.vmware.vim.eam" extension
Traceback (most recent call last):
  File "/usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py", line 175, in <module>
    update_extension_cert_in_VC()
  File "/usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py", line 163, in update_extension_cert_in_VC
    sessionMgr = si.content.sessionManager
  File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 577, in __call__
    return self.f(*args, **kwargs)
  File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 382, in _InvokeAccessor
    return self._stub.InvokeAccessor(self, info)
  File "/usr/lib/vmware/site-packages/pyVmomi/StubAdapterAccessorImpl.py", line 42, in InvokeAccessor
    return self.InvokeMethod(mo, info, (prop, ))
  File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1525, in InvokeMethod
    conn.request('POST', self.path, req, headers)
  File "/usr/lib/python3.7/http/client.py", line 1277, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1323, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1272, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1032, in _send_output
    self.send(msg)
  File "/usr/lib/python3.7/http/client.py", line 972, in send
    self.connect()
  File "/usr/lib/python3.7/http/client.py", line 1447, in connect
    server_hostname=server_hostname)
  File "/usr/lib/python3.7/ssl.py", line 423, in wrap_socket
    session=session
  File "/usr/lib/python3.7/ssl.py", line 870, in _create
    self.do_handshake()
  File "/usr/lib/python3.7/ssl.py", line 1139, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'sdkTunnel'. (_ssl.c:1076)


VMware Skyline Health Diagnostics for vSphere - FAQ
vecs-cli failed. Error 87: Operation failed with error ERROR_INVALID_PARAMETER (87)
How to stop, start, or restart vCenter Server 6.x services
Resolution: Internal error occurs during vSphere ESX Agent Manager pre-upgrade checks
"WARNING: VMware ESX Agent Manager may have failed to start", EAM Service fails to start after vCenter Server reboot

Read this article in different language here:
VMware vSphere 6.0 で vCenter Server 証明書を置き換えると ESX Agent Manager ソリューション ユーザーがログインに失敗する
Depois de substituir os certificados do vCenter Server no VMware vSphere 6.0, o usuário de solução ESX Agent Manager não consegue fazer login
Después de reemplazar los certificados de vCenter Server en VMware vSphere 6.0, el usuario de la solución de ESX Agent Manager no puede iniciar sesión
在 VMware vSphere 6.0 中替换 vCenter Server 证书后,ESX Agent Manager 解决方案用户无法登录