vCenter Server fails to start with "Remote login failed:N3Vim5Fault9HttpFault9ExceptionE(vim.fault.HttpFault)", After vCenter Server is restored from backup or snapshot
search cancel

vCenter Server fails to start with "Remote login failed:N3Vim5Fault9HttpFault9ExceptionE(vim.fault.HttpFault)", After vCenter Server is restored from backup or snapshot

book

Article ID: 338638

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

After restoring vCenter Server or vCenter Server Appliance from backup or a snapshot, you experience these symptoms:
 
  • In the /var/log/vmware/vpxd/vpxd.log file, you see entries similar to:
2017-02-10T16:54:07.586Z warning vpxd[7F38BD9C2700] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f38a41cc910, h:32, <TCP '0.0.0.0:0'>, <TCP '127.0.0.1:10080'>>, e: system:111(Connection refused)
2017-02-10T16:54:07.586Z warning vpxd[7F38CC164800] [Originator@6876 sub=VpxdAuthClient] [ConnectAndLogin] Failed to loginBySamlToken: N7Vmacore15SystemExceptionE(Connection refused)
--> [context]zKq7AVECAAAAAMJySwAPdnB4ZAAAkJ4rbGlidm1hY29yZS5zbwAAhE8bAOqXGADgXysAWVcjAApfJgD+YCYAhIMmAICKJgCfISQAb+0jAFbwIwB79isBVHQAbGlicHRocmVhZC5zby4wAAItjg5saWJjLnNvLjYA[/context]
2017-02-10T16:54:07.587Z info vpxd[7F38CC164800] [Originator@6876 sub=VpxdAuthClient] fallback to loginByCertificate
2017-02-10T16:54:07.587Z warning vpxd[7F38B7C78700] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f38a41c4310, h:32, <TCP '0.0.0.0:0'>, <TCP '127.0.0.1:10080'>>, e: system:111(Connection refused)
2017-02-10T16:54:07.587Z error vpxd[7F38CC164800] [Originator@6876 sub=httpUtil] [HttpUtil::ExecuteRequest] Error in sending request - Connection refused
2017-02-10T16:54:07.587Z error vpxd[7F38CC164800] [Originator@6876 sub=ServerAccess] Remote login failed: N3Vim5Fault9HttpFault9ExceptionE(vim.fault.HttpFault)
--> [context]zKq7AVECAAAAAMJySwAUdnB4ZAAAkJ4rbGlidm1hY29yZS5zbwAAhE8bAOqXGAEOylR2cHhkAAH65ogBK+6IAbvs9QEchqIBJ3GiArgWAWxpYmF1dGh6Y2xpZW50LnNvAALYSAEC8E4BAsraAQJI0AEBhuWgAQrpVAE48lQBapVTA+AFAmxpYmMuc28uNgABdY1T[/context]
2017-02-10T16:54:07.589Z error vpxd[7F38CC164800] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::CreateAuthzMgr] Failed to connect to IS: <N5Vmomi5Fault17HostCommunication9ExceptionE(vmodl.fault.HostCommunication)</font>
--> [context]zKq7AVECAAAAAMJySwATdnB4ZAAAkJ4rbGlidm1hY29yZS5zbwAAhE8bAOqXGAEOylR2cHhkAAF2E2IBsu/1ARyGogEncaICuBYBbGliYXV0aHpjbGllbnQuc28AAthIAQLwTgECytoBAkjQAQGG5aABCulUATjyVAFqlVMD4AUCbGliYy5zby42AAF1jVM=[/context]>
2017-02-10T16:54:07.590Z info vpxd[7F38CC164800] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::CreateAuthzMgr] Retry for this error: attempt count 1
2017-02-10T16:54:10.632Z warning vpxd[7F38B6346700] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f385c24cb90, h:32, <TCP '0.0.0.0:0'>, <TCP '127.0.0.1:10080'>>, e: system:111(Connection refused)
2017-02-10T16:54:10.633Z warning vpxd[7F38CC164800] [Originator@6876 sub=VpxdAuthClient] [ConnectAndLogin] Failed to loginBySamlToken: N7Vmacore15SystemExceptionE(Connection refused)
--> [context]zKq7AVECAAAAAMJySwAPdnB4ZAAAkJ4rbGlidm1hY29yZS5zbwAAhE8bAOqXGADgXysAWVcjAApfJgD+YCYAhIMmAICKJgCfISQAb+0jAFbwIwB79isBVHQAbGlicHRocmVhZC5zby4wAAItjg5saWJjLnNvLjYA[/context]
2017-02-10T16:54:10.633Z info vpxd[7F38CC164800] [Originator@6876 sub=VpxdAuthClient] fallback to loginByCertificate
2017-02-10T16:54:10.633Z warning vpxd[7F38B75EB700] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f385c24cb90, h:32, <TCP '0.0.0.0:0'>, <TCP '127.0.0.1:10080'>>, e: system:111(Connection refused)
2017-02-10T16:54:10.634Z error vpxd[7F38CC164800] [Originator@6876 sub=httpUtil] [HttpUtil::ExecuteRequest] Error in sending request - Connection refused
2017-02-10T16:54:10.634Z error vpxd[7F38CC164800] [Originator@6876 sub=ServerAccess] Remote login failed: N3Vim5Fault9HttpFault9ExceptionE(vim.fault.HttpFault)
--> [context]zKq7AVECAAAAAMJySwAUdnB4ZAAAkJ4rbGlidm1hY29yZS5zbwAAhE8bAOqXGAEOylR2cHhkAAH65ogBK+6IAbvs9QEchqIBJ3GiArgWAWxpYmF1dGh6Y2xpZW50LnNvAALYSAEC8E4BAsraAQJI0AEBhuWgAQrpVAE48lQBapVTA+AFAmxpYmMuc28uNgABdY1T[/context]
2017-02-10T16:54:10.635Z error vpxd[7F38CC164800] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::CreateAuthzMgr] Failed to connect to IS: <N5Vmomi5Fault17HostCommunication9ExceptionE(vmodl.fault.HostCommunication)</font>
--> [context]zKq7AVECAAAAAMJySwATdnB4ZAAAkJ4rbGlidm1hY29yZS5zbwAAhE8bAOqXGAEOylR2cHhkAAF2E2IBsu/1ARyGogEncaICuBYBbGliYXV0aHpjbGllbnQuc28AAthIAQLwTgECytoBAkjQAQGG5aABCulUATjyVAFqlVMD4AUCbGliYy5zby42AAF1jVM=[/context]>
 
...
 
2017-02-10T16:57:10.714Z error vpxd[7F38CC164800] [Originator@6876 sub=Authorize] Failed to initialize authorizeManager
2017-02-10T16:57:10.714Z warning vpxd[7F38CC164800] [Originator@6876 sub=VpxProfiler] Start [VpxdAuthorize::Start()] took 183263 ms
2017-02-10T16:57:10.714Z warning vpxd[7F38CC164800] [Originator@6876 sub=VpxProfiler] ServerApp::Start [TotalTime] took 183356 ms
2017-02-10T16:57:10.714Z error vpxd[7F38CC164800] [Originator@6876 sub=Default] Failed to start VMware VirtualCenter. Shutting down
2017-02-10T16:57:10.714Z info vpxd[7F38CC164800] [Originator@6876 sub=SupportMgr] Wrote uptime information
2017-02-10T16:59:05.419Z info vpxd[7F38BE15F700] [Originator@6876 sub=HostGateway] CmConnectionFSM::RunFSM(ST_CM_CALL_FAILED)
2017-02-10T16:59:05.420Z warning vpxd[7F38BE15F700] [Originator@6876 sub=HostGateway] State(ST_CM_LOGIN) failed with: Invalid argument : cmStub
2017-02-10T16:59:05.420Z warning vpxd[7F38BE15F700] [Originator@6876 sub=HostGateway] State(ST_ACQUIRE_TOKEN) failed with: Invalid argument : stsClient
2017-02-10T16:59:05.420Z info vpxd[7F38BE15F700] [Originator@6876 sub=HostGateway] stsUrlFromConfig: https://comp01psc01.sfo01.rainpole.local/sts/STSService/vsphere.local ssoAdminUrlFromConfig: https://comp01psc01.sfo01.rainpole.local/sso-adminserver/sdk/vsphere.local
2017-02-10T16:59:05.448Z info vpxd[7F38BE15F700] [Originator@6876 sub=vpxCrypt] Failed to read X509 cert; err: 151441516
2017-02-10T16:59:05.478Z info vpxd[7F38BE15F700] [Originator@6876 sub=vpxCrypt] Failed to read X509 cert; err: 151441516
  • In the /var/log/vmware/vmafdd/vmafdd-syslog.log files, you see entries similar to:
17-02-10T16:56:21.213900+00:00 err vmafdd t@139929541560064: VmAfSrvCfgGetMachineID failed. Error(9234)
17-02-10T16:56:21.214035+00:00 err vmafdd t@139929541560064: [Error - 9234, ../../../server/vmafd/config.c:440]
17-02-10T16:56:21.214166+00:00 err vmafdd t@139929541560064: VmAfSrvGetCMLocation failed. Error(9234)
17-02-10T16:56:21.214295+00:00 err vmafdd t@139929541560064: ERROR! [VmAfdIpcGetCMLocation] is returning [9234]
17-02-10T16:56:22.020892+00:00 err vmafdd t@139929716352768: [Error - 9234, ../../../server/vmafd/ldap.c:170]
17-02-10T16:56:22.021971+00:00 err vmafdd t@139929716352768: [Error - 9234, ../../../server/vmafd/rootfetch.c:256]
17-02-10T16:56:22.022297+00:00 notice vmafdd t@139929716352768: Failed to update trusted roots. Error [9234]
17-02-10T16:56:22.026505+00:00 err vmafdd t@139929564739328: [Error - 9234, ../../../server/vmafd/ldap.c:170]
17-02-10T16:56:33.704271+00:00 err vmafdd t@139929541560064: [Error - 9234, ../../../server/vmafd/vmdir.c:1468]
17-02-10T16:56:33.704528+00:00 notice vmafdd t@139929541560064: VmAfSrvDirOpenConnection failed. Error(9234)
17-02-10T16:56:33.704686+00:00 err vmafdd t@139929541560064: [Error - 9234, ../../../server/vmafd/vmdir.c:1023]
17-02-10T16:56:33.704825+00:00 notice vmafdd t@139929541560064: VmAfSrvDirGetMachineId failed. Error(9234)
17-02-10T16:56:33.704983+00:00 err vmafdd t@139929541560064: [Error - 9234, ../../../server/vmafd/config.c:1307]
  • When attempting to execute one of the VMAFD-CLI get-* or set-* commands, you see output similar to:
     
root@mgmt01vc01 [ /var/log/vmware ]# /usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost
Error 9234: Authentication to VMware Directory Service failed.
Verify the username and password.
 
root@mgmt01vc01 [ /var/log/vmware/vpxd ]# /usr/lib/vmware-vmafd/bin/vmafd-cli set-dc-name --server-name localhost --dc-name mgmt01psc01.sfo01.rainpole.local
Error 9234: Authentication to VMware Directory Service failed.
Verify the username and password.
 
 
Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
 


Environment

VMware vCenter Server 5.5.x
VMware vCenter Server 6.x

Cause

Periodically, the password used by the Machine Accounts' is automatically refreshed.
 
This password is stored on the Machine Account on vCenter Server and within the VMware Directory Service residing on the Platform Services Controllers. If the backup used to perform a restore which pre-dates the most recent password refresh, upon restore the Machine Account will be unable to log into the VMware Directory Service.

Additionally, if the vCenter Server is rolled back to a snapshot state which pre-dates the most recent password refresh, upon restore the Machine Account will be unable to log into the VMware Directory Service.

Resolution


To update the password used by the vCenter Server's Machine Account after a restore, use the vcenter-restore CLI to issue a reset request. For more information, see the Restore the Failed vCenter Server Virtual Machine With the Direct-to-Host Emergency Restore Operation section in the vSphere Installation and Setup Guide.
 
For vCenter Server Appliance
 
  1. Login to vCenter Server via SSH as the root user
     
  2. Run this command to switch to BASH

    shell
     
  3. Run this command to Stop all of the vCenter Server services:

    service-control --stop --all
     
  4. Run this command (Below does not work on embedded node)

    vcenter-restore -u administrator -p <[email protected] password>

    For example:

    vcenter-restore -u administrator -p VMware1!

    Note: This CLI can be executed without either option, and the user will be prompted to provide the administrator and [email protected] password.
  5. Run this command to confirm that all services are now started:

    service-control --status --all
 
For vCenter Server on Windows
 
  1. Open an elevated command prompt.
     
  2. Run this command to change the directories:

    cd C:\Program Files\VMware\vCenter Server\bin
     
  3. Run this command to Stop all of the vCenter Server services:

    service-control --stop --all
     
  4. Run this command (Below does not work on embedded node)

    vcenter-restore -u administrator -p <[email protected] password>

    For example:

    vcenter-restore -u administrator -p VMware1!

    Note: This CLI can be executed without either option, and the user will be prompted to provide the administrator and [email protected] password.
     
  5. Run this command to confirm that all services are now started:

    service-control --status --all


Additional Information

To manually invoke an update of vCenter Server's Machine Account password, use the dir-cli computer password-reset CLI to issue a reset request.
 
For vCenter Server Appliance
 
  1. Login to the vCenter Server via SSH as the root user
     
  2. Run this command to switch to BASH

    shell
     
  3. Run this command:

    /usr/lib/vmware-vmafd/bin/dir-cli computer password-reset --login administrator --live-dc-hostname <Platform Services Controller FQDN> --password <[email protected] password>

    For example:

    /usr/lib/vmware-vmafd/bin/dir-cli computer password-reset --login administrator --live-dc-hostname mgmt01psc01.sfo01.rainpole.local --password 'VMware1!'

    A successful password reset will output:

    Password for machine account reset.

    Note: If you are using a PSC HA configuration using a Load Balancer, execute this command against one of the backing PSC members and wait 30 seconds for the VMware Directory Service replication cycle to push this update to all nodes.
     
  4. Run these commands to Start the vCenter Server services:

    service-control --stop --all
    service-control --start --all
从备份或快照还原 vCenter Server 之后，vCenter Server 无法启动并显示“Remote login failed:N3Vim5Fault9HttpFault9ExceptionE(vim.fault.HttpFault)”
VCenter Server が起動に失敗し、次のエラーが表示される：「リモート ログインに失敗しました：N3Vim5Fault9HttpFault9ExceptionE(vim.fault.HttpFault)」、vCenter Server がバックアップまたはスナップショットからリストアされた後("Remote login failed:N3Vim5Fault9HttpFault9ExceptionE(vim.fault.HttpFault)", After vCenter Server is re

Powered by Wolken Software