SPS service start failing with VpxdException: Error while doing login to VPXD service
search cancel

SPS service start failing with VpxdException: Error while doing login to VPXD service

book

Article ID: 320872

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:
VMware vSphere Profile-Driven Storage Service and VMware Update Manager service failing to start
Permissions on the vCenter server object is also inaccessible on the web client 
  • sps.log:
<YYYY-MM-DDTHH:MM:SS> [main] INFO  opId=sps-Main-360774-610 com.vmware.vim.storage.common.identity.ServiceSolutionUserByHoKToken - Service solution user token acquired successfully. Expiration time: Wed May 15 20:40:38 UTC 2019
<YYYY-MM-DDTHH:MM:SS> [main] INFO  opId=sps-Main-360774-610 com.vmware.vim.storage.common.identity.ServiceSolutionUserByHoKToken - SPS solution user initialized successfully
<YYYY-MM-DDTHH:MM:SS> [pool-1-thread-1] WARN  opId=sps-Main-360774-610 com.vmware.vim.storage.common.serviceclient.vmomi.RequestRetryHandler - Error while trying to do relogin
java.lang.IllegalStateException: Client initialization is not complete!
        at com.google.common.base.Preconditions.checkState(Preconditions.java:174)
        at com.vmware.vim.storage.common.serviceclient.ConnectionInitializationTask.get(ConnectionInitializationTask.java:80)
        at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientManagerImpl.getClient(VpxdClientManagerImpl.java:169)
        at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientManagerImpl.getCurrentSession(VpxdClientManagerImpl.java:180)
        at com.vmware.vim.storage.common.serviceclient.vmomi.RequestRetryHandler.retry(RequestRetryHandler.java:76)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$RetryingFuture.setException(MethodInvocationHandlerImpl.java:525)
        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:230)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:156)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:53)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
<YYYY-MM-DDTHH:MM:SS> [main] ERROR opId=sps-Main-360774-610 com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientImpl - VPXD client login failed.
<YYYY-MM-DDTHH:MM:SS> [main] ERROR opId=sps-Main-360774-610 com.vmware.vim.storage.common.task.retry.CallableRetryDecorator - Caught exception -
com.vmware.vim.storage.common.serviceclient.vpxd.VpxdException: Error while doing login to VPXD service
        at com.vmware.vim.storage.common.serviceclient.vpxd.VpxdException.fromEx(VpxdException.java:53)
        at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientImpl.loginByToken(VpxdClientImpl.java:159)
        at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientLifeCycle.login(VpxdClientLifeCycle.java:129)
        at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientLifeCycle.login(VpxdClientLifeCycle.java:34)
        at com.vmware.vim.storage.common.serviceclient.ConnectionInitializationTask$CallableTemplate.call(ConnectionInitializationTask.java:118)
        at com.vmware.vim.storage.common.task.retry.CallableRetryDecorator.call(CallableRetryDecorator.java:64)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at com.vmware.vim.storage.common.serviceclient.vpxd.impl.VpxdClientManagerImpl.initialize(VpxdClientManagerImpl.java:106)
        at com.vmware.sps.StorageMain.commonInitialization(StorageMain.java:188)
        at com.vmware.sps.StorageMain.main(StorageMain.java:67)
Caused by: (vim.fault.NoPermission) {
   faultCause = null,
   faultMessage = null,
   object = ManagedObjectReference: type = Folder, value = group-d1, serverGuid = 4082b4f9-c668-4546-aba1-9f1cfed71f07,
   privilegeId = System.View
}
  • vmware-vum-server-1.log:
<YYYY-MM-DDTHH:MM:SS> info vmware-vum-server[64346] [Originator@6876 sub=Default] [shutdownMgr,92] Ufa shutdown manager stopping... first make sure this thread is enlisted.
<YYYY-MM-DDTHH:MM:SS> info vmware-vum-server[64346] [Originator@6876 sub=ThreadPool] Thread enlisted
<YYYY-MM-DDTHH:MM:SS> info vmware-vum-server[64346] [Originator@6876 sub=Default] [shutdownMgr,97] Ufa shutdown manager stopping... Total number of callbacks: 1
<YYYY-MM-DDTHH:MM:SS> info vmware-vum-server[64346] [Originator@6876 sub=VcIntegrity] Error on logout (ignored): Connection reset by peer: The connection is terminated by the remote end with a reset packet. Usually, this is a sign of a network problem,  timeout, or service overload.
<YYYY-MM-DDTHH:MM:SS> warning vmware-vum-server[21223] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007fa4dc03d230, h:15, <TCP '127.0.0.1 : 50448'>, <TCP '127.0.0.1 : 80'>>, e: 111(Connection refused)
<YYYY-MM-DDTHH:MM:SS> warning vmware-vum-server[21162] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007fa5105085c0, h:15, <TCP '127.0.0.1 : 50450'>, <TCP '127.0.0.1 : 80'>>, e: 111(Connection refused)
<YYYY-MM-DDTHH:MM:SS> info vmware-vum-server[21220] [Originator@6876 sub=VcIntegrity] Error on logout (ignored): Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.
<YYYY-MM-DDTHH:MM:SS> info vmware-vum-server[64346] [Originator@6876 sub=Default] [shutdownMgr,109] Ufa shutdown manager stopped
<YYYY-MM-DDTHH:MM:SS> info vmware-vum-server[21166] [Originator@6876 sub=ThreadPool] Thread delisted
  • vpxd.log:
<YYYY-MM-DDTHH:MM:SS> verbose vpxd[35323] [Originator@6876 sub=[SSO] opID=sps-Main-226767-398-34] [UserDirectorySso] NormalizeUserName(<SSO Domain>\vpxd-extension-25b86efe-fc6e-49a1-93d9-1b4b8694aa44, false) res: <SSO Domain>\vpxd-extension-25b86efe-fc6e-49a1-93d9-1b4b8694aa44
<YYYY-MM-DDTHH:MM:SS> info vpxd[35323] [Originator@6876 sub=AuthorizeManager opID=sps-Main-226767-398-34] [Auth]: User <SSO Domain>\vpxd-extension-25b86efe-fc6e-49a1-93d9-1b4b8694aa44
<YYYY-MM-DDTHH:MM:SS> verbose vpxd[35323] [Originator@6876 sub=Vmomi opID=sps-Main-226767-398-34] Invoke error: vim.SessionManager.loginByToken session: 528a0ce5-c69e-d8a6-bb4f-43448412dfe6 Throw: vim.fault.NoPermission
<YYYY-MM-DDTHH:MM:SS> info vpxd[35323] [Originator@6876 sub=vpxLro opID=sps-Main-226767-398-34] [VpxLRO] -- FINISH lro-40
<YYYY-MM-DDTHH:MM:SS> info vpxd[35323] [Originator@6876 sub=Default opID=sps-Main-226767-398-34] [VpxLRO] -- ERROR lro-40 -- SessionManager -- vim.SessionManager.loginByToken: vim.fault.NoPermission:
--> Result:
--> (vim.fault.NoPermission) {
--> faultCause = (vmodl.MethodFault) null


Environment

VMware vCenter Server Appliance 6.7.x

Cause

The issue is seen if there are duplicate entries for the default administrator

Resolution

This issue is resolved in vCenter Server 6.7 Update 3l build 17138064 and 7.0 Update 3c build 19234570 or later

Workaround:
** Ensure to take a functional backup of the nodes **

* Overview of Backup and Restore options in vCenter Server 6.x (2149237)
https://kb.vmware.com/s/article/2149237
    • log in to the VCSA using ssh
    • stop the service using the following command 
    service-control --stop vmware-vpxd
    service-control --stop vmware-content-library
    • log in to the database using the following command:
    command: /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
    • Execute the following steps to validate and remove the duplicate entries
    VCDB=# select * from vpx_access;

    Sample:
    id | principal | role_id | entity_id | flag | surr_key
    -----+---------------------------+---------+-----------+------+----------
    1 | SSO.VCENTER\Administrator | -1 | 1 | 1 | 1
    8 | SSO.VCENTER\Administrator | -1 | 7 | 1 | 2
    901 | SSO.VCENTER\Administrator | -1 | 1 | 1 | 11
    • Additionally validate the entries where principal value is <SSO Domain>\Administrator
    VCDB=# select * from vpx_access where principal like '%Administrator%';

     id  |          principal          | role_id | entity_id | flag | surr_key
    -----+-----------------------------+---------+-----------+------+----------
       1 | SSO.VCENTER\Administrator |      -1 |         1 |    1 |        1
     209 | SSO.VCENTER\Administrator |      -1 |         2 |    1 |        3
     601 | SSO.VCENTER\Administrator |      -1 |         1 |    1 |        7
    • Delete the entries except id = 1
    command: delete from vpx_access where id=<ID from above output>;
    Note: command needs to be executed for all the additional entries
    • start the service using the following command:
    service-control --start vmware-vpxd
    service-control --start vmware-content-library
    service-control --start vmware-sps
    service-control --start vmware-updatemgr
    • Log in to the web client and validate the functionality


    Additional Information

    VMware Skyline Health Diagnostics for vSphere - FAQ

    Impact/Risks:
    Unable to add permissions on the vCenter server objects

    Powered by Wolken Software