Symptoms:
On vCenter Server 8.0U2, when attempting to perform VM operations, the operation hangs and completes after an unknown amount of time or fails to complete.
- In /var/log/vmware/vsm/vsm.log, you may see:
2023-11-02T23:11:57.450-04:00 INFO [Thread-4] ServiceUtil.java 137 - Acquiring SAML token for user [email protected]
2023-11-02T23:11:57.787-04:00 ERROR [Thread-4] SoapBindingImpl.java 185 - SOAP fault
com.sun.xml.internal.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server: Password of the user logging on is expired. :: Password of the user logging on is expired. :: User account expired: {Name: vmware-vsm-758251d3-01bc-41a9-9269-c1b8287facba, Domain: vsphere.local} Please see the server log to find more detail regarding exact cause of the failure.
..
com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.java:983) [libwstclient.jar:?]
at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.executeRoundtrip(SecurityTokenServiceImpl.java:902) [libwstclient.jar:?]
at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl.acquireToken(SecurityTokenServiceImpl.java:155) [libwstclient.jar:?]
at com.vmware.vsm.utils.ServiceUtil.getSamlTokenForSvcAccount(ServiceUtil.java:138) [vsm.jar:?]
at com.vmware.vsm.vc.VCenterListener.loadSamlTokenAndPrivateKey(VCenterListener.java:473) [vsm.jar:?]
at com.vmware.vsm.vc.VCenterListener.initializeConnection(VCenterListener.java:268) [vsm.jar:?]
at com.vmware.vsm.vc.VCenterListener.run(VCenterListener.java:301) [vsm.jar:?]
2023-11-02T23:11:57.843-04:00 INFO [Thread-4] ServiceUtil.java 142 - Password expired. Resetting service account password
- OVF deployment may hang at power on
In
/var/log/vmware/vpxd/vpxd.log you will see events similar to below
2024-02-21T18:34:51.721-06:00 info vpxd[27763] [Originator@6876 sub=VmProv opID=TxId: 0625c4dc-8450-4f08-aee2-12d5add051b0-47-01] P
owering on VM '[vsanDatastore] 6b96d665-ca86-9992-f3f8-043201173960/Interconnect-C3-IX-R1.vmx' on host <host fqdn/ip>
2024-02-21T18:34:51.725-06:00 warning vpxd[27763] [Originator@6876 sub=vmomi.soapStub[258] opID=TxId: 0625c4dc-8450-4f08-aee2-12d5a
dd051b0-47-01] SOAP request returned HTTP failure; <<io_obj p:0x00007fdab0421b10, h:182, <TCP '127.0.0.1 : 37532'>, <TCP '127.0.0.1
: 15007'>>, /vsm/ovfConsumer/>, method: notifyPowerOn; code: 500(Internal Server Error)
2024-02-21T18:34:51.725-06:00 info vpxd[27763] [Originator@6876 sub=OvfConsumers opID=TxId: 0625c4dc-8450-4f08-aee2-12d5add051b0-47
-01] Failed to invoke OVF stub adapter, will re-try after login; N5Vmomi5Fault13SecurityError9ExceptionE(Fault cause: vmodl.fault.S
ecurityError
--> )
--> [context]zKq7AVECAQAAAIUcWQEsdnB4ZAAAxbVTbGlidm1hY29yZS5zbwAAUglDAIwxRACaSEsBoy8XbGlidm1vbWkuc28AAX6eJQFZICABOk4gAZ3HH4JJ8TUBdn
B4ZACCscNFAoKExUUCAX02GoNhyScBbGlidmltLXR5cGVzLnNvAIJGprsBgrXTuwGCNIHUAYLM9SACgn+wGwKC670kAoJI3SQCgkr9JAKCfQhiAoLPGGICgvkvYQKCUothA
oLJH9QBgk0M1QGChw3VAYJTDtUBgrIO1QGD5v4xAYK5d2MCAdXDG4IgokMCgn0IYgKCzxhiAoL5L2ECgpQCYgIA5ss3APkkOACTwFEEro4AbGlicHRocmVhZC5zby4wAAUv
3g9saWJjLnNvLjYA[/context]
- At the same time in /var/log/vmware/vsm/vsm.log
2024-02-21T18:34:51.724-06:00 ERROR [pool-5-thread-1] VsmActivationValidator.java 267 - Failed to validate user: only vpxd-svc-acct requests allowed and not