"[500] An error occurred while fetching identity providers" after upgrading VC to 7.0 U2

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

Post vCenter Server 7.0 U2 upgrade unable to login to VC getting an error "[500] An error occurred while fetching identity providers. Try again"

500-An error occurred while fetching identity providers

You might see similar log snippet in vsphere_client_virgo and trustmanagement-svcs.log log files

vsphere_client_virgo.log
[2021-03-10T09:24:46.626Z] [WARN ] http-nio-5090-exec-9 70000004 100004 ###### c.v.vsphere.client.security.oauth2.logout.LogoutRequestHandler Unable to determine the identity provider type. Logout request will be skipped.
[2021-03-10T09:24:46.645Z] [INFO ] http-nio-5090-exec-4 70000005 100004 ###### com.vmware.vsphere.client.security.oauth2.LoginRequestHandler Received Multi login request
[2021-03-10T09:24:46.677Z] [INFO ] http-nio-5090-exec-4 70000005 100004 ###### com.vmware.vise.vim.vapi.StaticEndpointVapiConnectionManager Connected to vAPI endpoint https://vcenter.test.lab:443/site/api
[2021-03-10T09:24:46.963Z] [ERROR] VapiAsyncCall-101 com.vmware.vise.vim.vapi.DefaultVapiConnectionControl Maximum number of attempts reached while trying to call com.vmware.vcenter.identity.providers.list
[2021-03-10T09:24:46.965Z] [ERROR] http-nio-5090-exec-4 70000005 100004 ###### com.vmware.vsphere.client.security.oauth2.LoginRequestHandler An error occurred while fetching providers com.vmware.vapi.std.errors.Unauthenticated: Unauthenticated (com.vmware.vapi.std.errors.unauthenticated) => {
messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
id = vapi.method.authentication.required,
defaultMessage = This method requires authentication.,
args = [],
params = <null>,
localized = <null>
}],
data = <null>,
errorType = UNAUTHENTICATED,
challenge = <null>
}
at java.lang.Thread.getStackTrace(Thread.java:1559)


trustmanagement-svcs.log
2021-03-10T09:27:03.474Z [tomcat-exec-14 INFO com.vmware.identity.token.impl.SamlTokenImpl opId=] SAML token for SubjectNameId [value=machine-<machineID>@vsphere.local, format=http://schemas.xmlsoap.org/claims/UPN] successfully parsed from XML
2021-03-10T09:27:03.474Z [tomcat-exec-14 INFO com.vmware.identity.token.impl.X509TrustChainKeySelector opId=] Failed to find trusted path to signing certificate <STS Certificate Subject, example - C=US,CN=ssoserverSign\,dc\=vsphere\,dc\=local>
java.security.cert.CertPathBuilderException: Unable to find certificate chain.
   at org.bouncycastle.jcajce.provider.PKIXCertPathBuilderSpi.engineBuild(Unknown Source)
   at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
   at com.vmware.identity.token.impl.X509TrustChainKeySelector.verifyTrustedPathExists(X509TrustChainKeySelector.java:197)
   at com.vmware.identity.token.impl.X509TrustChainKeySelector.select(X509TrustChainKeySelector.java:116)
   at org.jcp.xml.dsig.internal.dom.DOMXMLSignature$DOMSignatureValue.validate(DOMXMLSignature.java:557)
   at org.jcp.xml.dsig.internal.dom.DOMXMLSignature.validate(DOMXMLSignature.java:268)
   at com.vmware.identity.token.impl.SamlTokenImpl.validateSignature(SamlTokenImpl.java:720)
   at com.vmware.identity.token.impl.SamlTokenImpl.validate(SamlTokenImpl.java:562)
   at com.vmware.vim.sso.client.DefaultTokenFactory.parseToken(DefaultTokenFactory.java:70)
   at com.vmware.vapi.internal.cis.authn.json.JsonSignatureStruct.parseJsonSignatureStruct(JsonSignatureStruct.java:112)
   at com.vmware.vapi.internal.cis.authn.json.JsonSignerImpl.verifySignature(JsonSignerImpl.java:120)
   at com.vmware.vapi.cis.authn.json.JsonSignatureVerificationProcessor.validateSignature(JsonSignatureVerificationProcessor.java:178)
   at com.vmware.vapi.cis.authn.json.JsonSignatureVerificationProcessor.process(JsonSignatureVerificationProcessor.java:133)
   at com.vmware.vapi.protocol.server.msg.json.JsonServerConnection.requestReceived(JsonServerConnection.java:171)
   at com.vmware.vapi.protocol.server.rpc.http.impl.HttpStreamingServlet.doPostImpl(HttpStreamingServlet.java:119)
   at com.vmware.vapi.protocol.server.rpc.http.impl.HttpStreamingServlet.doPost(HttpStreamingServlet.java:88)

Environment

VMware vCenter Server 7.0.x

Resolution

This is a known issue affecting vCenter Server 7.x. Currently, there is no resolution.

Workaround:
To workaround the issue, please follow the below steps to reset the STS certificate :

Note: These steps are applicable only if we see the error snippets "Failed to find trusted path to signing certificate" & "Unable to find certificate chain" in the trust manager logs - /var/log/vmware/trustmanagement/trustmanagement-svcs.log.

Download the attached fixsts.sh script from this article and upload to the impacted PSC or vCenter Server with Embedded PSC to the /tmp folder.
If the connection to upload to the vCenter by the SCP client is rejected, run this from an SSH session to the vCenter: chsh -s /bin/bash
Connect to the PSC or vCenter Server with an SSH session if you have not already per Step 2.
Navigate to the /tmp directory:

cd /tmp

Run chmod +x fixsts.sh to make the file executable.
Run ./fixsts.sh.
Restart services on all vCenters and/or PSCs in your SSO domain by using below commands:

service-control --stop --all
service-control --start --all

Note: For more details on resetting STS certificate refer this KB article

Attachments

fixsts get_app