AD FS logins fail using the JRE truststore after upgrading vCenter Server

Products

VMware vCenter Server

Issue/Introduction

This article describes how to migrate AD FS server certificates from the JRE truststore to the Trusted Root Certificates Store (also known as the VMware Endpoint Certificate Store, or VECS). It also details the specific issues that occur after upgrade when the JRE truststore is still being used to store the AD FS server certificates. Going forward, all AD FS server communication certificates need to be added to VECS instead of the JRE truststore.

Symptoms:

After upgrading to vSphere 7.0 P02, if you were previously using the JRE truststore on vCenter Server to establish secure connections to your AD FS server, you may see the following error message in the vCenter login UI after authenticating to the AD FS server:

[400] An error occurred while processing an OAuth 2 authorization code assertion request

This corresponds with the following errors in the /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log file:

[2020-12-01T19:57:40.016Z] [INFO ] http-nio-5090-exec-1070000355 100067 ###### c.v.s.c.r.interceptor.retry.DefaultHttpRequestRetryInterceptor failure on https://adfs.pslabs.eng.vmware.com/adfs/oauth2/token/?***** status sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[2020-12-01T19:57:40.017Z] [ERROR] http-nio-5090-exec-1070000355 100067 ###### c.v.vsphere.client.security.oauth2.Oauth2CodeResponseHandler Oauth2 Authorization code assertion failed com.vmware.skyscraper.common.restclient.RestclientException: null
    at com.vmware.skyscraper.common.restclient.interceptor.retry.BaseRetryInterceptor.lambda$intercept$0(BaseRetryInterceptor.java:69)
    at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:287)
    at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:164)
    at com.vmware.skyscraper.common.restclient.interceptor.retry.BaseRetryInterceptor.intercept(BaseRetryInterceptor.java:62)
...

Environment

VMware vCenter Server 7.0.x

Cause

Prior to vSphere 7.0 Update 1, you had to import the certificates for your AD FS server into the JRE truststore. Starting in vSphere 7.0 Update 1, you can import them into VECS instead. If you did not choose to move your AD FS server certificates into VECS in vSphere 7.0 Update 1 and you continued to use the JRE truststore, then when you upgrade to vSphere 7.0 P02, any certificates you may have added to the JRE truststore are not preserved during the upgrade. As a result, vSphere cannot establish a secure connection to your AD FS server because it no longer has access to your server's trusted certificates.

Resolution

To resolve this issue, import the AD FS server root certificate into the Trusted Root Certificates Store (VECS); the certificate requires to be a Base64 encoded certificate. This can be done at any time before or after the upgrade. As soon as the certificate is added to VECS, vSphere will begin using that certificate to establish secure communication with the AD FS server. For more information about adding the AD FS root certificate to VECS, see Use the Trusted Root Certificates Store Instead of the JRE truststore

Note: Services do not need to be restarted after adding the certificate to VECS.