Upgrading to vCenter 8.0 U1 fails with "Failed to start the Workload Control Plane Service" when sso domain contains uppercase letters
search cancel

Upgrading to vCenter 8.0 U1 fails with "Failed to start the Workload Control Plane Service" when sso domain contains uppercase letters

book

Article ID: 318576

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:
  • Upgrading to vCenter 8.0 U1 fails with "Failed to start the Workload Control Plane Service" in the wizard.
  • Patching a vCenter from an earlier version of 8.0 to 8.0 U1 fails with "Exception occurred in postInstallHook" in the VAMI.
  • The vCenter SSO domain contains uppercase letters. You can see the case of an SSO domain with the following command:
/usr/lib/vmware-vmafd/bin/vmafd-cli get-domain-name --server-name localhost
  • Messages in the /var/log/vmware/wcp/wcpsvc.log file shows a failure like the following:
2023-04-17T13:04:54.017Z error wcp [gcauth/gc_auth.go:94] Failed to parse VC JWKS: invalid character '<' looking for beginning of value
2023-04-17T13:04:54.017Z error wcp [gcauth/gc_auth.go:59] Unable to get VC public key configuration : invalid character '<' looking for beginning of value
2023-04-17T13:04:54.017Z fatal wcp [gcauth/gc_auth.go:47] Unable to get VC public key configuration : invalid character '<' looking for beginning of value


Environment

VMware vCenter Server 8.0.x
VMware vCenter Server 8.0.1

Cause

This issue is due to a flaw in a non-normalized string of the SSO domain being used for comparison with a normalized version.

Resolution

This issue is resolved in vCenter Server 8.0 Update 1b.

Workaround:
To workaround the issue, update the vmwSTSDefaultTenant name in VMDIR to lower-case by following below steps :

Notes:
  • Take concurrent powered-off snapshots of every vCenter in the SSO domain before following these steps.
  • Below steps needs to be performed on the Source vCenter Server Appliance before attempting the upgrade. 
  • Domain name is used as vsphere.local in the commands mentioned below. Please change the values depending on the environment.
    • For example, if the SSO domain name is "vcsso.local", replace "dc=vsphere,dc=local" with the "dc=vcsso,dc=local" and replace "vsphere.local" with "vcsso.local" (these texts are highlighted in Blue).
  • Change the <PASSWORD> field in each example before executing the command.
  1. Confirm Default Tenant has upper-case in VMDIR using ldapsearch command 
ldapsearch -x -h localhost -p 389 -D "cn=administrator,cn=users,dc=vsphere,dc=local" -W -s base -b cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local vmwSTSDefaultTenant

Example:
root@vcsa1 [ ~ ]# ldapsearch -x -h localhost -p 389 -D "cn=administrator,cn=users,dc=vsphere,dc=local" -w "<PASSWORD>" -s base -b cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local vmwSTSDefaultTenant
…
# Tenants, IdentityManager, Services, vsphere.local
dn: cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local
vmwSTSDefaultTenant: Vsphere.Local
…
  1. Verify the wrong OIDC endpoint with the upper case Tenant using the Curl command
curl  -k https://localhost/openidconnect/.well-known/openid-configuration

Example:
root@vcsa1 [ ~ ]#  curl  -k https://localhost/openidconnect/.well-known/openid-configuration

{"response_types_supported":["code","id_token","token id_token"],"jwks_uri":"https:\/\/vcsa1.example.com\/openidconnect\/Vsphere.local","end_session_endpoint":"https:\/\/vcsa1.example.com\/openidconnect\/Vsphere.local","subject_types_supported":["public"],"id_token_signing_alg_values_supported":["RS256"],"issuer":"https:\/\/vcsa1.example.com\/openidconnect\/Vsphere.local","authorization_endpoint":"https:\/\/vcsa1.example.com\/openidconnect\/Vsphere.local","token_endpoint":"https:\/\/vcsa1.example.com\/openidconnect\/Vsphere.local"}root@vcsa1 [ ~ ]# 
 
  1. Change Default Tenant to lower-case using ldapmodify:
ldapmodify -x -h localhost -p 389 -D "cn=administrator,cn=users,dc=vsphere,dc=local" -W <<EOF
dn: cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local
changetype: modify
replace: vmwSTSDefaultTenant
vmwSTSDefaultTenant: vsphere.local
EOF

Example:
root@vcsa1 [ ~ ]# ldapmodify -x -h localhost -p 389 -D "cn=administrator,cn=users,dc=vsphere,dc=local" -w "<PASSWORD>" <<EOF
dn: cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local
changetype: modify
replace: vmwSTSDefaultTenant
vmwSTSDefaultTenant: vsphere.local
EOF
 
modifying entry "cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local"
  1. Restart the STS Service:
service-control --restart vmware-stsd
or
vmon-cli --restart sts

Example:
root@vcsa1 [ ~ ]# vmon-cli --restart sts
Completed Restart service request.
 
  1. Verify correct OIDC endpoints using the Curl command and make sure the domain name is in lower-case:
curl  -k https://localhost/openidconnect/.well-known/openid-configuration

Example:
root@vcsa1 [ ~ ]#  curl  -k https://localhost/openidconnect/.well-known/openid-configuration
{"response_types_supported":["code","id_token","token id_token"],"jwks_uri":"https:\/\/vcsa1.example.com\/openidconnect\/jwks\/vsphere.local","end_session_endpoint":"https:\/\/vcsa1.example.com\/openidconnect\/logout\/vsphere.local","subject_types_supported":["public"],"id_token_signing_alg_values_supported":["RS256"],"issuer":"https:\/\/vcsa1.example.com\/openidconnect\/vsphere.local","authorization_endpoint":"https:\/\/vcsa1.example.com\/openidconnect\/oidc\/authorize\/vsphere.local","token_endpoint":"https:\/\/vcsa1.example.com\/openidconnect\/token\/vsphere.local"}


Attachments

update_default_tenant_kb92436 get_app
Powered by Wolken Software