Symptoms:
The procedure outlines the steps that you can use to rotate TKGI control plane CA and leaf certificates. The procedure in this article can be used for TKGI version 1.11.x and above.

The following CA and leaf certificates can be rotated using this approach

Certificate Authorities

• uaa_active_pks_saml_key_2018
• pxc_galera_ca
• pxc_server_ca
• kubo_odb_ca_2018

Leaf Certificates

• galera_server_certificate
• telemetry_db_client_2020
• mysql_server_certificate
• mysql_server_certificate_proxy
• pks_db_client_2020
• pks_api_internal_2018
• telemetry_forward_tls_2020

During this certificate rotation procedure your TKGI API and database will become inaccessible while the new certificates are being pushed to the VMs and the API and Database are being restarted.

Important Note

• Please remember to substitute the parameters in the below commands with the relevant parameters from your environments.
• Certification rotation script used here is attached to the KB
• After certificate rotation the leaf certificate pks_api_internal_2018 will still show up as expiring for service-instance_UUID deployments in the Opsmgr UI even after successful certificate rotation. This is because of the way variables are cross references between service-instance deployment manifest and TKGI deployment. It is safe to ignore post certificate rotation as service-instance deployments does not have dependency on pks_api_internal_2018. To verify this is not an issue
  • Make sure that Opsmgr UI does not show expiry entry against TKGI control plane bosh deployment 

Retrieve the TKGI deployment name

bosh deployments --column=name

Using environment '172.30.0.11' as client 'ops_manager'
Name
pivotal-container-service-c1c2109e8bbeaacede93
service-instance_e284311e-f533-45b1-89cd-a35866a0c1ec

Export TKGI deployment manifest

bosh manifest -d pivotal-container-service-c1c2109e8bbeaacede93 > tkgi.yaml

Setup Bosh and credhub credentials

cat > bosh.env << EOF
export BOSH_CLIENT='ops_manager'
export BOSH_CLIENT_SECRET='zypfG-JORm_Kxc0nQaHCvhyrPO95NNvx'
export BOSH_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate
export BOSH_ENVIRONMENT=172.30.0.11
EOF

cat > credhub.env << EOF
export CREDHUB_CLIENT=$BOSH_CLIENT
export CREDHUB_SECRET=$BOSH_CLIENT_SECRET
EOF

Download rotate_control_plane_certs script

Download the rotate_control_plane_certs.sh script attached to this KB and make it executable

chmod +x rotate_control_plane_certs.sh

List currently deployed certificates

./rotate_control_plane_certs.sh -a list_certificates -d pivotal-container-service-c1c2109e8bbeaacede93 &> list_certificates

Regenerate CA and leaf certificates

./rotate_control_plane_certs.sh -a regenerate_certificates -d pivotal-container-service-c1c2109e8bbeaacede93 &> regenerate_certificates.out

Deploy TKGI 

This step will restart the TKGI API and DB processes and push new certificates to the VMs. TKGI control plane will be unavailable while the command below is running

bosh deploy -d pivotal-container-service-c1c2109e8bbeaacede93 tkgi.yaml

List currently deployed certificates

./rotate_control_plane_certs.sh -a list_certificates -d pivotal-container-service-c1c2109e8bbeaacede93 &> list_certificates_new