Products

VMware vCenter Server

Issue/Introduction

Symptoms:
vSphere with Tanzu supervisor certificates or spherelet certificates have expired or are about to expire.

You can use the following command while connected via SSH into either of the Supervisor Control Plane VMs.

# find / -type f $ -name "*.cert" -o -name "*.crt" $ -print 2>/dev/null | egrep -iv 'ca.crt$|ca-bundle.crt$|kubelet\/pods|var\/lib\/containerd|run\/containerd|backup' | xargs -L 1 -t -i bash -c 'openssl x509 -noout -text -in {}|grep After'

SSH into Supervisor Control Plane VM Instructions: Troubleshooting vSphere with Tanzu (TKGS) Supervisor Control Plane VM's (90194)

Main vSphere with Tanzu Cert Guide

Environment

VMware vCenter Server 7.0.x

Resolution

Install the wcp_cert_manager tool to vCenter

Move the attached file titled wcp_cert_manager.zip to the vCenter Server where vSphere with Tanzu is deployed. (Use WinSCP from Windows OS's if required):

$ scp ./wcp_cert_manager.zip [email protected]:/root

Example Output:

The authenticity of host '192.168.111.135 (192.168.111.135)' can't be established.
ECDSA key fingerprint is SHA256:RkfHc8xvRJ8ihqMD1CTQeMXEPrYJ6yaNEOhwKpCbt3w.

Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.111.135' (ECDSA) to the list of known hosts.

VMware vCenter Server 7.0.3.01000

Type: vCenter Server with an embedded Platform Services Controller

Password:
wcp_cert_manager.zip 100% 8473KB 8.3MB/s 00:00

Unzip the file:

# unzip wcp_cert_manager.zip

Archive: wcp_cert_manager.zip
inflating: certmgr

# ls -l

total 30956
-rwxr-xr-x 1 root root 23019418 Nov 28 01:24 certmgr
-rw-r--r-- 1 root root 8675846 Jan 17 16:09 wcp_cert_manager.zip

Using the wcp_cert_manager tool

Run './certmgr certificates rotate command' to rotate all supervisor control plane certificates and spherelet certificates. Below is an example of a successful cert rotation.

If you have multiple vSphere with Tanzu deployments on your vCenter, then you need to use the -c command to specify the cluster you want to replace certificates on.
In order to gather the cluster id you can run
./certmgr supervisors
ie
root@vxlan-vm-111-100 [ ~ ]# ./certmgr supervisors
2023/02/07 20:46:27 Cluster: domain-c8:79dbdf45-9f6d-4e9d-aa8a-6094fb36c979
IP: 192.168.111.200
Password: FMfPDQso8XSIrS0NS9870KVMuZNBByhgFUhhRNiQrasuQbL24IIwuQejIelQHC8lgQdNsD98KuAj3ga+aHUYGO7J11CCMKEInrxvuABXgMOYfo0EY7Zh4f/Dx/fukKxm8xYkwTtOLIV5Pjz/cMwAQVv/q5fle5WY1QE6pK3ZeLo=

In the above example the cluster id would be "domain-c8:79dbdf45-9f6d-4e9d-aa8a-6094fb36c979"

An example of running the tool on a specific cluster would be.
# ./certmgr certificates rotate -c domain-c8:79dbdf45-9f6d-4e9d-aa8a-6094fb36c979

All logs for this tool are logged under /var/log/vmware/certmgr.log

Additional Information

Main vSphere with Tanzu Cert Page: https://kb.vmware.com/s/article/89324

Updated tool on 2/1/23 to fix an issue with older Supervisor Cluster's not having the /etc/vmware/wcp/tls/ca folder.

Updated on 7/11/23 to fix https://bugzilla.eng.vmware.com/show_bug.cgi?id=3238436

Attachments

wcp_cert_manager get_app

Replace vSphere with Tanzu Supervisor Certificates

Article ID: 322994

Updated On: