vSphere with Tanzu Certificate Guide

Products

VMware vSphere ESXi VMware vSphere with Tanzu

Issue/Introduction

This article provides guidance on vSphere with Tanzu Supervisor and Guest Cluster certificate management.

---------------------------------------------------------------------

Certificates for vSphere with Tanzu are utilized in 4 places depending on where the authentication is being performed, these are: vCenter Server, Supervisor Cluster Nodes, Guest Cluster Nodes, ESXi hosts (used to support vSphere POD functionality)

There are 2 trust domains in vSphere with Tanzu:

VC trust domain: The default signer for certificates in this trust domain is the VMware Certificate Authority (VMCA) built into vCenter Server
K8s trust domain: The default signer for certificates in this trust domain is the Kubernetes CA.

---------------------------------------------------------------------

WCP requires certificates for the following operations:

Server certificate for wcpsvc (running on vCenter Server) to talk to the Supervisor Cluster api-servers over the management network
Server certificate for the authproxy server endpoint hosted on Supervisor Cluster VM
Server certificate for the schedext endpoint hosted on Supervisor Cluster VM
Client certificate for the WCP Solution user used by authproxy to query WCP vAPI

Regular k8s operations:

Client certificates for kubelet to authenticate to the API server
Kubelet server certificates for the API server to talk to the kubelets
Server certificate for the API server endpoint (TLS over the workload network)
Client certificates for administrators of the cluster to authenticate to the API server
Client certificates for the API server to talk to the kubelets
Client certificate for the API server to talk to etcd
Client certificate/kubeconfig for the controller manager to talk to the API server
Client certificate/kubeconfig for the scheduler to talk to the API server.
Client and server certificates for the front-proxy

---------------------------------------------------------------------

The following command can be used from SSH to Supervisor or Guest Clusters to identify certificate expiry timeframe:

# find / -type f $ -name "*.cert" -o -name "*.crt" $ -print 2>/dev/null | egrep -v 'ca.crt$|ca-bundle.crt$|kubelet\/pods|var\/lib\/containerd|run\/containerd' | xargs -L 1 -t -i bash -c 'openssl x509 -noout -text -in {}|grep After'

Environment

VMware vSphere 7.0 with Tanzu

Resolution

Delineation of WCP certificates, usage and locations

On supervisor control plane nodes:

WCP specific certificates
Cert Path	Signed By	Used for	Cert Lifetime
/etc/vmware/wcp/tls/vip.crt	VMCA/Custom	TLS certificate served by the nginx proxy running in front of each CP VM on the workload network	1 Year / Custom
/etc/vmware/wcp/tls/mgmt.crt	K8s CA	TLS certificate served by the nginx proxy running in front of each CP VM on the management network	1 Year
/etc/vmware/wcp/tls/ncp/lb-default.cert	VMCA/Custom	Certificate applied to Service IP's built on the Ingress network in NSX-T	1 Year / Custom
/etc/vmware/wcp/tls/wcpusr.cert	VMCA	Client certificate for VC solution user for WCP	2 Years*
/etc/vmware/wcp/tls/schedext.cert	self-signed	TLS certificate served by schedext	2 Years
/etc/vmware/wcp/tls/authproxy.crt	K8s CA	TLS certificate served by authproxy	2 Years
/etc/vmware/wcp/tls/docker-reg.crt	K8s CA	TLS certificate served by the internal docker registry	2 Years
/etc/vmware/wcp/tls/wcpagent.cert	VMCA	Was: TLS certificate for docker registry and authproxy. No long in use after 7.0 U1

Kubernetes internal certificates
Cert Path	Signed By	Used for	Cert Lifetime
/var/lib/kubelet/pki/kubelet.crt	K8s CA	Currently not used. Kubelet serves "content" to metrics servers	1 Year
/etc/kubernetes/pki/scheduler.crt	K8s CA	Used to authenticate with the scheduler pod	1 Year
/etc/kubernetes/pki/apiserver.crt	K8s CA	Used to authenticate with K8s API server	1 Year
/etc/kubernetes/pki/apiserver-etcd-client.crt	K8s CA	Used by API server to authenticate with ETCD	1 Year
/etc/kubernetes/pki/apiserver-kubelet-client.crt	K8s CA	Used by API server to authenticate with kubelet	1 Year
/etc/kubernetes/pki/front-proxy-client.crt	K8s CA		1 Year
/etc/kubernetes/pki/etcd/server.crt	K8s CA	Cert used for ETCD Server authentication	1 Year
/etc/kubernetes/pki/etcd/peer.crt	K8s CA	Cert used for ETCD Peer server authentication	1 Year
/etc/kubernetes/pki/etcd/healthcheck-client.crt	K8s CA		1 Year
/etc/kubernetes/pki/bootstrapper.crt	K8s CA	Used for initial cluster bootstrap and customization	n/a
/etc/kubernetes/pki/front-proxy-ca.crt	K8s CA	K8s Front Proxy certificate authority	10 Year
/etc/kubernetes/pki/etcd/ca.crt	K8s CA	K8s ETCD certificate authority	10 Year
/etc/kubernetes/pki/ca.crt	K8s CA	K8s Cluster certificate authority	10 Year

----------------------------------------------------------------------------------

On ESXi Hosts (Only when WCP is deployed with NSX-T):

Cert Path	Cert Lifetime
/etc/vmware/spherelet/spherelet.crt	1 Year
/etc/vmware/spherelet/client.crt	1 Year

----------------------------------------------------------------------------------

TKGS Guest Cluster Certificates:

TKGS Guest Cluster Control Plane VMs
Cert Path	Cert Lifetime
/var/lib/kubelet/pki/kubelet.crt	1 Year
/etc/kubernetes/pki/apiserver.crt	1 Year
/etc/kubernetes/pki/apiserver-etcd-client.crt	1 Year
/etc/kubernetes/pki/etcd/server.crt	1 Year
/etc/kubernetes/pki/etcd/peer.crt	1 Year
/etc/kubernetes/pki/etcd/healthcheck-client.crt	1 Year
/etc/kubernetes/pki/front-proxy-client.crt	1 Year
/etc/ssl/certs/extensions-tls.crt	10 Year

TKGS Guest Cluster Worker Node VMs
Cert Path	Expiration Date
/var/lib/kubelet/pki/kubelet.crt	1 Year
/etc/ssl/certs/extensions-tls.crt	10 Year

Workaround:

You can use the following kb to rotate the Certificates for Supervisor Control Plane VM's and ESXi hosts here: https://kb.vmware.com/s/article/90627

TKGS Guest Cluster Certificates can be rotated by upgrading the cluster. If they have expired then you can follow this kb to rotate them https://kb.vmware.com/s/article/95425

Additional Information

Impact/Risks:
If certificates expire on the Supervisor or Guest Clusters, access and management of the clusters will fail.

VMware engineering is aware of this and is working to build a script that will allow replacement of Supervisor Cluster certificates manually.

NOTE: If your certificates expire, please notify VMware support for assistance with manual replacement.