For updating private image repository CA certificate to existing legacy clusters.

NOTE: This is applicable to legacy clusters only (not ClusterClass-based clusters).

Symptoms:
There are various components that needs to be updated if you wish to update a custom CA certificate on an already deployed management or workload clusters on Tanzu Kubernetes Grid.

Updating custom CA certificate to existing clusters.

Step 1 - Nodes modification:

     A) On the management cluster context execute the below commands to collect kubeadmconfigtemplates, KubeadmControlPlane and MachineDeployment names.

kubectl get kubeadmconfigtemplates -A
kubectl get KubeadmControlPlane -A
kubectl get machinedeployment -A

     B) Modify/add the new CA certificate on both kubeadmconfigtemplates and KubeadmControlPlane using the below commands: where KCT-NAME and KCP-NAME are the names acquired from step 1.A.

kubectl edit KubeadmConfigTemplate KCT-NAME

       spec:
          template:
            spec:
              files:
              - content: |
                  -----BEGIN CERTIFICATE-----
                  MIIFazCCA1OgAwIBAgIQMfZy08muvIVKdZVDz7/rYzANBgkqhkiG9w0BAQsFADBI
                  [...]
                  yiDghW7antzYL9S1CC8sVgVOwFJwfFXpdiir35mQlySG301V4FsRV+Z0cFp4Ni0=
                  -----END CERTIFICATE-----
                owner: root:root
                path: /etc/ssl/certs/tkg-custom-ca.pem
                permissions: "0644"

kubectl edit KubeadmControlPlane KCP-NAME

       spec:
         kubeadmConfigSpec:
           files:
           - content: |
               -----BEGIN CERTIFICATE-----
               MIIFazCCA1OgAwIBAgIQMfZy08muvIVKdZVDz7/rYzANBgkqhkiG9w0BAQsFADBI
               [...]
               yiDghW7antzYL9S1CC8sVgVOwFJwfFXpdiir35mQlySG301V4FsRV+Z0cFp4Ni0=
               -----END CERTIFICATE-----
             owner: root:root
             path: /etc/ssl/certs/tkg-custom-ca.pem
             permissions: "0644"

     C) Check preKubeadmCommands block on both KubeadmControlPlane and KubeadmConfigTemplate and add the below commands if not already there.

 preKubeadmCommands:
       - '! which rehash_ca_certificates.sh 2>/dev/null || rehash_ca_certificates.sh'
       - '! which update-ca-certificates 2>/dev/null || (mv /etc/ssl/certs/tkg-custom-ca.pem
          /usr/local/share/ca-certificates/tkg-custom-ca.crt && update-ca-certificates)'

     D) After modifying the CA certificate, MachineDeployment patching will be needed to start the reconciling the nodes using the below command where MACHINE-DEPLOYMENT is the name acquired from step 1.A.

kubectl patch machinedeployment MACHINE-DEPLOYMENT --type merge -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"date\":\"`date +'%s'`\"}}}}}"

       Note: Executing this command triggers a rolling update of the cluster nodes and updates their timestamp.

Step 2 - Kapp controller modification for the management cluster.

     A) Edit configmap kapp-controller-config in tkg-system namespace. (update the certificate under data:caCerts:)

kubectl edit cm kapp-controller-config -n tkg-system

        data:
          caCerts: |
            -----BEGIN CERTIFICATE-----
            MIIFazCCA1OgAwIBAgIQMfZy08muvIVKdZVDz7/rYzANBgkqhkiG9w0BAQsFADBI
            [...]
            yiDghW7antzYL9S1CC8sVgVOwFJwfFXpdiir35mQlySG301V4FsRV+Z0cFp4Ni0=
            -----END CERTIFICATE-----

     B) Delete Kapp controller pod in tkg-system namespace for reconciliation.


Step 3 - tkr controller modification for the management cluster.

     A) Edit configmap tkr-controller-config in tkr-system namespace. (update the certificate under data:caCerts:)

kubectl edit cm tkr-controller-config -n tkr-system

       data:
         caCerts: |
           -----BEGIN CERTIFICATE-----
           MIIFazCCA1OgAwIBAgIQMfZy08muvIVKdZVDz7/rYzANBgkqhkiG9w0BAQsFADBI
           [...]
           yiDghW7antzYL9S1CC8sVgVOwFJwfFXpdiir35mQlySG301V4FsRV+Z0cFp4Ni0=
           -----END CERTIFICATE-----

     B) Delete tkr-controller pod in tkr-system for reconciliation.


Step 4 - kapp controller addon modification for workload cluster. (repeat for all workload clusters)

     A) Decode WORKLOAD-kapp-controller-addon secret value.yaml encrypted section. where WORKLOAD is the name of the workload cluster.

kubectl get secret WORKLOAD-kapp-controller-addon -o jsonpath="{.data.values\.yaml}" | base64 -d

     B) Modify the certificate.

       config:
         caCerts: |
           -----BEGIN CERTIFICATE-----
           MIIFazCCA1OgAwIBAgIQMfZy08muvIVKdZVDz7/rYzANBgkqhkiG9w0BAQsFADBI
          [...]
          yiDghW7antzYL9S1CC8sVgVOwFJwfFXpdiir35mQlySG301V4FsRV+Z0cFp4Ni0=
          -----END CERTIFICATE-----

     C) Base64 Encode the value.yaml modified value.

     D) Edit WORKLOAD-kapp-controller-addon secret by replacing value.yaml value with the modified encoded value.where WORKLOAD is the name of the workload cluster.

kubectl edit secret WORKLOAD-kapp-controller-addon

      This will reconcile the certificate to the data-value secret and consequently to the kapp configmap on the workload cluster. 

      You can verify by executing the below on the workload cluster.

kubectl get cm kapp-controller-config -n tkg-system -oyaml

Impact/Risks:
CA certificate authorization between clusters and the private image repository.