• You see that deploying, upgrading, extending and deleting Tanzu Kubernetes Grid (TKG) clusters using an account than the [email protected] account.fails with a permission error.

• Creating the management cluster failed because the TKG user didn't have permissions on the Distributed Switch object

++/var/log/pods/capv-system_capv-controller-manager-546d5b4b78-k9z8l_4f06f1b2-d130-472c-a67a-29cafd0859d1/manager/0.log++

2020-06-29T18:26:31.011719996Z stderr F E0629 18:26:31.011472 1 controller.go:258] controller-runtime/controller "msg"="Reconciler error" "error"="failed to reconcile VM: error getting network specs for \"infrastructure.cluster.x-k8s.io/v1alpha3, Kind=VSphereVM tkg-system/tkg7mgmt-tkg-system-lb\": 
unable to create new ethernet card backing info for network \"DSwitch-Management\" on \"infrastructure.cluster.x-k8s.io/v1alpha3, Kind=VSphereVM tkg-system/tkg7mgmt-tkg-system-lb\": 
failed to create EthernetCardBackingInfo for /RegionA01/network/DSwitch-Management: System.Read privilege required for config.distributedVirtualSwitch" "controller"="vspherevm" "request"={"Namespace":"tkg-system","Name":"tkg7mgmt-tkg-system-lb"}

• ​​​​Creating the management cluster failed because the TKG user didn't have Virtual Machine permissions to "Add or remove device" for the worker nodes.

vCenter Event :
 Description:
07/01/2020, 8:18:17 PMPrivilege check failed for user VSPHERE.LOCAL\blueuser
for missing permission VirtualMachine.Config.AddRemoveDevice. Session user
performing the check:
Related events:
 There are no related events.

• Creating a cluster with a worker node that has a different disk size fails since the TKG user didn't have Virtual machine "Extend virtual disk" permission.
  

Type:Warning
Target: RegionA01
Description:
 07/02/2020, 1:38:25 AMPrivilege check failed for user VSPHERE.LOCAL\blueuser 
for missing permission VirtualMachine.Config.DiskExtend. Session user performing the check:

• Creating the PVC pod fails (pending state) since the TKG user doesn't have:-
  ​​​​​- Profile-driven storage:  "Profile-driven storage view"
  - Datastore: "Browse datastore" and "Low level file operations"
  
  

++/var/log/ pods/kube-system_vsphere-csi-controller-6c46bb949c-47n2f_a8b17c79-b9a6-429e-a4fb-ac28bb382c62/csi-provisioner/0.log++

2020-07-02T18:56:07.047362808Z stderr F I0702 18:56:07.046838       1 event.go:255] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"mysql-pv-claim", UID:"01833c26-3b02-4b56-9e40-869f66caa13a", APIVersion:"v1", ResourceVersion:"34264", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' 
failed to provision volume with StorageClass "k8s-policy": rpc error: code = Internal desc = Failed to create volume. Error: ServerFaultCode: NoPermission

++vmware-sps/sps.log++

2020-07-02T17:06:51.092Z [pool-3-thread-15] INFO  opId=01b43754-5b12-423f-98a8-3150f1854692 com.vmware.vim.storage.common.security.CommonActivationValidator - [getUserFromVpxdClientManager] 
Validating session for user VSPHERE.LOCAL\blueuser for method PbmQueryProfile having correlator 231449
2020-07-02T17:06:51.111Z [pool-3-thread-15] ERROR opId=01b43754-5b12-423f-98a8-3150f1854692 com.vmware.vim.storage.common.security.CommonActivationValidator - Failed to validate session

• Velero backups fail to successfully complete and fails on uploading a snapshot since the the TKG user is missing the following Global permissions
  

Disable methods
Enable methods
Licenses

++velero+tmp/vmware-root/vixDiskLib.log++
2020-08-17T18:41:18.267Z| host-13| E110: VixDiskLib: VixDiskLib_PrepareForAccess: Disable Storage VMotion failed.
Error 3014 (Insufficient permissions in the host operating system) (No permission to perform this action.) at 5001.

++Velero datamanager pod logs++

kubectl logs datamgr-for-vsphere-plugin-bqvs5 -n velero

│ 2020-08-17T18:32:02.093Z warning -[00008] [Originator@6876 sub=vimaccess] cannot get thumbprint: SSL error code '151441516', exception: 'Wrong X.509 Certificate format'
│ 2020-08-17T18:32:02.094Z warning -[00032] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3
│ time="2020-08-17T18:32:02Z" level=error msg="Failed at copying to remote repository" Local PEID="ivd:7e15fbcf-aa75-4162-b55b-3a84e987ec1d:d12d3b07-95d6-488c-9a99-56ea6869fg"
error="Prepare for access failed. The error code is 3014. with error code: 3014" error.file

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

The Required Permission for the TKG Roles Are:-

Datastore

• Allocate space
• Browse datastore
• Low level file operations

• Disable methods

• Enable methods

• Licenses

• Cloud Admin (vSphere 7 with Kubernetes only)

• Assign network

• Assign virtual machine to resource pool

• Message
• Validate session

• Profile-driven storage view

• Import

• Configuration

  • Change Configuration

  • Add existing disk

  • Add new disk

  • Add or remove device

  • Advanced configuration
  • Change CPU count
  • Change Memory
  • Change Settings
  • Configure Raw device
  • Extend virtual disk
  • Modify device settings
  • Remove disk
  • Create from existing
  • Remove
• Interaction 
  • Power off
  • Power on
• Provisioning 
  • Deploy template

The Objects that need to get assigned this role

1. The vCenter.

2. Datacenters or datacenter folders.

3. Datastores or datastore folders 

4. Hosts and  clusters.

5. The deployed  TKG OVF templates.

6. TKG resource pools  (With Propagate to children).

7. The Networks to which the clusters will be assigned to.

ex "DSwitch-Management" Distributed Port Group.

8. The Distributed Switch.

9. The TKG VM and Template folders (With Propagate to children).