Unable to create Load Balancer Objects in vSphere with Tanzu with NSX Advanced Load Balancer(Formerly known as AVI) on 7.0U3e or later
Article ID: 323440
Updated On:
Products
Issue/Introduction
Symptoms
AKO pod fails to finish provisioning a load balancer object with this error.
<yyyy-mm-dd>T16:38:16.536Z ESC[33mWARNESC[0m rest/rest_operation.go:279 Error in rest operation is not of type AviError, err: Rest request error, returning to caller, *errors.errorString
<yyyy-mm-dd>T16:38:16.536Z ESC[33mWARNESC[0m rest/dequeue_nodes.go:220 key: admin/domain-c2014--ccams-test-test-control-plane-service, msg: got error while executing rest request: Error during POST: Rest request error, returning to caller, adding to slow retry queue
This issue only happens when you have more than the one Default Service Engine Group configured in AVI.
SE Groups can be viewed in AVI via: Infrastructure- > Service Engine Group
Note: The example below shows 2 Service Engine Groups. If you only have 1 default group here, then you would not be running into this issue.
Issue can happen either:
- After updating VMware vSphere with Tanzu's Supervisor cluster to 7.0U3e
OR
- While deploying VMware vSphere with Tanzu on or after 7.0U3e
You can correlate if supervisor version to VC version via the chart below.
The Supervisor version will look something like "v1.21.0+vmware.wcp.2-vsc0.0.12-18735554"
To correlate this to the chart below, the first part of the version is the k8's version noted in the 4th column, and the other one is the vcs0.0.x.
| Version | Release Date | vCenter build | Supported K8 versions | vSphere with Tanzu Version |
|---|---|---|---|---|
| vCenter Server 7.0 Update 3m | 2023-06-22 | 21784236 | 1.23 1.22 1.21 |
0.0.22 |
| vCenter Server 7.0 Update 3L |
2023-03-30 | 21477706 | 1.23 1.22 1.21 |
0.0.22 |
| vCenter Server 7.0 Update 3k |
2022-02-23 | 21290409 | 1.23 1.22 1.21 |
0.0.21 |
| vCenter Server 7.0 Update 3j | 2022-12-22 | 20990077 | 1.22 1.21 1.20 |
0.0.20 |
| vCenter Server 7.0 Update 3i | 2022-12-8 | 20845200 | 1.22 1.21 1.20 |
0.0.20 |
| vCenter Server 7.0 Update 3h | 2022-09-13 | 20395099 | 1.22 1.21 1.20 |
0.0.19 |
| vCenter Server 7.0 Update 3g | 2022-07-23 | 20150588 | 1.22 1.21 1.20 |
0.0.17 |
| vCenter Server 7.0 Update 3f | 2022-07-12 | 20051473 | 1.22 1.21 1.20 |
0.0.17 |
| vCenter Server 7.0 Update 3e | 2022-05-12 | 19717403 | 1.22 1.21 1.20 |
0.0.15 |
| vCenter Server 7.0 Update 3d | 2022-03-29 | 19480866 | 1.21.0 1.20.2 1.19.1 |
0.0.13 |
| vCenter Server 7.0 Update 3c | 2022-01-27 | 19234570 | 1.21.0 1.20.2 1.19.1 |
0.0.12 |
| vCenter Server 7.0 Update 3a | 2021-10-21 | 18778458 | 1.21.0 1.20.2 1.19.1 |
0.0.12 |
Environment
Cause
Known issue with the AKO pod being unable to deploy lb objects when the backing AVI infrastructure has more than one SE Group.
Resolution
This issue is resolved in 7.0U3k / Supervisor version 0.0.21. The code fix is in the AKO pod on the supervisor which means that you must update the supervisor cluster in addition to updating the VMware vCenter Server.
Workaround
Remove the extra SE Group so you only have one default SE Group.
Feedback
