VCF Pre-check fails on NSX-T manager with "NSX-T Manager Audit failed with unknown exception"

Products

VMware Cloud Foundation VMware NSX Networking

Issue/Introduction

Symptoms:

VCF Pre-check fails on NSX-T manager with "NSX-T Manager Audit failed with unknown exception"
SDDC manager logs report similar to below

2021-03-22T10:36:15.248+0000 ERROR [vcf_lcm,0c4fe0dfc82a4479,60c3,auditId=db1ff808-3565-44b8-82be-a48376ce2a92,resourceType=NSX_T_MANAGER,resourceId=abc.domian.com] [c.v.e.s.l.p.impl.nsxt.NsxtAud
itImpl,pool-6-thread-9] Error auditing NSX-T Cluster labc.domain.com with exception {}
com.vmware.evo.sddc.lcm.model.error.LcmException: Failed to load NSX-T Cluster from the Inventory
        at com.vmware.evo.sddc.lcm.primitive.impl.nsxt.NsxtInventoryLoader.loadNsxtInventory(NsxtInventoryLoader.java:87)
        at com.vmware.evo.sddc.lcm.primitive.impl.nsxt.NsxtAuditImpl.doAudit(NsxtAuditImpl.java:68)
        at com.vmware.evo.sddc.lcm.audit.NsxtAuditService.doAudit(NsxtAuditService.java:120)
        at com.vmware.evo.sddc.lcm.audit.NsxtInventoryAuditScheduler.auditNsxtInventory(NsxtInventoryAuditScheduler.java:84)
        at com.vmware.evo.sddc.lcm.audit.NsxtInventoryAuditScheduler$$FastClassBySpringCGLIB$$2ae06d0f.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
Caused by: java.lang.RuntimeException: NSX-T Upgrade Coordinator could not be fetched
        at com.vmware.evo.sddc.lcm.adapter.inventory.impl.InventoryClientImpl.getNsxtUcConnection(InventoryClientImpl.java:1706)
        at com.vmware.evo.sddc.lcm.adapter.inventory.impl.InventoryClientImpl$$FastClassBySpringCGLIB$$82f85f63.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:687)
        at com.vmware.evo.sddc.lcm.adapter.inventory.impl.InventoryClientImpl$$EnhancerBySpringCGLIB$$985e7950.getNsxtUcConnection(<generated>)
        at com.vmware.evo.sddc.lcm.primitive.impl.nsxt.NsxtInventoryLoader.loadNsxtComponents(NsxtInventoryLoader.java:351)
        at com.vmware.evo.sddc.lcm.primitive.impl.nsxt.NsxtInventoryLoader.loadNsxtInventory(NsxtInventoryLoader.java:83)
        ... 35 common frames omitted

NOTE:The preceding log excerpts are only examples.Date,time and environmental variables may vary depending on your environment.

Environment

VMware NSX-T Data Center

Resolution

To resolve this issue.
1. Check the default response to below API command returns an FQDN
root@sddc-manager [ /home/vcf ]# curl -k -u admin -X GET -H "Content-Type:application/json" https://<nsxt-manager-ip>/api/v1/node/services/install-upgrade | json_pp
{
   "service_name" : "install-upgrade",
   "_self" : {
      "rel" : "self",
      "href" : "/node/services/install-upgrade"
   },
   "_schema" : "NodeInstallUpgradeServiceProperties",
   "service_properties" : {
      "enabled" : true,
      "enabled_on" : "abc.domian.com" <---- Here it returned FQDN instead of IP
   }
}

2. If above output enabled_on provides FQDN then follow below steps
i). SSH into the node that you want to set as an orchestrator node, for example IP (10.0.0.xx)

ii). run command
set repository-ip 10.201.x.xx

iii). Re-run above API, and see if it is returning the following expected response

vcf@sddc-manager [ ~ ]$ curl -k -u admin -X GET -H "Content-Type:application/json" https://<nsxt-manager-ip>/api/v1/node/services/install-upgrade | json_pp
{
   "service_properties" : {
      "enabled" : true,
      "enabled_on" : "10.201.x.xx" <-- expected the node where we enable upgrade service
   },
   "_schema" : "NodeInstallUpgradeServiceProperties",
   "_self" : {
      "href" : "/node/services/install-upgrade",
      "rel" : "self"
   },
   "service_name" : "install-upgrade"
}