Adding a host or a cluster to an existing VMware Cloud Foundation 3.7 workload domain fails
Article ID: 316934
Updated On:
Products
VMware Cloud Foundation
Issue/Introduction
Symptoms:
- Adding a host or a cluster to an existing VMware Cloud Foundation 3.7 workload domain fails
- A password rotation against all components in the affected workload domain was recently completed.
- No new metrics are being received by one or more vCenter Server adapters in vRealize Log Insight.
- You see messages similar to the following in the /var/log/vmware/domainmanager/domainmanager.log file on the SDDC Manager VM:
2019-02-28T09:52:49.878+0000 DEBUG [601be10b47ffecbe,87ac] [c.v.e.s.c.util.HttpClientService,pool-2-thread-1] Making request: POST https://load-balancer.vrack.vsphere.local:9543/api/v1/sessions
2019-02-28T09:52:50.082+0000 DEBUG [601be10b47ffecbe,87ac] [c.v.e.s.c.util.HttpClientService,pool-2-thread-1] Received response: for POST request with host: load-balancer.vrack.vsphere.local, port: 9543, isSecure: true, path: /api/v1/sessions, queryParamMap: null, headers: {Content-type=application/json, Accept=application/json}
2019-02-28T09:52:50.082+0000 DEBUG [601be10b47ffecbe,87ac] [c.v.e.s.c.util.HttpClientService,pool-2-thread-1] Response Body: {"userId":"76fb1c40-2b33-4ae3-9878-d55c82a11707","sessionId":"fllIpiI9O4ZNSk5S1FIwQAG1UjEt3PBzk6BEzLuuNzbeQrcBga8OyrcCBw6YdG/Le8bY8qp1x086Y2RA1lg5dmuMMBHQl5oX605M/DTR5tqVg0kTXlKk33GTAJ65gksbYnmRNIlO/jQPi5kLAOkeGkrxiyu6Kap9fEJNEQRsV+t9K2KNyUunh4K9xmnx/E63jsiSZ2F4DsgFqP3VqmuUc5SOoP6K9yK2uWQnonHb/dN9DSTpuU79m61I8WtuvOCrGJgn20nmim1k4eMgvaSD3w==","ttl":1800}
2019-02-28T09:52:50.084+0000 DEBUG [601be10b47ffecbe,87ac] [c.v.e.s.c.util.HttpClientService,pool-2-thread-1] Starting PUT request from host: load-balancer.vrack.vsphere.local, port: 9543, isSecure: true, path: /api/v1/vsphere/vi-vc-1.vrack.vsphere.local/hosts, queryParamMap: null, headers: {Content-type=application/json, Accept=application/json}
2019-02-28T09:52:50.084+0000 DEBUG [601be10b47ffecbe,87ac] [c.v.e.s.c.util.HttpClientService,pool-2-thread-1] Making request: PUT https://load-balancer.vrack.vsphere.local:9543/api/v1/vsphere/vi-vc-1.vrack.vsphere.local/hosts
2019-02-28T09:53:08.758+0000 DEBUG [601be10b47ffecbe,87ac] [c.v.e.s.c.util.HttpClientService,pool-2-thread-1] Received response with status: 401 reason: PUT ... for Unauthorized request with host: load-balancer.vrack.vsphere.local, port: 9543, isSecure: true, path: /api/v1/vsphere/vi-vc-1.vrack.vsphere.local/hosts, queryParamMap: null, headers: {Content-type=application/json, Accept=application/json}
2019-02-28T09:53:08.759+0000 DEBUG [601be10b47ffecbe,87ac] [c.v.e.s.c.util.HttpClientService,pool-2-thread-1] Response Body: {"errorMessage":"Invalid username or password.","errorCode":"*****","errorDetails":{"errorCode":"com.vmware.loginsight.api.errors.vsphere.unauthorized"}}
2019-02-28T09:53:08.761+0000 ERROR [601be10b47ffecbe,87ac] [c.v.e.s.c.s.l.i.LogInsightApiConfigurator,pool-2-thread-1] POST request to vRLI load-balancer.vrack.vsphere.local failed. Response code 401
2019-02-28T09:53:08.761+0000 WARN [601be10b47ffecbe,87ac] [c.vmware.vcf.domainmanager.AppConfig,pool-2-thread-1] Attempt #5: Retryable method public void com.vmware.evo.sddc.common.services.loginsight.impl.LogInsightApiConfigurator.retryPutRequest(java.lang.Object,java.lang.String,int) threw exception Failed post request in vRLI load-balancer.vrack.vsphere.local
2019-02-28T09:53:08.761+0000 ERROR [601be10b47ffecbe,87ac] [c.v.e.s.c.f.l.i.ConfigureVCenterInLogInsightWithSyslog,pool-2-thread-1] Failed to configure vCenter vi-vc-1.vrack.vsphere.local in vRLI load-balancer.vrack.vsphere.local
2019-02-28T09:53:08.764+0000 ERROR [601be10b47ffecbe,87ac] [c.v.e.s.o.model.error.ErrorFactory,pool-2-thread-1] [6AV1OO] VRLI_VSPHERE_HOSTS_CONFIGURATION_FAILURE Cannot configure hosts from vCenter vi-vc-1.vrack.vsphere.local in vRLI load-balancer.vrack.vsphere.local
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Cannot configure hosts from vCenter vi-vc-1.vrack.vsphere.local in vRLI load-balancer.vrack.vsphere.local
at com.vmware.evo.sddc.common.fsmplugins.loginsight.impl.BaseVCenterInLogInsightConfiguration.execute(BaseVCenterInLogInsightConfiguration.java:84)
at com.vmware.evo.sddc.common.fsmplugins.loginsight.impl.ConfigureVCenterInLogInsightWithSyslog.execute(ConfigureVCenterInLogInsightWithSyslog.java:36)
at com.vmware.evo.sddc.common.fsmplugins.loginsight.impl.ConfigureVCenterInLogInsightWithSyslog.execute(ConfigureVCenterInLogInsightWithSyslog.java:19)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.lambda$static$0(FsmActionState.java:14)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:62)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:164)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:151)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:324)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.processTask(ProcessingTaskSubscriber.java:417)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:96)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:55)
at reactor.bus.EventBus$3.accept(EventBus.java:317)
at reactor.bus.EventBus$3.accept(EventBus.java:310)
Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
2019-02-28T09:52:50.082+0000 DEBUG [601be10b47ffecbe,87ac] [c.v.e.s.c.util.HttpClientService,pool-2-thread-1] Received response: for POST request with host: load-balancer.vrack.vsphere.local, port: 9543, isSecure: true, path: /api/v1/sessions, queryParamMap: null, headers: {Content-type=application/json, Accept=application/json}
2019-02-28T09:52:50.082+0000 DEBUG [601be10b47ffecbe,87ac] [c.v.e.s.c.util.HttpClientService,pool-2-thread-1] Response Body: {"userId":"76fb1c40-2b33-4ae3-9878-d55c82a11707","sessionId":"fllIpiI9O4ZNSk5S1FIwQAG1UjEt3PBzk6BEzLuuNzbeQrcBga8OyrcCBw6YdG/Le8bY8qp1x086Y2RA1lg5dmuMMBHQl5oX605M/DTR5tqVg0kTXlKk33GTAJ65gksbYnmRNIlO/jQPi5kLAOkeGkrxiyu6Kap9fEJNEQRsV+t9K2KNyUunh4K9xmnx/E63jsiSZ2F4DsgFqP3VqmuUc5SOoP6K9yK2uWQnonHb/dN9DSTpuU79m61I8WtuvOCrGJgn20nmim1k4eMgvaSD3w==","ttl":1800}
2019-02-28T09:52:50.084+0000 DEBUG [601be10b47ffecbe,87ac] [c.v.e.s.c.util.HttpClientService,pool-2-thread-1] Starting PUT request from host: load-balancer.vrack.vsphere.local, port: 9543, isSecure: true, path: /api/v1/vsphere/vi-vc-1.vrack.vsphere.local/hosts, queryParamMap: null, headers: {Content-type=application/json, Accept=application/json}
2019-02-28T09:52:50.084+0000 DEBUG [601be10b47ffecbe,87ac] [c.v.e.s.c.util.HttpClientService,pool-2-thread-1] Making request: PUT https://load-balancer.vrack.vsphere.local:9543/api/v1/vsphere/vi-vc-1.vrack.vsphere.local/hosts
2019-02-28T09:53:08.758+0000 DEBUG [601be10b47ffecbe,87ac] [c.v.e.s.c.util.HttpClientService,pool-2-thread-1] Received response with status: 401 reason: PUT ... for Unauthorized request with host: load-balancer.vrack.vsphere.local, port: 9543, isSecure: true, path: /api/v1/vsphere/vi-vc-1.vrack.vsphere.local/hosts, queryParamMap: null, headers: {Content-type=application/json, Accept=application/json}
2019-02-28T09:53:08.759+0000 DEBUG [601be10b47ffecbe,87ac] [c.v.e.s.c.util.HttpClientService,pool-2-thread-1] Response Body: {"errorMessage":"Invalid username or password.","errorCode":"*****","errorDetails":{"errorCode":"com.vmware.loginsight.api.errors.vsphere.unauthorized"}}
2019-02-28T09:53:08.761+0000 ERROR [601be10b47ffecbe,87ac] [c.v.e.s.c.s.l.i.LogInsightApiConfigurator,pool-2-thread-1] POST request to vRLI load-balancer.vrack.vsphere.local failed. Response code 401
2019-02-28T09:53:08.761+0000 WARN [601be10b47ffecbe,87ac] [c.vmware.vcf.domainmanager.AppConfig,pool-2-thread-1] Attempt #5: Retryable method public void com.vmware.evo.sddc.common.services.loginsight.impl.LogInsightApiConfigurator.retryPutRequest(java.lang.Object,java.lang.String,int) threw exception Failed post request in vRLI load-balancer.vrack.vsphere.local
2019-02-28T09:53:08.761+0000 ERROR [601be10b47ffecbe,87ac] [c.v.e.s.c.f.l.i.ConfigureVCenterInLogInsightWithSyslog,pool-2-thread-1] Failed to configure vCenter vi-vc-1.vrack.vsphere.local in vRLI load-balancer.vrack.vsphere.local
2019-02-28T09:53:08.764+0000 ERROR [601be10b47ffecbe,87ac] [c.v.e.s.o.model.error.ErrorFactory,pool-2-thread-1] [6AV1OO] VRLI_VSPHERE_HOSTS_CONFIGURATION_FAILURE Cannot configure hosts from vCenter vi-vc-1.vrack.vsphere.local in vRLI load-balancer.vrack.vsphere.local
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Cannot configure hosts from vCenter vi-vc-1.vrack.vsphere.local in vRLI load-balancer.vrack.vsphere.local
at com.vmware.evo.sddc.common.fsmplugins.loginsight.impl.BaseVCenterInLogInsightConfiguration.execute(BaseVCenterInLogInsightConfiguration.java:84)
at com.vmware.evo.sddc.common.fsmplugins.loginsight.impl.ConfigureVCenterInLogInsightWithSyslog.execute(ConfigureVCenterInLogInsightWithSyslog.java:36)
at com.vmware.evo.sddc.common.fsmplugins.loginsight.impl.ConfigureVCenterInLogInsightWithSyslog.execute(ConfigureVCenterInLogInsightWithSyslog.java:19)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.lambda$static$0(FsmActionState.java:14)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:62)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:164)
at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:151)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:324)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.processTask(ProcessingTaskSubscriber.java:417)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:96)
at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:55)
at reactor.bus.EventBus$3.accept(EventBus.java:317)
at reactor.bus.EventBus$3.accept(EventBus.java:310)
Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
Environment
VMware Cloud Foundation 3.7.x
Cause
This issue can occur when the password rotation task does not properly save the updated password for the [email protected] account for the vCenter Server adapter in vRealize Log Insight.
Resolution
This is a known issue affecting VMware Cloud Foundation 3.7. There is currently no resolution.
Workaround:
Use the following steps to workaround this issue: Run the lookup_passwords command on the SDDC Manager VM to obtain the load-balancer address for vRealize Log Insight as well as the [email protected] account password for each vCenter Server. Log in to the load-balancer address for vRealize Log Insight. Navigate to Administration , Integration , vSphere . Click on the Edit this configuration button for the first vCenter Server adapter. Check the box next to Update Password and enter the [email protected] account's password for that vCenter Server instance. Click the TEST CONNECTION button. If the test is successful, click the SAVE button. Repeat Steps 4 through 6 for any other vCenter Server adapter instances. Restart the failed workflow in the SDDC Manager UI.
Workaround:
Use the following steps to workaround this issue: Run the lookup_passwords command on the SDDC Manager VM to obtain the load-balancer address for vRealize Log Insight as well as the [email protected] account password for each vCenter Server. Log in to the load-balancer address for vRealize Log Insight. Navigate to Administration , Integration , vSphere . Click on the Edit this configuration button for the first vCenter Server adapter. Check the box next to Update Password and enter the [email protected] account's password for that vCenter Server instance. Click the TEST CONNECTION button. If the test is successful, click the SAVE button. Repeat Steps 4 through 6 for any other vCenter Server adapter instances. Restart the failed workflow in the SDDC Manager UI.
Additional Information
To be alerted when this article is updated, click the subscribe button [cid:[email protected]] . For more information on KB subscription features, see the Knowledge Base Article FAQs: How to Subscribe to VMware Knowledge Base Articles (76417).
Feedback
Yes
No
Powered by
