VMware Cloud Foundation sos tool returns a failed status for health check
Article ID: 330378
Updated On:
Products
VMware Cloud Foundation
Issue/Introduction
Symptoms:
- VMware Cloud Foundation sos tool returns a failed status for health-check. A message similar to the following is seen at the end of the sos commands output:
Operation failed for : [HEALTH-CHECK]
- Running the /opt/vmware/sddc-support/sos --health-check command shows that one or more ESXi hosts are in a RED state:
General : RED
+-----+-----------------------------------------+-------------------------------------------------------------+--------+
| SL# | Area | Title | State |
+-----+-----------------------------------------+-------------------------------------------------------------+--------+
| 1 | ESXi : 192.168.100.103 | ESXi entries across all sources | RED |
| 2 | ESXi : r1n0.vrack.vsphere.local | ESXi entries across all sources | GREEN |
| | | Check for error dumps in ESXi | YELLOW |
| | | Operational status | GREEN |
| 3 | ESXi : r1n1.vrack.vsphere.local | ESXi entries across all sources | GREEN |
| | | Check for error dumps in ESXi | YELLOW |
| | | Operational status | GREEN |
| 4 | ESXi : r1n2.vrack.vsphere.local | ESXi entries across all sources | GREEN |
| | | Check for error dumps in ESXi | YELLOW |
| | | Operational status | GREEN |
| 5 | ESXi : r1n4.vrack.vsphere.local | ESXi entries across all sources | GREEN |
| | | Check for error dumps in ESXi | GREEN |
| | | Operational status | GREEN |
| 6 | ESXi : r1n5.vrack.vsphere.local | ESXi entries across all sources | GREEN |
| | | Check for error dumps in ESXi | GREEN |
| | | Operational status | GREEN |
| 7 | ESXi : r1n6.vrack.vsphere.local | ESXi entries across all sources | GREEN |
| | | Check for error dumps in ESXi | GREEN |
| | | Operational status | GREEN |
| 8 | NSX : nsx-manager-1.vrack.vsphere.local | Cluster status [10.6.0.21 -> 10.6.0.22] | GREEN |
| | | Cluster status [10.6.0.22 -> 10.6.0.20] | GREEN |
| | | Cluster status [10.6.0.22 -> 10.6.0.21] | GREEN |
| | | Controller Node-d021215e-63dc-4362-ab5d-c19c4077778d status | GREEN |
| | | Controller Node-017744b8-5df6-4216-95ec-383cbf2b24e9 status | GREEN |
| | | Controller Node-fda74fae-6d56-4c6e-b573-0584f0bb3e8f status | GREEN |
| | | NSX Manager Status | GREEN |
| | | NSX Host Preparation status for Cluster : vRack-Cluster | GREEN |
+-----+-----------------------------------------+-------------------------------------------------------------+--------+
+-----+-----------------------------------------+-------------------------------------------------------------+--------+
| SL# | Area | Title | State |
+-----+-----------------------------------------+-------------------------------------------------------------+--------+
| 1 | ESXi : 192.168.100.103 | ESXi entries across all sources | RED |
| 2 | ESXi : r1n0.vrack.vsphere.local | ESXi entries across all sources | GREEN |
| | | Check for error dumps in ESXi | YELLOW |
| | | Operational status | GREEN |
| 3 | ESXi : r1n1.vrack.vsphere.local | ESXi entries across all sources | GREEN |
| | | Check for error dumps in ESXi | YELLOW |
| | | Operational status | GREEN |
| 4 | ESXi : r1n2.vrack.vsphere.local | ESXi entries across all sources | GREEN |
| | | Check for error dumps in ESXi | YELLOW |
| | | Operational status | GREEN |
| 5 | ESXi : r1n4.vrack.vsphere.local | ESXi entries across all sources | GREEN |
| | | Check for error dumps in ESXi | GREEN |
| | | Operational status | GREEN |
| 6 | ESXi : r1n5.vrack.vsphere.local | ESXi entries across all sources | GREEN |
| | | Check for error dumps in ESXi | GREEN |
| | | Operational status | GREEN |
| 7 | ESXi : r1n6.vrack.vsphere.local | ESXi entries across all sources | GREEN |
| | | Check for error dumps in ESXi | GREEN |
| | | Operational status | GREEN |
| 8 | NSX : nsx-manager-1.vrack.vsphere.local | Cluster status [10.6.0.21 -> 10.6.0.22] | GREEN |
| | | Cluster status [10.6.0.22 -> 10.6.0.20] | GREEN |
| | | Cluster status [10.6.0.22 -> 10.6.0.21] | GREEN |
| | | Controller Node-d021215e-63dc-4362-ab5d-c19c4077778d status | GREEN |
| | | Controller Node-017744b8-5df6-4216-95ec-383cbf2b24e9 status | GREEN |
| | | Controller Node-fda74fae-6d56-4c6e-b573-0584f0bb3e8f status | GREEN |
| | | NSX Manager Status | GREEN |
| | | NSX Host Preparation status for Cluster : vRack-Cluster | GREEN |
+-----+-----------------------------------------+-------------------------------------------------------------+--------+
- The host in question has been decommissioned from VMware Cloud Foundation.
Cause
This issue occurs when the decommissioned host is not removed from the postgres database.
Resolution
This is a known issue affecting VMware Cloud Foundation 2.2. Currently, there is no resolution.
Workaround:
To work around this issue:
Note: At this point, running the sos command should not produce the error noted in the Symptoms section.
Workaround:
To work around this issue:
- In the SDDC Manager UI, on the Status > Workflow Tasks page, verify that there are no Decommission workflow tasks in process.
- Download the attached 000051992_decommission_cleanup.zip file. Use a file transfer program to copy the file to the /tmp folder on the SDDC Manager Controller virtual machine.
- Log in to the SDDC Manager Controller virtual machine as the root user and extract the contents of the /tmp/000051992_decommission_cleanup.zip file:
unzip -d /tmp /tmp/000051992_decommission_cleanup.zip
- Execute the extracted /tmp/decommission_cleanup.py script to remove the offending database entry related to the decommissioned host:
python /tmp/decommission_cleanup.py
Note: The following prompt will be displayed:
This operation has to be performed when there is no Decommission workflow in progress.
Enter y to proceed with running the script.
Note: The following prompt will be displayed:
This operation has to be performed when there is no Decommission workflow in progress.
Enter y to proceed with running the script.
Note: At this point, running the sos command should not produce the error noted in the Symptoms section.
Additional Information
To be alerted when this document is updated, click the Subscribe to Article link in the Actions box.
Attachments
Feedback
Yes
No
Powered by
