“Unexpected status code: 503” error in vSAN health check plug-in

Products

VMware vSAN

Issue/Introduction

The error "HTTP status 503" in vSAN health check plug-in is resolved in VMware vSphere 6.0 Update 2. Correct the group permissions of the certificate file to work around this issue.

Symptoms:
When replacing vCenter Server with the SSL certificate Automation tool, you experience these symptoms:

If the VMware vSAN (formerly known as Virtual SAN) Health Check plug-in is deployed and working correctly, you see these symptoms:

The vSphere Web Client screen for the vSAN Health Check plug-in displays an error similar to:

Unexpected status code: 503 org.apache.http.client.ClientProtocolException
In the /storage/log/vmware/vsan-health/vmware-vsan-health-service.log file, you see an entry similar to:

Failed to log into VC, retrying in 10 seconds
When running the vsan.health.cluster_status command in the Ruby vSphere Console (RVC), you see an error similar to:

Failed to get status of cluster Test_Cluster: Got HTTP 503: Service unavailable

If the vSAN Health Check plug-in is not yet deployed before the SSL certificate tool is used, you see these symptoms:

The option to enable the plug-in is missing in the web client.
In the /storage/log/vmware/vsan-health/vmware-vsan-health-service.log file, you see an entry similar to:

Failed to log into VC, retrying in 10 seconds
The vSphere Web Client screen for the vSAN Health Check plug-in displays the error:

Unexpected status code: 400
When running the vsan.health.cluster_status command, you see an error in RVC similar to:

Failed to get status of cluster Test_Cluster: Undefined namespace prefix: //soapenv:Body/*

Environment

VMware vSAN 6.0.x

Cause

When the certificates are changed, it sets incorrect group ownership permissions for the certificate files, preventing the vSAN Health Check user from being able to read them. This causes the vSAN Health Check plug-in to fail or become unavailable.

Resolution

This issue is resolved in VMware vSphere 6.0 Update 2, available at VMware downloads

To work around this issue if you do not want to upgrade, correct the group permissions of the certificate file:

Log in to the vSphere vCenter Server Appliance using SSH.
Run this command to enable access to the Bash shell:

shell.set --enabled true
Type shell and press Enter.
Correct the group permissions, by running these commands:

cd /etc/vmware-vpx/ssl
chgrp cis rui.* vcsoluser.*
chmod g+r rui.* vcsoluser.*
After performing the preceding steps, the /var/log/vmware/eam/eam.log file on the vCenter Server Appliance may report an error similar to:

[YYYY-MM-DD]T[HH:MM:SS] | ERROR | eam-0 | VcConnection.java | 179 | Failed to login to vCenter as extension. vCenter has probably not loaded the EAM extension.xml yet.: Cannot complete login due to an incorrect user name or password.

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Additional Information

To be alerted when this document is updated, click the Subscribe to Article link in the Actions box.
ESX Agent Manager solution user fails to log in after replacing the vCenter Server certificates in vSphere 6.0
vSAN 健全性チェックプラグインの「予期しないステータスコード：503 (Unexpected status code: 503)」エラー
vSAN 运行状况检查插件中显示 “异常状态代码: 503 (Unexpected status code: 503)” 错误