vSAN Health service fails to start after performing vCenter upgrade

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

vSAN Health service fails to start in a VCHA configuration after performing a vCenter upgrade.
In /var/log/vmware/vsan-health/vsanvp.log on vCenter Server you can see errors similar to:

2020-05-07T12:18:31.108Z INFO vsanvp[thread-140087258990336] [ServiceInstance::RegisterVsanVP opID=vsan-PC-5a50db754a83e-W446]
Register VSAN VP to SMS service.
2020-05-07T12:18:36.256Z ERROR vsanvp[thread-140087258990336] [ServiceInstance::RegisterVsanVP opID=vsan-PC-5a50db754a83e-W446]
Failed to register VSAN VP to sms service
Traceback (most recent call last):
File "/usr/lib/vmware-vpx/vsan-health/pyMoVsan/ServiceInstance.py", line 81, in RegisterVsanVP
raise info.error
VsanVmomiPatcher.sms.fault.ProviderRegistrationFault: (sms.fault.ProviderRegistrationFault) {
msg = 'Validation of VMOMI server version got from provider failedCould not find version class for: null',
faultCause = <unset>,
faultMessage = (vmodl.LocalizableMessage) []
}

In /var/log/vmware/vsan-health/vmware-vsan-health-service.log on vCenter Server you see errors similar to:

2020-05-07T12:18:31.108Z CRITICAL vsan-mgmt[MainThread] [VsanMgmtServer::UncaughtExcpetionHandler opID=noOpId] Traceback (most recent call last):
File "/usr/lib/vmware-vpx/vsan-health/VsanMgmtServer.py", line 436, in <module>
_VsanMgmtServer.Start()
pyVmomi.VmomiSupport.vmodl.RuntimeFault: (vmodl.RuntimeFault) {
msg = 'N7Vmacore23FileIONotFoundExceptionE: Could not find file : /etc/vmware-vsan-health/.cns_pgpass',
faultCause = <unset>,
faultMessage = (vmodl.LocalizableMessage) []
}

One may observe the following error when trying to add a VM Disk:

"PBM error occurred during PreReconfigureCallback: Fault cause: vmodl.fault.SystemError"

Environment

VMware vCenter Server 6.7.x

Cause

The issue observed on the SPBM side is occurring due to vSAN VP inability to register itself with the SMS, service since vmware-vsan-health service is failing to start due to missing /etc/vmware-vsan-health/.cns_pgpass file.

Resolution

This is a known issue affecting vCenter upgrade - e.g., when upgrading from any 6.7 to 6.7 U3 or when upgrading from 6.7 U3 to any 6.7 U3 patch release (e.g. 6.7 U3g).

Fix due in a future release, subscribe to this KB to be updated when the fixed in version is released.

Workaround:
To work around this issue, you must regenerate the missing file via the following procedure:

Login via SSH to the vCenter Appliance which failed to start vsan-health service after upgrade to 6.7 U3.
Run the following command to re-generate the missing file .cns_pgpass:

# PYTHONPATH=/usr/lib/vmware-vpx/pyJack/:/usr/lib/vmware/site-packages/ python /usr/lib/vmware-vpx/firstboot/CnsVcConfigure.py

You should see the following output in the terminal:

2020-05-07T12:18:31.108Z Configuring DB for CNS service
2020-05-07T12:18:41.108Z Getting value for install-parameter: vpostgres.int.server_port
2020-05-07T12:18:51.108Z CNS DB configured successfully.

Start vsan-health service by running command:

# service-control --start vmware-vsan-health

Verify the vsan-health service has started.

# service-control --status vsan-health

6. If step 4 doesn't work then reboot vCenter