After updating vCSA to 6.7 U2 or higher, unable to log into the VAMI page- "Invalid Credentials" or "Unable to Login"

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

Unable to login with root credentials after updating vCSA 6.7 with embedded PSC to vCSA 6.7 U2 or higher using the VAMI option.
Logging into the VAMI page with root credentials prompts "Invalid Credentials" or "Unable to Login" message.
Login to the vCSA with virtual machine console works using same root credentials.

Status of services:

root@nsi-vsphere55 [ ~ ]# service-control --status
Stopped:
applmgmt vmcam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-rbd-watchdog vmware-vcha vsan-dps
Running:
lwsmd pschealth vmafdd vmcad vmdird vmdnsd vmonapi vmware-analytics vmware-certificatemanagement vmware-cis-license vmware-cm vmware-content-library vmware-eam vmware-perfcharts vmware-pod vmware-postgres-archiver vmware-rhttpproxy vmwa
re-sca vmware-sps vmware-sts-idmd vmware-stsd vmware-topologysvc vmware-updatemgr vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-client vsphere-ui vmware-statsmonitor

Environment

VMware vCenter Server Appliance 6.7.x

Cause

This happens because the vmware-statsmonitor service takes a long time to start upon the vCSA 6.7 U2+ reboot.
In this delayed duration, the applmgmt service gets marked as stopped.
This causes the failed logins to the VAMI page and a simple start of applmgmt service fixes it, but the behavior happens on every reboot.

Resolution

The solve the issue change the timeout for the applmgmt service so that it is started after the statsmonitor service has successfully started with its delay.

To do this, follow the below steps given below:

Go to the below directory:

cd /etc/vmware/vmware-vmon/svcCfgfiles/

Backup the applmgmt and statsmonitor json files:

mkdir /root/backup
cp /etc/vmware/vmware-vmon/svcCfgfiles/applmgmt.json /root/backup/
cp /etc/vmware/vmware-vmon/svcCfgfiles/statsmonitor.json /root/backup/

Change permissions on the existing json files:

chmod 700 applmgmt.json
chmod 700 statsmonitor.json

Manually increase the timeout in applmgmt.json file to 600 seconds or 1200 seconds (using VI editor). For the below example, 600 seconds has been used.

The timeout in the this file is 60 seconds by default.

/etc/vmware/vmware-vmon/svcCfgfiles/applmgmt.json:

{
"Name" : "applmgmt",
"PreStartCommand" : "/usr/lib/applmgmt/support/scripts/prestart-applmgmt.sh",
"StartCommand" : "/usr/lib/applmgmt/applmgmt.launcher",
"ApiHealthCommand": "/usr/bin/python",
"ApiHealthCommandArgs": [
"/usr/lib/applmgmt/applmgmt_vmonhealth.py"
],
"DependsOn" : ["statsmonitor"],
"StartTimeout" : 600,
"StopTimeout" : 20,
"StartupType" : "AUTOMATIC",
"DumpLiveCoreOnApiHealthFail" : false,
"StreamRedirectFile": "%VMWARE_LOG_DIR%/vmware/applmgmt/applmgmt_vmonsvc",
"RecoveryActionProfiles" :
{
"DEFAULT" :
{
"CRASH" : ["RESTART_SERVICE", "RESTART_SERVICE", "NO_ACTION"],
"HEALTHFAIL" : ["RESTART_SERVICE", "RESTART_SERVICE", "NO_ACTION"]
},
"FAILOVER" :
{
"CRASH" : ["RESTART_SERVICE", "RESTART_SERVICE", "NO_ACTION"],
"HEALTHFAIL" : ["RESTART_SERVICE", "RESTART_SERVICE", "NO_ACTION"]
}
}
}

Modify statsmonitor service config for vMon to set higher startup timeout:

sed -i '/StartTimeout/d' /etc/vmware/vmware-vmon/svcCfgfiles/statsmonitor.json
sed -i '/ApiHealthFile/a "StartTimeout": 600,' /etc/vmware/vmware-vmon/svcCfgfiles/statsmonitor.json

This adds a StartTimeout": 600 to the statsmonitor.json.

Stop and start statsmonitor service explicitly:

/usr/lib/vmware-vmon/vmon-cli -k statsmonitor
/usr/lib/vmware-vmon/vmon-cli -i statsmonitor

Reboot the vCSA VM.

Note: Upon reboot, all the services should be up and running in between 5-10 minutes.

After this, login to the VAMI page with root credentials should be successful.

Change permissions on the existing json files to the original permissions

chmod 444 applmgmt.json
chmod 444 statsmonitor.json

Workaround:
Run the below command to workaround this issue. However, the issue will re-occur upon a reboot.

service-control --start applmgmt

Additional Information

VMware Skyline Health Diagnostics for vSphere - FAQ