"Authentication token is no longer valid new one required","Failed to start cm services" VCSA 6.5 update failed

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

vSphere 6.5 Patch Update fails with Install Failed error message in VAMI Page
In the /var/log/vmware/applmgmt/software-packages.log log file, you see entry similar to:

Executing /lib/systemd/systemd-sysv-install enable vami-lighttp
Created symlink from /etc/systemd/system/multi-user.target.wants/vami-lighttp.service to /usr/lib/systemd/system/vami-lighttp.service.

You are required to change your password immediately (password aged)
useradd: PAM: Authentication token is no longer valid; new one required
You are required to change your password immediately (password aged)
usermod.bk: PAM: Authentication token is no longer valid; new one required

Traceback (most recent call last):
File "/usr/sbin/usermod", line 19, in <module>
errcode = subprocess.check_call(sys.argv)
File "/opt/vmware/lib/python2.7/subprocess.py", line 186, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/sbin/usermod.bk', '-a', '-G', 'cis', 'cm']' returned non-zero exit status 1

error: %pre(vmware-cm-6.5.0-9451637.x86_64) scriptlet failed, exit status 1
error: vmware-cm-6.5.0-9451637.x86_64: install failed
error=Service-control failed. Error Failed to start vmon services.vmon-cli RC=1, stderr=Failed to start cm services. Error: Operation timed out
rc=1
[21889]CRITICAL:root:Installation process failed
[21889]DEBUG:root:Command Failed

Exception:
Traceback (most recent call last):
  File "/usr/lib/applmgmt/support/scripts/software-packages.py", line 471, in install
    answers=answer)
  File "/usr/lib/applmgmt/base/py/vmware/vherd/base/software_update.py", line 1949, in installPackagesEP
    logger.error("Package update failed")
  File "/usr/lib/applmgmt/base/py/vmware/vherd/base/task_manager.py", line 160, in trackedfunc
    value = func(*args, **kwargs)
  File "/usr/lib/applmgmt/base/py/vmware/vherd/base/software_update.py", line 1013, in serviceAction
    progress = True, message = "%s all services " % messageDict[action]))
  File "/usr/lib/applmgmt/base/py/vmware/vherd/base/software_update.py", line 814, in _processOutput
    raise GenericException("Command Failed")
GenericException: Command Failed

In the /var/log/messages log file, you see entry similar to:

useradd[24785]: pam_unix(useradd:account): expired password for user root (password aged)
useradd[24785]: Authentication token is no longer valid; new one required
useradd[24785]: failed adding user 'cm', data deleted
usermod[24788]: pam_unix(usermod:account): expired password for user root (password aged)
usermod[24788]: Authentication token is no longer valid; new one required

vCenter Services fail to start after the Patch attempt.
Component Manager (cm) service fails to start after the Patch attempt
Starting services using service-control --start --all fails at vmware-cm service
In the /var/log/vmware/cm.log log file, you see entry similar to:

2018-07-26T20:16:10.457Z [main [] ERROR com.vmware.cis.services.cm.service.Main ()] start: Component Manager failed to start
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'httpServer' defined in class path resource [http.xml]: Cannot resolve reference to bean 'vapiService' while setting bean property 'services' with key [1]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'vapiService' defined in class path resource [http.xml]: Initialization of bean failed; nested exception is java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/JsonNode

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server Appliance 6.5.x
VMware vCenter Server 6.5.x

Cause

This issue occurs due to expired root password in the vCenter Server Appliance

Resolution

To resolve this issue, reset the root password:

Run this command to check if the root password has expired:

chage -l root

For example:

chage -l root
You are required to change your password immediately (root enforced)
chage: PAM: Authentication token is no longer valid; new one required

Change the root password using passwd command
Verify the root account password has been changed:

chage -l root

Minimum: 0
Maximum: 365
Warning: 7
Inactive: -1
Last Change: Feb 03, 2017
Password Expires: Feb 03, 2018
Password Inactive: Never
Account Expires: Never

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Start the VAMI services

systemctl restart vami-lighttp

Attempt to patch the vCenter Server Appliance again using VAMI or using the below command line:

software-packages install --url --acceptEulas

If Patch Update continue to fail, you may need to :

Restore the vCenter Server Appliance to a snapshot state before attempting Patch Update or Restore from Backup
Perform step 1 to 3 before attempting Patch Update

Additional Information

How to reset the lost or forgotten root password in vCenter Server Appliance 6.5