Firstboot fails when upgrading vCenter Server Appliance 6.7 and vCenter Server 7.x due to time synchronization issue

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

The components of the vSphere environment are not time synchronized.
Firstboot fails during Install/Deployment, Upgrade or Migration.
In the firstbootStatus.json file, you may see one of these services have failed to configure or start during firstboot:
- "failedSteps": "cmfirstboot"
- "failedSteps": "analytics_firstboot"
- "failedSteps": "vpxd_firstboot"
- "failedSteps": "pschealth-firstboot”
- "failedSteps": "sms_spbm_firstboot"
- "failedSteps": "vmafd-firstboot"
- "failedSteps": "vapi_firstboot"
- "failedSteps": "mgmt-firstboot"
- "failedSteps": "scafirstboot"
- "failedSteps": "updatemgr-firstboot"
- "failedSteps": "ngc_firstboot"

In the cmfirstboot.py_####_stderr.log file, you see the error:

PAM: Authentication token is no longer valid

An error occurred while performing security operation: 'Failed to add user: cm to group: cis’

In the analytics_firstboot.py_####_stderr.log file, you see the error:

Analytics Service registration with Component Manager failed
ns0:MessageExpired
The time now (date + time) does not fall in the request lifetime interval extended with clock tolerance of 600000 ms [ (date + time); (date +time)]. This might be due to a clock skew problem.

In the vpxd_firstboot.py_####_stdout.log file, you see the error:

ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:720)

In the pschealth-firstboot.py_#####_stderr.log file, you see the error:

An error occurred while starting service 'pschealth'

In the sms_spbm_firstboot.py_####_stderr.log file, you see the error:

VMware vSphere Profile-Driven Storage Service failed to start

In the vmafdd-syslog.log file, you see the error:

Vmdir server is down.

In the vmdird-syslog.log file, you see the error:

DecodeEntry failed (9605) DN:()

LoadServerGlobals: (9700)()

In the vapi_firstboot.py_####_stderr.log file, you see the error:

Failed to configure vAPI Endpoint Service at the firstboot time

In the mgmt-firstboot.py_####_stderr.log file, you see the error:

UnboundLocalError: local variable 'e' referenced before assignment

In the scafirstboot.py_####_stderr.log file, you see the error:

[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:720)

In the updatemgr-firstboot.py_6012_stderr.log file, you see the error:

Failed to register updatemgr extension

In the ngc_firstboot.py_####_stderr.log file, you see the error:

SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:720)

In the %ProgramData%\VMware\vCenterServer\logs\cm\cm.log file, you see entries similar to:

Caused by: com.vmware.vim.vmomi.client.exception.VlsiCertificateException: Server certificate chain is not trusted and thumbprint verification is not configured
at com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager.checkServerTrusted(ThumbprintTrustManager.java:183)
at sun.security.ssl.AbstractTrustManagerWrapper.checkServerTrusted(SSLContextImpl.java:984)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1496)
... 78 more
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: timestamp check failed

In the fbInstall.json file shows an unexpected time discrepancy between the start time and end time. This could be a large jump forward or backward in time.

For example:

"start_time": "2018-05-07T13:00:00.000Z
"end_time": "2018-05-07T18:00:00.000Z"

or

"start_time": "2018-05-07T18:05:00.000Z
"end_time": "2018-05-07T18:00:00.000Z"

Environment

VMware vCenter Server Appliance 6.7.x
VMware vCenter Server 7.0.x

Cause

This issue occurs due to time inconsistencies in the vSphere environment. The issue most commonly happens when the target ESXi host for the destination vCenter Server Appliance is not synchronized with NTP. This issue can also happen if the destination vCenter Server Appliance migrates to an ESXi host with different time due to fully automated DRS.

Resolution

To avoid time synchronization issues, ensure the following is correct before deploying, migrating, or upgrading a vCenter Server Appliance:

The target ESXi host where the destination vCenter Server Appliance will be deployed is synchronized to NTP.
If the target ESXi host is part of a Fully Automated DRS cluster, change the automation level to Manual.
The ESXi host running the source vCenter Server Appliance is synchronized to NTP.
If the vCenter Server Appliance will be connected to an external Platform Services Controller, ensure the ESXi host running the external Platform Services Controller is synchronized to NTP.
Verify that the source vCenter Server or vCenter Server Appliance and external Platform Services Controller have the correct time.

For more information on:

Managing time in vSphere, see Synchronizing Clocks on the vSphere Network.
Changing DRS Automation, see Edit Cluster Settings.
vCenter Server Appliance requirements, see System Requirements for the vCenter Server Appliance and Platform Services Controller Appliance.

Additional Information

Note: Firstboot logs are located at:

vCenter Server Appliance - Firstboot logs are located in the /var/log/firstboot directory.
vCenter Server on Windows - Firstboot logs are located in the VMware-VCS-logs-.zip/vcs_logs/uninstall directory

or

VMware-VCS-logs-/vcs_logs/uninstall/.zip/ProgramData/VMware/vCenterServer/logs/firstboot directory

Note: In vSphere 7.0, vCenter Server for Windows has been removed and support is not available. For more information, see Farewell, vCenter Server for Windows .

To collect a log bundle or review log files:

For more information, see VMware vCenter Failed Firstboot.

-------------------------------------------------------------------------------------------------------------------------------------------------
For more information see: