vCenter Server upgrade from 7.0 U2 to 7.0 U3c fails with error "Exception occured in postInstallHook"

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

While upgrading vCenter server from 7.0U2 to 7.0U3c, upgrade failure reported withe error "Exception occured in postInstallHook"
Patching reports successful, but some of the services failed to start due to timeout hence the upgrade marked as failed
/var/log/vmware/applmgmt/update_microservice.log shows similar entries:

2021-07-04 10:25:04,206 - 15535 -update_b2b::_runScriptHook: 354 -DEBUG - update script result file '{\n    "end_time":
2021-07-04T10:25:04.083Z",\n    "error": {\n "componentKey": "last_component:Patch",\n "detail": [\n "id": "ur.services.start.text",\n "localized": "Failed to start all services after successful patching.",\n "translatable": "Failed to start all services after successful patching."\n
2021-07-04 10:25:04,210 - 15535- task_manager::update:  80 - DEBUG - UpdateTask: status=RESUMABLE, progress=80, message={'id': 'com.vmware.appliance.postinstall_errors', 'default_message': 'Data conversion/Post install hook failed', 'args': []}
2021-07-04 10:25:04,249 - 15535 - dbfunctions::executeDML:  54 -    DEBUG - Executing Query {INSERT INTO progress VALUES(NULL, ?,         DATETIME('now'), ?, ?, ?)} with parameters ('failed', 'Installation failed', 100, 95)
2021-07-04 10:25:04,265 - 15535-update_b2b::postInstallHook:2345 -    DEBUG - Returning from postInstallHook(), data conversion failed
2021-07-04 10:25:04,265 - 15535 -update_b2b:: postInstallHook:2352 -    ERROR - Exception occured in postInstallHook
2021-07-04 10:25:04,266 - 15535 -task_manager:: update:  80 -    DEBUG - UpdateTask: status=RESUMABLE, progress=80, message={'id': 'com.vmware.appliance.plain_message', 'default_message': '%s', 'args': ['Exception occured in postInstallHook']}

startAllServices.err.log shows similar entries

2021-07-04T10:12:31.242Z ERROR service-control Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start imagebuilder, updatemgr, vsan-health services. Error: Operation timed out
2021-07-04T10:25:02.984Z ERROR service-control Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start imagebuilder, updatemgr, vsan-health services. Error: Operation timed out

/var/log/vmware/cloudvm/service-control.log shows similar entries

2021-07-04T10:25:01.726Z INFO service-control Stopped:
vmcam vmware-hvc vmware-imagebuilder vmware-pod vmware-vcha vmware-vdtc vmware-vsan-health
2021-07-04T10:25:02.984Z ERROR service-control Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start imagebuilder, updatemgr, vsan-health services. Error: Operation timed out

Note: The preceding log excerpts are only examples. Date, time and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server 7.0.2
VMware vCenter Server 7.0.3

Cause

The imagebuilder, updatemgr, vsan-health services failed to start due to timeout post vCenter server patching activity and hence the upgrade is marked failed.

Resolution

This issue is resolved in VMware vCenter Server release 7.0 U3f. To download go to Customer Connect .

Workaround:
To workaround this issue, please follow the below steps:

Take offline snapshot of vCenter Server (vCenter Server virtual machine powered off)
Connect to the VCSA command line using the root user via SSH.
Make sure all vCenter Server services and running using the below command.

#service-control --status --all
Note: Some services might be stopped by default.

Exception occured in postInstallHook

Use cd to go the the below directory:

#cd /etc/vmware/vmware-vmon/svcCfgfiles/

Backup all these 3 json files (imagebuilder.json, updatemgr.json and vsan-health.json) using cp command to any other directory.

Ex: # mkdir /root/backup -> To create a new directory "backup" under /root
# cp imagebuilder.json /root/backup/
# cp updatemgr.json /root/backup/
# cp vsan-health.json /root/backup/

Increase startup timeout ("StartTimeout") value of all these three services to 1500 seconds and save respective json files:

Use the vi command to start editing in the file(s). Ex: # vi imagebuilder.json
Look for the word StartTimeout
Press the i letter to change to insert mode.
Change the StartTimeout value to 1500

Press ESC key to exit insert mode.
Press :wq! to save and exit.

Note: Do the same procedures with (updatemgr.json and vsan-health.json)

Perform vCenter server upgrade.