"Exception occured in postInstallHook", Secure Token Service (vmware-stsd) crash while performing vCenter Server Patching to 7.0 U3f

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

vCenter Server patching to 7.0 U3f (Build 20051473) fails with error message "Exception occured in postInstallHook" in VAMI page.

Patching VCSA through CLI fails with error "An error occured while starting sts"
Upgrading vCenter Server from 6.x to 7.0 U3f will fail with "A problem occurred while - Starting VMware Security Token Service"

VCSA is joined to Active Directory domain and is currently using, or has used in the past, Integrated Windows Authentication for identity source
Log file /var/log/vmware/applmgmt/Patchrunner.log will show similar to below entries :

2022-07-15T15:50:31.439Z ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got unhandled exception.
Traceback (most recent call last):
File "/storage/seat/software-updatew2oofv0c/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 203, in patch
_patchComponents(ctx, userData, statusAggregator.reportingQueue)
File "/storage/seat/software-updatew2oofv0c/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 84, in _patchComponents
_startDependentServices(c)
File "/storage/seat/software-updatew2oofv0c/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 53, in _startDependentServices
serviceManager.start(depService)
super(VMwareServiceController, self).start(serviceName)
File "/storage/seat/software-updatew2oofv0c/stage/scripts/patches/libs/sdk/service_manager.py", line 665, in start
raise IllegalServiceOperation(errorText)
service_manager.IllegalServiceOperation: Service cannot be started. Error: Error executing start on service sts. Details {
"detail": [
{
"id": "install.ciscommon.service.failstart",
"translatable": "An error occurred while starting service '%(0)s'",
"args": [
"sts"
],
"localized": "An error occurred while starting service 'sts'"
}

STS runtime log file log/vmware/sso/sts-runtime.log.stderr will show below error message :

Starting service process with pid: 38715.
Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M -Dcom.sun.org.apache.xml.internal.security.ignoreLineBreaks=true -Dorg.apache.xml.security.ignoreLineBreaks=true
free(): corrupted unsorted chunks

Log file will show any of the memory corruption errors mentioned below :

free(): corrupted unsorted chunks

double free or corruption (!prev)

corrupted size vs. prev_size

free(): invalid next size (normal)

/var/core directory will show core dump files like below :

core.Thread-2.38715
core.Thread-2.54064

Environment

VMware vCenter Server 7.0.3

Cause

This issue is caused due to memory corruption in Secure Token Service (vmware-stsd) when VCSA is joined to Active Directory and is currently using, or has used in the past, Integrated Windows Authentication for identity source.

Resolution

This issue is resolved in VMware vCenter Server 7.0 U3g build 20150588, available at VMware Downloads.

Note:

vCenter Server needs to be reverted to a healthy state to apply this patch, applying a patch in already failed state is not feasible.
CVE-2021-22048 stands as un-resolved in 7.0 U3g as well, please refer to VMSA-2021-0025 for more information.

Workaround:
To workaround this issue, identity source configuration needs to be changed from IWA to "AD over LDAP" or "AD over LDAPs" and remove the VCSA from AD domain.

You may follow below steps to change the identity source configuration :

Revert/restore vCenter Server to a healthy state, which was taken before initiating the patch
Remove the IWA Identity Source configuration
Remove vCenter Server from the Active Directory domain.

/opt/likewise/bin/domainjoin-cli leave

Reboot the vCenter Server
Add the identity source as "AD over LDAP" or "AD over LDAPS"
Retry the Patching

Note:

This workaround cannot be applied in the Patching failed state, vCenter needs to be reverted to a healthy state, then apply the workaround.
You may move back to IWA (Integrated Windows Authentication) Identity Source after patching to VMware vCenter Server 7.0 U3g build 20150588
Detailed steps for each action is available in the documents mentioned in Related Information

Additional Information

Join or Leave an Active Directory Domain
Add or Edit a vCenter Single Sign-On Identity Source
Active Directory over LDAP and OpenLDAP Server Identity Source Settings
Configuring a vCenter Single Sign-On Identity Source using LDAP with SSL (LDAPS)