"Panic: NOT_REACHED vum/sysimage/integrity/lib/vcidb/baseline.cpp", Update Manager Service Crashes while upgrading VMware Cloud Foundation (VCF) to version 4.4

Products

VMware Cloud Foundation VMware vCenter Server

Issue/Introduction

Symptoms:

VMware Update Manager service crashes during ESXi upgrade workflow
LCM log file /var/log/vmware/vcf/lcm/lcm-debug.log on SDDC Manager will show errors similar to :

2022-02-25T12:10:25.405+0000 ERROR [vcf_lcm,71b5754e5ea347bb,2d34,precheckId=fea7bf63-6577-499f-b2d1-51f8f569b1e8,resourceType=BUNDLE,resourceId=d214e445-8509-4d50-adac-59d56acd86ae] [c.v.e.s.l.p.i
.e.EsxVumUpdateStageRunnerImpl,Precheck-83] Error while checking existing baselines
com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 503
        at com.vmware.vim.vmomi.client.common.Response$Status.getStatus(Response.java:58)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:150)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:48)
        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226)
        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:109)

2022-02-25T12:10:25.410+0000 ERROR [vcf_lcm,71b5754e5ea347bb,2d34,precheckId=fea7bf63-6577-499f-b2d1-51f8f569b1e8,resourceType=BUNDLE,resourceId=d214e445-8509-4d50-adac-59d56acd86ae] [c.v.e.s.l.p.i
.e.EsxVumUpdateStageRunnerImpl,Precheck-83] Failure when creating baseline: {}
com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 503
        at com.vmware.vim.vmomi.client.common.Response$Status.getStatus(Response.java:58)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:150)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:48)
        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226)

Status of Update Manager service shows as Stopped on vCenter Server

service-control --status updatemgr

Update Manager logs on vCenter Server shows similar to below stack trace

/var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log

[#1] 2022-02-25T12:17:52.809Z verbose vmware-vum-server[12929] [Originator@6876 sub=BaselineMgr] [baselineMgr 1244] Got spec for baseline id: 11, name: VMware ESXi 7.0 U3c (build 19193900)
[#1] 2022-02-25T12:17:52.816Z panic vmware-vum-server[12929] [Originator@6876 sub=Default]
[#1] -->
[#1] --> Panic: NOT_REACHED vum/sysimage/integrity/lib/vcidb/baseline.cpp:406
[#1] -->
[#1] --> Backtrace:
[#1] --> [backtrace begin] product: VMware Update Manager, version: 7.0.3, build: build-19234571, tag: vmware-vum-server, cpu: x86_64, os: linux, buildType: release
[#1] --> backtrace[00] libvmacore.so[0x0037E963]
[#1] --> backtrace[01] libvmacore.so[0x002C8917]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
[#1] --> backtrace[02] libvmacore.so[0x002D7AB7]: Vmacore::System::SystemFactory::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
[#1] --> backtrace[03] libvmacore.so[0x00371AAF]
[#1] --> backtrace[04] libvmacore.so[0x00371BCB]: Vmacore::PanicExit(char const*)
[#1] --> backtrace[05] libvmacore.so[0x00337031]
[#1] --> backtrace[06] libvmacore.so[0x003370CF]
[#1] --> backtrace[07] libvci-vcIntegrity.so[0x0068111D]: Integrity::Vcidb::Baseline::GetUpgradeRelease(VimVdb::VdbConnection&, int&)
[#1] --> backtrace[08] libvci-vcIntegrity.so[0x00681194]: Integrity::Vcidb::Baseline::GetUpgradeVersion(VimVdb::VdbConnection&, std::string&)
[#1] --> backtrace[09] libvci-vcIntegrity.so[0x00427EF9]: Integrity::BaselineMgrImpl::GetOneBaselineInfo(VimVdb::VdbWriteConnection&, Integrity::Vcidb::Baseline&, bool, Vmacore::Ref<Integrity::BaselineInfo>&)
[#1] --> backtrace[10] libvci-vcIntegrity.so[0x00428648]: Integrity::BaselineMgrImpl::GetBaseline(Vmacore::Ref<Vmomi::DataArray<Integrity::BaselineInfo> >&)
[#1] --> backtrace[11] libintegrity-types.so[0x001CEB1C]
[#1] --> backtrace[12] libvmomi.so[0x0013F1AE]
[#1] --> backtrace[13] libvmomi.so[0x0013F80B]
[#1] --> backtrace[14] libvmacore.so[0x00235A97]
[#1] --> backtrace[15] libvmacore.so[0x0023B03F]
[#1] --> backtrace[16] libvmacore.so[0x00377398]
[#1] --> backtrace[17] libpthread.so.0[0x00007F87]
[#1] --> backtrace[18] libc.so.6[0x000F35EF]
[#1] --> backtrace[19] (no module)
[#1] --> [backtrace end]

Multiple core dumps of worker (core.worker.<pid)> process will be created under /var/core/ directory on vCenter Server

Environment

VMware vCenter Server 7.0.3
VMware Cloud Foundation 4.4

Cause

This issue is due to an inconsistency in the VMware Update Manager database after VCF deletes the ESXi patch (iso) and baseline used to perform the ESXi update.

Resolution

For VCF 4.4 a second config drift bundle is released. The below manual steps can be skipped once that second config drift bundle is applied.

Bundle details:

Bundle ID	b27c6275-742c-4d51-a907-eba0e158ede3
Product Version	4.4.0.0
Kind of Bundle	Patch Bundle
Type	SDDC Manager
ID	56535
Bundle Version	4.12.4-155987

Workaround:
Please follow below steps to workaround the issue :

NOTE: The below instructions can be run on the SDDC Manager before the issue has been encountered to prevent it in the future, OR after the issue has been encountered to recover from it and prevent it in the future.

Download the Zip file "toggle-vsan-recommendation.zip" attached to this KB
Copy the downloaded zip file to the SDDC Manager /tmp folder by using a secure file copy utility (Eg. WinSCP). Ensure to use the vcf account to authenticate and upload the file
SSH to the SDDC Manager using the vcf account
Extract the zip file using unzip command:

cd /tmp
unzip toggle-vsan-recommendation.zip

This will extract two Python scripts : ToggleVsanRecommendation.py and ToggleVsanRecommendationWrapper.py

Execute the script "ToggleVsanRecommendationWrapper.py", substituting your API username and password as necessary:

python ToggleVsanRecommendationWrapper.py <api user> <api password>

Example:
python ToggleVsanRecommendationWrapper.py [email protected] Password0fAdmin!

The script should complete in less than 2 minutes (about 10 seconds per vCenter in inventory)
You should see output that looks similar to the following:

2022-03-09 16:41:56 INFO Fetching vCenters and credentials
2022-03-09 16:42:04 INFO ====== Toggling vSAN recommendation for all vSAN clusters in vCenter vcenter-1.vrack.vsphere.local ======
2022-03-09 16:42:04 INFO Enabling bash shell on vCenter
2022-03-09 16:42:09 INFO Creating temporary directory on vCenter: /tmp/togglevsanrec
2022-03-09 16:42:12 INFO Copying ToggleVsanRecommendation.py to vCenter
2022-03-09 16:42:14 INFO Executing /tmp/togglevsanrec/ToggleVsanRecommendation.py on vCenter
2022-03-09 16:42:18 INFO Output from vCenter:
======
2022-03-09 21:42:16 INFO Cleaning vSAN VUM baselines and restarting VUM (if necessary)
2022-03-09 21:42:17 INFO Setting vSAN baseline preference for SDDC-Cluster1 to noRecommendation
2022-03-09 21:42:18 INFO baselinePreferenceType=noRecommendation
======
2022-03-09 16:42:18 INFO Restoring appliance shell on vCenter

Attachments

toggle-vsan-recommendation get_app