"Insufficient free space for the Database" error causes vpxd services to shut down during upgrade failure or report VC Database Health Alarm

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

Database Health Alarm changes from green to yellow or red
In the vpxd.log file, you see that available free space on database disk has crossed the error threshold of 95%:

2017-06-17T00:57:17.080+05:30 info vpxd[01664] [Originator@6876 sub=HostAccess opID=PollQuickStatsLoop-3f3aa0c5] Using vpxapi.version.version10 to communicate with vpxa at host <HOSTNAME>
2017-06-17T00:57:17.310+05:30 warning vpxd[02232] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3
2017-06-17T00:57:17.470+05:30 info vpxd[11388] [Originator@6876 sub=vpxdVdb] WarningThreshold: 80% ErrorThreshold: 95%.
2017-06-17T00:57:17.470+05:30 error vpxd[11388] [Originator@6876 sub=vpxdVdb] Insufficient free space for the Database (used: 96%; threshold: 95%)

vCenter Server service shuts down
vCenter upgrade fails during the first boot of any service such as Auto deploy, EAM, vSan Health, etc.
In the /var/log/firstboot/autodeploy-firstboot.py_stderr.log file, you see the entries similar to:

2017-06-16T14:17:55.539Z RC = 1229 Stdout =
Stderr = error: cannot connect to vCenter -- 503 Service Unavailable (Error connecting tunnel on TCP socket: (null))
2017-06-16 19:47:55,539 Auto Deploy register Exception {{
"resolution": null,
"detail": [
{
"args": [
"Stderr: error: cannot connect to vCenter -- 503 Service Unavailable (Error connecting tunnel on TCP socket: (null))\r\n"
],
"id": "install.ciscommon.command.errinvoke",
"localized": "An error occurred while invoking external command : 'Stderr: error: cannot connect to vCenter -- 503 Service Unavailable (Error connecting tunnel on TCP socket: (null))\r\n'",
"translatable": "An error occurred while invoking external command : '%(0)s'"

Running journalctl -b shows messages similar to this:

Event [30816383] [1-1] [2021-01-04T17:32:06.395769Z] [vim.event.AlarmScriptCompleteEvent] [info] [] [] [30816383] [Alarm 'Database Health Alarm - Solution' on Datacenters ran script /bin/service-control --ignore --stop vmware-vpxd]

Environment

VMware vCenter Server 6.5.x

Cause

This issue usually occurs due to predefined vCenter Server Database Health Alarm, which operates at the vCenter Server level.

Note: It is also possible that a custom alarm has been created with the database size triggers. Monitoring solutions have been know to create these alarms automatically.

Every 15 minutes, vCenter Server checks how much free space a database has and triggers an alarm in accordance with pre-configured thresholds.

Alarm Name	Warning Condition	Critical Condition	Actions
Database Health Alarm	Database is 80% full	Database is 95% full	vCenter shuts down

A warning is issued when the database is 80% full. vCenter Server shuts down when the database is 95% full.

Resolution

To resolve this issue:

Increase the storage space for the database.
Delete unnecessary historical data from the database and shrink database to reduce the DB file size.

Note:

If you want to restart vCenter without fixing the problem, you must override the startup check.
If you restart vCenter Server without fixing the problem, any corrective action afterwards might cause the database to go out of space.

Workaround:

To workaround this issue:

Reduce the error threshold percentage by executing this command against the database. For more information, see Interacting with the vCenter Server Appliance 6.5 embedded vPostgres Database (2147285).

update vpx_parameter set value=99 where name='vpxd.vdb.space.errorPercent';

Note: If VC upgrade fails due to this issue, update the threshold in DB during the upgrade process at VirtualCenter Server firstboot.

Change the default settings for the Database Health Alarm through the Advanced Settings in vCenter Server:

In vSphere Web Client, navigate to vCenter Server.
In the Configure tab, select Advanced Settings and click Edit.

Use the Filter drop-down menu to find the configuration parameters that are related to the Database Health Alarm and set their new values.

Configuration parameter	Description	Default value
vpxd.vdb.space.warningPercent	Sets the percentage at which a warning is triggered. Changing this setting does not require you to restart vCenter Server.	80%
vpxd.vdb.space.errorPercent	Sets the percentage at which an error is triggered, and an action takes place. Changing this setting does not require you to restart vCenter Server.	95%
config.vpxd.vdb.space.skipInitialCheck	Turns off the startup checking for free space	false
config.vpxd.vdb.space.checkIntervalSec	Defines the interval at which vCenter Server checks a database for available space.	15 minutes

If you cannot find a configuration parameter, you must add it to the list:
1. In the Edit Advanced vCenter Server Settings dialog box, enter the name of the configuration parameter in the Name text box.
2. In the Value text box, set the value for the configuration parameter and click Add.
3. Click OK.