"stats-monitor service" service not starting after VCSA reboot

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

stats-monitor service not starting after reboot of VCSA
Manual start of service is working
CPU, Memory, and Database does not have any graphs
Manual start of statsmonitor is working fine
In the StatsMonitor.log (latest-log) file, located in located in the directory /var/log/vmware/applmgmt, you see entries similar to

2017-11-07T12:20:38.485+01:00 info StatsMonitor[7F2A12D51700] [Originator@6876 sub=ThreadPool] Entering worker thread loop
2017-11-07T12:20:38.485+01:00 info StatsMonitor[7F2A12D10700] [Originator@6876 sub=ThreadPool] Thread enlisted
2017-11-07T12:20:38.485+01:00 info StatsMonitor[7F2A12D10700] [Originator@6876 sub=ThreadPool] Entering IO thread loop
2017-11-07T12:20:38.485+01:00 info StatsMonitor[7F2A12E15780] [Originator@6876 sub=ThreadPool] Thread pool fair initial threads spawned. IO: 2, Min workers: 4, Max workers: 13, Reservation ratio: 9
2017-11-07T12:20:38.485+01:00 info StatsMonitor[7F2A12E15780] [Originator@6876 sub=ThreadPool] Thread enlisted
2017-11-07T12:20:38.485+01:00 info StatsMonitor[7F2A12E15780] [Originator@6876 sub=Default] Syscommand enabled: true
2017-11-07T12:20:38.485+01:00 info StatsMonitor[7F2A12CCF700] [Originator@6876 sub=ThreadPool] Thread enlisted
2017-11-07T12:20:38.485+01:00 info StatsMonitor[7F2A12CCF700] [Originator@6876 sub=ThreadPool] Entering IO thread loop
2017-11-07T12:20:38.485+01:00 info StatsMonitor[7F2A12C8E700] [Originator@6876 sub=ThreadPool] Thread enlisted
2017-11-07T12:20:38.485+01:00 info StatsMonitor[7F2A12C8E700] [Originator@6876 sub=ThreadPool] Entering fair thread loop
2017-11-07T12:20:38.599+01:00 info StatsMonitor[7F2A12E15780] [Originator@6876 sub=Default] ReaperManager Initialized
2017-11-07T12:20:38.599+01:00 info StatsMonitor[7F2A12E15780] [Originator@6876 sub=StatsMonitor] Setting up signal handlers
2017-11-07T12:20:38.599+01:00 error StatsMonitor[7F2A12E15780] [Originator@6876 sub=StatsMonitor] Failed to register handler for signal: 0
2017-11-07T12:20:38.599+01:00 info StatsMonitor[7F2A12E15780] [Originator@6876 sub=StatsMonitor] Initializing
2017-11-07T12:20:38.600+01:00 info StatsMonitor[7F2A12E15780] [Originator@6876 sub=LinuxStatsProvider(738847377312)] Registered 162 stats (sources:114, derivatives:48)
2017-11-07T12:20:38.601+01:00 info StatsMonitor[7F2A12E15780] [Originator@6876 sub=StatsMonitor] Found 12 file systems and 7 dirs in config for monitoring.
2017-11-07T12:20:38.601+01:00 info StatsMonitor[7F2A12E15780] [Originator@6876 sub=LinuxStorageStatsProvider(738847702096)] Registered 3 stats
2017-11-07T12:20:38.601+01:00 info StatsMonitor[7F2A12E15780] [Originator@6876 sub=StatsMonitor] SqliteStorageEngine using SQLite version: 3.17.0
2017-11-07T12:24:35.215+01:00 info StatsMonitor[7F2A12E15780] [Originator@6876 sub=StatsMonitor] Received signal 15
2017-11-07T12:24:35.215+01:00 info StatsMonitor[7F2A12E15780] [Originator@6876 sub=StatsMonitor] Shutting down

Environment

VMware vCenter Server Appliance 6.5.x
VMware vCenter Server Appliance 6.7.x

Cause

This issue occurs due to start up timeout for StatsMonitor Service. The startup DB health check completed in a given time, occasionally, the IO might be too slow to start the service.

Resolution

This issue is resolved in vCenter Server 6.5 U3f and vCenter Server 6.7 U3

You can download newer versions from here if needed: Download VMware vSphere
You can download product patches from here if needed: VMware Product Patches

Workaround:

To workaround this issue, take a snapshot of VCSA without memory:

Take SSH to VCSA using root login.
Modify statsmonitor service config for vMon to set higher startup timeout by running this command:

sed -i '/StartTimeout/d' /etc/vmware/vmware-vmon/svcCfgfiles/statsmonitor.json
sed -i '/ApiHealthFile/a "StartTimeout": 600,' /etc/vmware/vmware-vmon/svcCfgfiles/statsmonitor.json

Run the kill -HUP $(cat /var/run/vmon.pid) command.
Stop and start statsmonitor service explicitly by running this command:

/usr/lib/vmware-vmon/vmon-cli -k statsmonitor

/usr/lib/vmware-vmon/vmon-cli -i statsmonitor

reboot the VCSA and check after 10-15 mins and the Statsmonitor Service should start up automatically.