This article provides information on how to troubleshoot and resolve the /storage/log partition being full on vCenter Appliance.

Symptoms:

• vCenter Server Appliance (vCSA) reports errors similar to:
  • vSphere UI Health Alarm", "Log disk exhaustion on vcenter name
  • Database Health Alarm", "Core and Inventory Disk Exhaustion on vcenter name
• vCenter Server may be inaccessible with 503 Service Unavailable errors.
• vSphere Appliance Management Interface (VAMI) Monitor > Disk shows /storage/log at 75% full or more:
  • At 75% of continuous disk use the space will trigger Yellow status warnings
  • At 85% the space will trigger Red critical alarms
  • At 95% the vpxd service will shut down to prevent file and database corruption

VMware vCenter Server Appliance 6.0.x
VMware vCenter Server Appliance 6.7.x
VMware vCenter Server Appliance 6.5.x
VMware vCenter Server 8.0
VMware vCenter Server 7.0.x

Potential causes include:

• vCenter Server log bundles not being cleared after generation
• Very high frequency events filling up logs
• Services such as the Apache Tomcat Java Servlet service failing to clean up files
• /storage/log partition set too small

1. Verify whether one of these known issues is not causing the problem:

Affected Versions	Associated Knowledgebase Article link
6.0	/storage/log partition full due to cloudvm-ram-size.log file rotation is not working in vCenter Server Appliance
6.0 before Update 3, 6.5 before Update 1	/storage/log partition full due to SSO log files are not compressed in vCenter Server Appliance/storage/log partition full due to SSO log files are not compressed in vCenter Server Appliance
7.0 before Update 1C	vCenter Appliance /storage/log partition full due to excessive pod-startup.log files
7.0 before Update 3c	vmafdd.log is not being compressed which eventually leads to "log disk exhaustion" warning on the vCenter (83238) (vmware.com) vCenter Server /storage/log filling up due to localhost_access.log and catalina.log in sso and lookupsvc log directories (85475) (vmware.com)
7.0 before Update 3o, 8.0 before Update 1	/storage/log filling up with imfile-state files | rsyslogd (87669) (vmware.com)
7.0, 8.0, not resolved in any newer version	vCenter Server Appliance 7.0.x /storage/log partition runs out of space due to VMware Analytics service log file (analytics-runtime.log.stderr) (85468)
7.0 Update 1 through Update 2. Fixed in Update 3	vCenter has a large number of localhost_access log files generated under /storage/log/vmware/eam/web/ (85249)
7.0 Update 1 and newer releases. Fixed in 8.0.	vCenter has a large number of localhost_access log files generated under /storage/log/vmware/eam/web/ (85249)
7.0 Update 2. Fixed in Update 3.	The /storage/log volume is filling up in vCenter 7.0 U2 due to growing sps-runtime.log.stderr (83671) (vmware.com)
All versions	/storage/log partition full due to Large Java dump files created under /storage/log/vmware/perfcharts in vCenter Server Appliance /storage/log becomes 100% full when exporting the vCenter Appliance log bundles vCenter Appliance /storage/log partition full due to excessive pod-startup.log files VCSA /storage/log/ partition runs out of space due to java hprof dumps created by VMware Analytics service

2. If the known issues don't identify the problem, look for excessive Tomcat log files in the partition.
   1. Connect to the vCenter Server Appliance through SSH or through the vCenter VM console
   2. Type shell to switch to the BASH shell

3. U​​​​​​se the following command to list the Apache Tomcat service log files under the below paths:

4. If there are a lot of these you may delete them to free up space

5. Check whether space has significantly improved or not. If it has, the drive might be too small for the log traffic. You might increase the size of the /storage/log virtual disk per vCenter Server Appliance disk space is full

3. Check for and remove excessive PostgreSQL service health alarm log files
   1. In  /storage/log/vmware/sso/tomcat/, /storage/log/vmware/eam/web/ & /storage/log/vmware/lookupsvc/tomcat/ (vCenter 7.0 only) directories, run the commands:

2. Check again whether space has significantly improved or not. If it has, the drive might be too small for the log traffic. You might increase the size of the /storage/log virtual disk per vCenter Server Appliance disk space is full

4. Check for and remove excessive Storage Profile Service access logs

5. If the previous steps did not Identify the problem, follow vCenter Server Appliance disk space is full in the "Further troubleshooting" section. 

vCenter Server Appliance disk space is full
vCenter Server /storage/log filling up due to localhost_access.log and catalina.log in sso and lookupsvc log directories

Impact/Risks:
Under default settings, when the partition reaches 75% full of continuous disk use for at least 10 minutes:

• You will see alarms being triggered.

When the partition reaches 95% full at any point

• The operating system will prevent the main vmware-vpxd service from starting to try to prevent file corruption.

Please keep in mind that this does not require it to be at 95% for a continuous period so the threshold might be crossed, shut down vpxd service, and then the use may recede,
Deleting critical files may prevent the vCenter Server Appliance from working.
Resizing vCenter Appliance virtual disks carries the risk of data corruption.
 
WARNING: Ensure you have a good backup of vCenter Appliance before deleting files or resizing disks.