How to safely clean /storage/var space in vRealize Log Insight

Products

VMware Aria Suite

Issue/Introduction

This article provides a safe procedure to reclaim disk space consumed by logs in vRealize Log Insight.

Resolution

In general - it's safe to delete all the log files, but in the scope of this document, only log files older than one day are deleted.

Log into the vRealize Log Insight node as root via SSH or Console, pressing ALT+F1 in a Console to log in.
Run the following command to truncate the logs:

find /storage/var/loginsight
-maxdepth 1 -mount -mtime +1 -type f -regex ".*[\._]log.+" -exec echo {} \; -exec rm {} \; 2>&1 | tee /tmp/deleted_logs.txt

Repeat steps 1-2 on all other affected nodes.

Note: To delete logs generated specifically by Apache Tomcat, execute the following command:

find apache-tomcat/logs -maxdepth 1 -mount -mtime +1 -type f -exec echo {} \; -exec rm {} \; 2>&1 | tee /tmp/deleted_tomcat_logs.txt

This command is similar to the command above with a slight deference. Apache Tomcat logs have format LogName.date.log and it's hard determine if it's an already rotated log file or the one currently in use. Thus, we are deleting all the files that has modification date greater than 1 day (the -mtime +1 option).

Additional Information

/storage/var/loginsight
- This is the data volume where all logs are kept.
-maxdepth 1
- Maximum number of nested directories to search through. (1 for only in current directory without going deeper)
-mount
- This prevents find from looking outside the log volume.
-type f
- This prevents find from looking at directories, links, or other special files.
-mtime +1
- This causes find to only look at files that have not been modified in the last 1 day. You can increase or remove this value as desired

-regex ".*[\._]log.+"
- The regular expression to search pattern of log files.

-exec echo {} \;
- This prints file paths that are discovered so that you can see affected files.
-exec rm {} \;
- This removes the file
2>&1 | tee /tmp/deleted_logs.txt
- This records all output of this command to a file at /tmp/deleted_logs.txt

Impact/Risks:

97% disk space usage on /storage/core is considered normal and healthy operation for vRealize Log Insight.
Logs in this document refer to the internal logs generated by vRealize Log Insight, and containing information about its execution and not the logs that are ingested and stored within
Logs generated by vRealize Log Insight have a rotation period by default and usually should not occupy too much space. The rotation period is set from 5-10 days for most of the logs, but others do not have strict time rules. These can be bound to disk space (e.g. keep maximum of 10 log files with maximum 500MB disk usage).
Running the following commands will delete all the logs generated by vRealize Log Insight, which might contain useful information about other issues that may be occurring, and present a challenge to VMware support if you open a request. It would be a good idea to copy the logs off first if you are concerned with losing useful data.

Important: It is recommended to take a snapshot of the vRealize Log Insight VM(s) before proceeding. Please remember to delete the snapshots within ~72 hours to prevent performance issues