Search the VMware Knowledge Base (KB)
View by Article ID

vCenter Appliance root Partition 100% full due to Audit.log files not being rotated (2149278)

  • 6 Ratings
Language Editions

Symptoms

  • 100% capacity used for /dev/sda3.
  • Size of audit.log file is very large and /var/log/audit folder consumes majority of the space.
  • Saved logs from log rotate policy reference a date that is not in line with the policy.
  • Unable to connect to the vCenter Server as services are not started.
  • Running /etc/cron.daily/logrotate manually rotates logs as expected.

Purpose

This article provides steps to reduce the audit.log size.

Resolution

To resolve this issue, remove the audit.log files and verify the cron job is working correctly:
  1. Log in to the vCenter Server Appliance through SSH.
  2. Run this command to enable access the Bash shell:

    shell.set --enabled true

  3. Type shell and press Enter.
  4. Navigate to the /var/log/audit folder with this command:

    cd /var/log/audit

  5. Run this command to verify the issue is with the audit.log files:

    ls -lh

    For example:

    ls -lh

    total 3.5G
    drwx------ 2 root root 4.0K May 5 2016 audispd
    -rw------- 1 root root 3.5G Feb 3 16:55 audit.log
    -rw------- 1 root root 445K Apr 8 2016 audit.log-20160408.bz2
    -rw------- 1 root root 447K Apr 9 2016 audit.log-20160409.bz2
    -rw------- 1 root root 444K Apr 10 2016 audit.log-20160410.bz2
    -rw------- 1 root root 447K Apr 11 2016 audit.log-20160411.bz2
    -rw------- 1 root root 445K Apr 12 2016 audit.log-20160412.bz2
    -rw------- 1 root root 446K Apr 13 2016 audit.log-20160413.bz2
    -rw------- 1 root root 447K Apr 14 2016 audit.log-20160414.bz2
    -rw------- 1 root root 446K Apr 15 2016 audit.log-20160415.bz2
    -rw------- 1 root root 447K Apr 16 2016 audit.log-20160416.bz2
    -rw------- 1 root root 447K Apr 17 2016 audit.log-20160417.bz2
    -rw------- 1 root root 445K Apr 18 2016 audit.log-20160418.bz2
    -rw------- 1 root root 446K Apr 19 2016 audit.log-20160419.bz2
    -rw------- 1 root root 446K Apr 20 2016 audit.log-20160420.bz2
    -rw------- 1 root root 445K Apr 21 2016 audit.log-20160421.bz2
    -rw------- 1 root root 449K Apr 22 2016 audit.log-20160422.bz2
    drwx------ 2 root root 4.0K Apr 23 2015 auditd


    In this example, the audit.log was rotated last time on 22nd Apr 2016.

  6. Remove the the audit.log file with this command:

    rm -rf audit.log

  7. Run this command to see when the cron job was last ran successfully:

    ls -l /var/spool/cron/lastrun/

    For example:

    ls -l /var/spool/cron/lastrun/

    total 0
    -rw------- 1 root root 0 Apr 22 2016 cron.daily
    -rw------- 1 root root 0 Apr 22 2016 cron.hourly
    -rw------- 1 root root 0 Apr 21 2016 cron.weekly

    In this example, the cron job last ran on Apr 22, this is the same as when the logs were last rotated.

  8. Run this command to check for credential failures running the cron job:

    grep "Authentication token is no longer valid; new one required" /var/log/messages.0.log | head

    For example:

    grep "Authentication token is no longer valid; new one required" /var/log/messages.0.log | head

    2016-11-07T00:20:01.617180+00:00 vcenter /usr/sbin/cron[18972]: Authentication token is no longer valid; new one required
    2016-11-07T00:20:01.617183+00:00 vcenter /usr/sbin/cron[18974]: Authentication token is no longer valid; new one required
    2016-11-07T00:25:01.619783+00:00 vcenter /usr/sbin/cron[31405]: Authentication token is no longer valid; new one required
    2016-11-07T00:30:01.622177+00:00 vcenter /usr/sbin/cron[9978]: Authentication token is no longer valid; new one required
    2016-11-07T00:30:01.622180+00:00 vcenter /usr/sbin/cron[9975]: Authentication token is no longer valid; new one required
    2016-11-07T00:30:01.622182+00:00 vcenter /usr/sbin/cron[9977]: Authentication token is no longer valid; new one required


    Run this command to check if the root password has expired:

    chage -l root

    For example:

    chage -l root

    Password change requested. Choose a new password.
    Old Password:
    New password:


  9. Change the root password as prompted.
  10. Verify the root account password has been changed:

    chage -l root

    For example:

    chage -l root

    Minimum: 0
    Maximum: 365
    Warning: 7
    Inactive: -1
    Last Change: Feb 03, 2017
    Password Expires: Feb 03, 2018
    Password Inactive: Never
    Account Expires: Never


  11. Restart all vCenter Server services.

    service-control --stop --all
    service-control --start --all

See Also

Language Editions

zh_cn,2150525

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 6 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.

What can we do to improve this information? (4000 or fewer characters)




Please enter the Captcha code before clicking Submit.
  • 6 Ratings
Actions
KB: