/storage/log partition full due to Large Java dump files created under /storage/log/vmware/perfcharts in vCenter Server Appliance
search cancel

/storage/log partition full due to Large Java dump files created under /storage/log/vmware/perfcharts in vCenter Server Appliance

book

Article ID: 318709

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:
  • You see that the /storage/log directory is full and in critical state the appliance manager page in VCSA 6.5.
     
  • You see that large Java dump files are created at /storage/log/vmware/perfcharts
     
  • You see that the alerts are triggered in the vCenter Server VAMI page
     
  • You notice that the /storage/log is almost full
     
    root@VCENTER [ ~ ]# df -h
    Filesystem                            Size  Used  Avail  Use%  Mounted on
    /dev/mapper/autodeploy_vg-autodeploy  9.8G   33M   9.2G    1%  /storage/autodeploy
    /dev/mapper/seat_vg-seat               25G  3.3G    20G   15%  /storage/seat
    /dev/mapper/db_vg-db                  9.8G  1.8G   7.5G   20%  /storage/db
    /dev/mapper/updatemgr_vg-updatemgr     99G  757M    93G    1%  /storage/updatemgr
    /dev/mapper/log_vg-log                 74G   67G   4.2G   95%  /storage/log
    /dev/mapper/core_vg-core               50G   52M    47G    1%  /storage/core
    /dev/sda1                             120M   28M    87M   24%  /boot
     
  • When you browse to /storage/log/vmware/perfcharts directory, you see lots of Java dump (.hprof) files.
     
    root@VCENTER [ /storage/log/vmware/perfcharts ]# ls -l
    total 67433848
    -rw-r--r-- 1 perfcharts users 102100    May 19 19:03 java_error23079.log
    -rw-r--r-- 1 perfcharts users 106892    Mar 29 23:25 java_error45346.log
    -rw------- 1 perfcharts users 772752973 May 18 04:57 java_pid1290.hprof
    -rw------- 1 perfcharts users 775950352 Apr 23 22:59 java_pid13403.hprof
    -rw------- 1 perfcharts users 776215829 Apr 14 08:18 java_pid14007.hprof
    -rw------- 1 perfcharts users 776821858 Apr 10 16:00 java_pid14105.hprof
    -rw------- 1 perfcharts users 776148983 Apr 24 17:43 java_pid14437.hprof
    -rw------- 1 perfcharts users 775075757 Apr 30 03:48 java_pid14502.hprof
    -rw------- 1 perfcharts users 777414024 Mar  7 21:55 java_pid1463.hprof
    -rw------- 1 perfcharts users 777816472 Mar  3 14:35 java_pid15043.hprof
    -rw------- 1 perfcharts users 773803956 May 14 20:16 java_pid16276.hprof
    -rw------- 1 perfcharts users 776556635 Apr 19 04:38 java_pid17986.hprof
    -rw------- 1 perfcharts users 774637444 May  6 03:29 java_pid18867.hprof
    -rw------- 1 perfcharts users 776309973 Apr 17 14:34 java_pid19069.hprof
     
  • Files are getting re-generated even after removing the java files by running the rm -rf /storage/log/vmware/perfcharts/java_pid*.hprof; command


Environment

VMware vCenter Server Appliance 6.5.x
VMware vCenter Server 6.5.x

Resolution

To resolve this issue upgrade to vCenter Server 6.5 U1d, available at VMware Download.

Note: Should the /storage/log partition be full but the symptoms in this KB don't match your issue, please see Troubleshooting disk space related issues: vCenter Server Appliance Maintenance Series (76563) for additional issues related to insufficient disk space. 


Workaround:
To workaround this issue:
  1. Log in to vCenter Server appliance using SSH or console and enter the Bash shell
  2. Take a backup of the /etc/vmware/service-layout.mfx file by running this command:
cp /etc/vmware/service-layout.mfx /etc/vmware/service-layout.mfx.bak;
  1. Stop the vmware-perfcharts service by running this command:
service-control --stop vmware-perfcharts;
  1. Open the service-layout.mfx file using a text editor by running this command:
vi service-layout.mfx;
  1. Change CompressClassSize size from 94 to 64 for the row vmware-perfcharts
  2. Save and close the service-layout.mfx file
  3. Start the vmware-perfcharts service by running this command:
service-control --start vmware-perfcharts;
  1. Now increase the allocated memory of the vmware-perfcharts service and restart it by running this command:
cloudvm-ram-size -C 1024 vmware-perfcharts;service-control --stop vmware-perfcharts;service-control --start vmware-perfcharts;


Additional Information

Troubleshooting vCenter Appliance /storage/log directory is 80% or more full

Powered by Wolken Software