VMware
 

Knowledge Base

Search the Knowledge Base:
Products:
Search In:
 

Investigating disk space on an ESX host

Symptoms

  • Virtual machine fails to power on with following error message:
    Could not power on VM: No space left on device. Failed to power on VM  
  • The management agent (hostd) cannot start because the root partition is full
  • The VirtualCenter agent (vpxa) cannot start because the root partition is full
  • The system cannot create any new files or directories on  / or /tmp partition, with the error:

    no space left on device

  • The /tmp directory is filled with cimclient_root* logs, potentially thousands (~600,000 or more)
  • The vpxa log may contain these error:

    [2008-10-13 11:02:05.423 'Libs' 3076454304 warning] Cannot make directory /var/run/vmware/root/27591: No space left on device
    [2008-10-13 11:02:05.423 'App' 3076454304 error] Exception: Failed to initialize authd server
    [2008-10-13 11:02:05.423 'App' 3076454304 error] Backtrace:
    [00] eip 0x909dd92
    [01] eip 0x9043444
    [02] eip 0x907f975
    [03] eip 0x908024c
    [04] eip 0x9033d74

  • VMware VMotion failure at 10%
  • VMware High Availability (HA) configuration produces the error:

    Configuration of host IP address is inconsistent on host <hostname>: address resolved to <IP Address> and <IP Address>

Purpose

For troubleshooting purposes, it may be necessary to check the available free disk space on your ESX host. This provides you with the steps to check the available disk space and steps to free up space if required.

Resolution

Checking disk space usage on the ESX Server service console partitions

To check the free space on an ESX Server service console partitions:
  1. Log in to the ESX Server service console as root from either an SSH session or directly from the console of the server.
  2. Type df -h .
  3. When you have finished reviewing the output, type logout and press Enter to exit the system.
The output appears similar to:
 
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2             4.9G  3.0G  1.6G  66% /
/dev/sda1              99M   18M   77M  19% /boot
none                  145M     0  145M   0% /dev/shm
/dev/sda7             2.0G  135M  1.7G   8% /var/log
[root@server]#
 
Note: The partitions shown are from a default installation of an ESX host. If you have modified the partition configuration the output may show differently.
 
Review the Use% for each of the listed items. If any of the volumes listed are 100% full they must be investigated to determine if space can be freed. The most important mount points to investigate on a default installation of ESX are the / and /var/log mounts because if they are full they can prevent proper operation of the ESX host .  

Checking disk space usage on a VMFS volume of an ESX Server

 
To check the Free space on a VMFS volume of an ESX Server:
  1. Log in to the ESX Server service console as root from either an SSH session or directly from the console of the server.
  2. Type vdf -h .
  3. When you have finished reviewing the output, type logout and press Enter to exit the system.
The output appears similar to:
 
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2             4.9G  3.0G  1.6G  66% /
/dev/sda1              99M   18M   77M  19% /boot
none                  145M     0  145M   0% /dev/shm
/dev/sda7             2.0G  135M  1.7G   8% /var/log
/vmfs/devices         439G     0  439G   0% /vmfs/devices
/vmfs/volumes/458865ba-b31110fd-43d5-00127994e616
                       68G   47G   20G  69% /vmfs/volumes/San_Storage
/vmfs/volumes/45b5eb1a-808343db-ecab-00114335854b
                       26G  9.7G   16G  36% /vmfs/volumes/Local_Storage
[root@server]#
 
Note: The partitions shown are dependant on the VMFS volumes you have defined and presented to the ESX host.
 
Review the Use% for each of the listed items. If any of the volumes listed are 100% full they must be investigated to determine if space can be freed. If a VMFS volume is full you cannot create any new virtual machines and any virtual machines that are using snapshots may fail.
 
Another useful command is du -h --max-depth=1 <dir>
This command lists the directories within a given filesystem that contain the largest files. By starting at the root (/) directory and finding the largest directories you can then drill down into these directories (using cd) and execute the same command recursively until you find the files themselves which are occupying space.
 

Freeing disk space on an ESX Server

Freeing disk space can become a large task if there has not been any recent changes to the ESX host. There are a few common reasons that partitions become full:
  • Information, such as a virtual machine or an ISO, was copied to the system.
  • Snapshots are being used on virtual machines.
  • Log files have not yet been rotated after a substantial amount of information was written to them.
There are also several common places to start for determining what is taking up the space on the volume:
  • The /vmimages folder. This folder is used to store operating system install files such as the VMware Tools or other ISO files.
  • The /var/core folder. This folder is used to store crash files for processes on the service console. 
  • The /var/log folder. This folder stores the majority of the logs for the ESX host.
  • Any VMFS volume. These are used to store the virtual machine data.

If you are unable to determine where the space is being used, you can run the following command:

[root@server]# find / -size +10240000c -exec du -h {} \; | less

This command provides a list of all files that are larger than 10MB in size. See the Additional Information section of this article for more options on tweaking this command.
 
After you have determined what is taking up the free space you can use the rm command to free up space by deleting files. 
 
Warning: When you delete a file there is no way to recover it, so use caution when deleting anything.
 
The following are a list of files that are safe to delete:
  • Old vm-support logs.
  • Virtual machines that are not being used and are not needed
  • ISO files that were copied to the system

    Note: Do not delete the VMware Tools ISO files.

  • Old log files that are no longer needed.
  • Virtual Machine log files can be removed if there are a lot (thousands) of vmware*.log files in the virtual machine's folder.  You may need to delete the vmware*.log files using “rm vmware*.log” or if there are too many arguments for this to complete successfully you can try using “rm vmware-1[1-5]*.log” followed by “rm vmware-1*.log”, “rm vmware-2[1-5]*.log” followed by “rm vmware-2*.log”, and then “rm vmware*.log”.  This will delete all the log files for this virtual machine.

If you are unsure about deleting a specific file please contact VMware Support for assistance with this. If a system file is removed inadvertently this may cause damage to your ESX host that can require a re-installation of the software.

Additional Information

Note: Recommended partition sizes can be found in the "Datastore Partitioning" chapter in the VI3 Install & Upgrade guide which can be found at http://www.vmware.com/pdf/vi3_301_201_installation_guide.pdf
 
Note: Here are additional ways to use the find command:
  • To look for files larger than 1MB, change the command so that -size is: +1024000c
  • The find command listed above looks through the entire file system tree. The -mount switch will stop find from crossing mount points. This is useful as you don't want to look in the /vmfs part of the tree, since all the files there are big and not what you are looking for.

Feedback

Rating: 1 - Lowest 2 3 4 5 - Highest (6 Ratings)   

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.
What can we do to improve this information? (2000 or fewer characters)
Submit
Rating: 1 - Lowest 2 3 4 5 - Highest (6 Ratings)   
Actions
  • KB Article: 1003564
  • Updated: Nov 17, 2009
  • Products:
    VMware ESX
  • Product Versions:
    VMware ESX 2.5.x
    VMware ESX 3.0.x
    VMware ESX 3.5.x