Knowledge Base

The VMware Knowledge Base provides support solutions, error messages and troubleshooting guides
 
Search the VMware Knowledge Base (KB)   View by Article ID
 

ESX error: No free space left on device (1007638)

Symptoms

  • ESX/ESXi host console indicates that/ is out of space
  • You see the error:

    No free space left on device

  • A virtual machine fails to start.
  • A log message reports that the file system is full.
  • The df command indicates that there is free space.
  • The vdf command indicates that there is free space.
  • You cannot disable VMware High Availability (HA).
  • Processes become defunct.

Resolution

If a filesystem has unused space on it, but the number of inodes has reached the maximum, no new files can be created. This can appear like a full filesystem.

You must determine if the ESX console file system and the VMFS file system have reached the inode or filesystem limit.
 
To determine if the ESX console file system and the VMFS file system have reached the inode or filesystem limit:
  1. Gain root access to the command line.
  2. On the ESX host, run the command:

    [root@esx /]$ stat -f /

    The output appears similar to:

    File: "/"
    ID: 0 Namelen: 255 Type: ext2/ext3
    Blocks: Total: 1259079 Free: 898253 Available: 834295 Size: 4096
    Inodes: Total: 640000 Free: 580065

    In this example, there are 640000 inodes total and 580065 are free, so there are inodes available.


  3. If, however, you have exhausted the inode limit, you must remove some files. To free some space, see Freeing disk space on an ESX Server in Investigating disk space on an ESX host (1003564) or Freeing ESXi inodes (1008643).

    To generate a list of log files that may be candidates for deletion, run:

    find / -path "/vmfs" -prune -o -type f -size +50000k -exec ls -l '{}' \;

    This command seeks out files 50,000 KB (50MB) or larger, skips the /vmfs directory where virtual machines may be located, and provides detailed output for each file. This may return locally-stored ISO files, log bundles, large log files, etc.

    Some typical paths where excess files may be located are:
    • /tmp/vmware-root
    • /var/core
    • /var/log/vmware/aam/rule

    Warning:
    • Carefully review the files in question before removal using rm. Files cannot be recovered after deletion.
    • Do not delete log information unless absolutely necessary. Consider archiving older logging elsewhere.


  4. Run the following command against the VMFS volume:

    vmkfstools -P -v 10 /vmfs/volumes/466e7eb9-3692da25-a15c-001321caa748

    The output appears similar to:

    VMFS-3.21 file system spanning 1 partitions.
    File system label (if any): lun1 (2)
    Mode: public
    Capacity 5100273664 (4864 file blocks * 1048576), 148897792 (142 blocks) avail
    Volume Creation Time: Tue Jun 12 12:08:41 2007
    Files (max/free): 30720/30707
    Ptr Blocks (max/free): 61440/61436
    Sub Blocks (max/free): 3968/3963
    UUID: 466e7eb9-3692da25-a15c-001321caa748
    Partitions spanned:
            vmhba1:0:1:1

    In this example, none of the values for Files, Ptr Blocks and Sub Blocks are at zero so there are available VMFS entries for new files.

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 9 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.
What can we do to improve this information? (4000 or fewer characters)
  • 9 Ratings
Actions
KB: