Troubleshooting a datastore or VMFS volume that is full or near capacity (1003412)
ESX hosts store all files necessary for virtual machine operation in a VMware Virtual Machine File System (VMFS). Proper capacity management is critical because this file system is required for any operations related to a virtual machine. This document outlines potential issues related to a VMFS becoming full.
All VMFS (vmfs2, vmfs3, vmfs5) must retain a level of free space to ensure that the virtual machine functions properly. Ensure that these capacity thresholds are met:
- 200 Megabyte Capacity: VMware suggests 200 MB as the threshold where ESX system administrators take action to resolve capacity issues. VMFS volumes vary widely in size and it is impossible to define a precise percentage, but the risk of unexpected problems become greater at this point.
- 100 Megabyte Remaining: If left unchecked, a VMFS volume grows until only 100 MB of space remains, at which time issues may occur. 100 MB is defined as a critical threshold at which ESX system administrators must take immediate action to correct the issue because VMware VMFS stops growing at this point.
For more information, see Extending a VMFS volume to use new LUN space after expansion (1752).
This is a list of some potential issues that may affect virtual machines residing on a VMFS that is full:
- Virtual machine startup failure: ESX hosts are able to provide more RAM space than is available in the total physical RAM installed in the server. This is called Over Commitment. ESX hosts create
.vswpfiles for any virtual machine that has allocated memory greater than the physical RAM that is available to the virtual machine. The
<VMName>.vswpfile is created when the virtual machine is started. Without sufficient space, this file cannot be created, and the virtual machine is unable to start. For more information, see ESX Virtual Machine Won't Start (Insufficient Memory) (1330).
- VMotion failures: ESX hosts that are in a cluster of ESX hosts that have VMware HA/DRS enabled may encounter failures when VMotion attempts to move a virtual machine from one host to another. It is possible for this failure to occur during off hours when it may not be immediately identified as a failure because VMware DRS can be configured to VMotion virtual machines without interaction from an administrator.
- Snapshot creation/commit: When a snapshot is taken or committed to the virtual machine, data is written to the VMFS. If the VMFS is unable to accommodate the additional data, the snapshot does not grow, does not get written, or is not committed. As a result,the virtual machine may fail.
- Poor performance: File copy,
lscommands, and a variety of Linux level functions perform poorly when the space is full because the VMware VMFS is a filesystem. Though less likely to affect virtual machine availability, these issues impact ESX system administrators and must be resolved.
Note: Journaling filesystems (like VMFS) are generally capable of maintaining proper performance when at nearly full capacity. However, the probability of performance problems is greatly increased when the filesystem becomes 100% full.
- Unpredictability: It is difficult to predict precisely how a full filesystem may create issues because VMFS is a core element of the ESX host. To ensure more efficient troubleshooting and system operation, keep VMFS below the thresholds defined above.
Note: This is not a comprehensive list of all possible issues, but it identifies important issues that have been reported in past instances of a full VMFS.
What to do about a full VMware VMFS
When a VMFS is full, you have several courses of action:
- Increase capacity: A VMFS may be stored on an external storage array. Depending on the features of the storage array, it may be possible to extend the external storage. For more information on increasing datastore, see Growing or expanding a VMFS volume or datastore (1017662). For adding an extent, see Guidelines for extending a VMFS Volume in ESX 3.x (1884).
- Move virtual machines to other storage: If you cannot increase the storage available on the existing VMware VMFS, but another VMFS is accessible through that host, you can move virtual machines to the other VMware VMFS. For more information on moving virtual machines, see Moving or copying a virtual machine within a VMware environment (1000936).
- Add additional datastores: ESX hosts are able to maintain up to 128 (256 for ESX 3.0) datastores. If you are unable to identify existing free space where the virtual machine can be stored, you may add additional storage. To do so, refer to the SAN Configuration Guide for the version of the ESX host you are running.
- Remove old snapshots: It is possible that additional space can be reclaimed by either committing or removing snapshots because old snapshots also reside on VMware VMFS. This procedure requires direct knowledge of each virtual machine on the filesystem and must be performed by trained ESX system administrators. For more information, see the VMware Server Administration Guide. For more information on the topic of snapshots, see No more space for the redo log error when attempting to start a virtual machine (1002103) and Understanding virtual machine snapshots in VMware ESX (1015180).
- Clean up VMware VMFS: You can store files not related to the virtual machine operations because VMware VMFS is a filesystem. In this case, an experienced administrator carefully reviews the contents of the filesystem, and determines what files are safe to remove. VMware strongly recommends that any files that are deleted from a VMware VMFS are first backed up, in case the file proves to be necessary.
Additional InformationFor translated versions of this article, see:
- 日本語: 空き容量がない、または少ないデータストアおよび VMFS ボリュームのトラブルシューティング (2000467)
- 简体中文: 对容量已满或将满的数据存储或 VMFS 卷进行故障排除 (2077565)