Knowledge Base

The VMware Knowledge Base provides support solutions, error messages and troubleshooting guides

Search the VMware Knowledge Base (KB)

View by Article ID

Too many vmware.log files created on VMFS datastores results in delays or failure to power on virtual machines

Symptoms

Too many files on the VMFS volume, can cause slowdowns of directory content enumeration or reading, as well as service failures in ESX components such as vmware-hostd.

Backup products are unable to complete virtual machine backups if there are too many files on the VMFS datastore.

Inordinate amounts of vmware.log files reside in a given virtual machine's directory (several hundreds or thousands).

Attempting to power on virtual machines after a host reboot results in an error:

Unable to power up vm guests. Error: cannot find swap file.

Trying to associate a new virtual machine with existing disks fails as the new virtual machine fails on creation.
File creation on an affected VMFS datastore fails, some file deletion attempts may fail.
Virtual machines that do not power on are cited in frequent /var/log/vmkwarning logging with errors similar to:

WARNING: Swap: vm 13001: 1480: Failed to create swap file '/volumes/DatastoreName/VMfolder/VMname-6bc43c2b.vswp': Out of resources
In VMware vCenter or VirtualCenter, after selecting Events in the Tasks & Events tab for a virtual machine, you see numerous, current, Failover failed for this virtual machine messages.
Virtual machines appear as powered off, while in fact they are running and accessible on the virtual machine network.

Resolution

Due to VMware High Availability issues, one or more virtual machines have been attempting to power on repeatedly, generating new vmware.log files over an extended period of time. To resolve this issue, remove the extraneous vmware.log files.

Caution: This article provides steps that require you to permanently delete files from your datastore. Use extreme caution when performing these steps.

Disable and re-enable VMware HA on the cluster to stop the VMware HA issue from continuing further. This can be verified as failover errors stop appearing on the Events tab of affected virtual machines in VirtualCenter.
At the service console, change directory to the folder of a virtual machine that contains extraneous amounts of vmware-x.logs.

Note: File access is delayed, so commands to search all datastores for log files return too many files or fails. Proceed through affected virtual machines, one by one.

To get a count on the number of vmware-x.log files in the folder, run:

# ls *.log | wc -l

To remove the logs, run:

# rm -f vmware-*.log

This deletes all log files, unless there are too many arguments. That is, too many files for the rm command to handle. In this case, try deleting all files beginning with vmware-1:

# rm -f vmware-1*.log

If there are tens of thousands of files, there may be over a thousand beginning with 1, so again too many arguments. You must be more specific:

# rm -f vmware-10*.log
# rm -f vmware-11*.log
# rm -f vmware-12*.log

If necessary, continue to narrow down the command's deletion candidates until you reach an acceptable file count.
Keep removing the log files until only vmware.log remains. Repeat for any other virtual machines affected.

Additional Information

Some additional commands which might assist with deleting these files:

This command will list all vmware.log files that are older than 1 day:

cd /; find /vmfs/volumes/ -name vmware-*.log -mtime +1 -exec ls {} \;

This command can be used to delete vmware.log files that are older than 1 day:

cd /; find /vmfs/volumes/ -name vmware-*.log -mtime +1 -exec ls {} \; | xargs rm

This command produces the below output. Note the line starting with Files (max/free). There are too many files on the datastore, consuming all available inodes. This prevents creation of further files.

vmkfstools -Ph -v 10 /vmfs/volumes/<volume name>

VMFS-3.31 file system spanning 9 partitions.
File system label (if any):
Mode: public
Capacity 9.4T, 1.1T available, file block size 8.0M
Volume Creation Time: Thu Oct 16 17:10:08 2008
Files (max/free): 30720/0
Ptr Blocks (max/free): 61440/60208
Sub Blocks (max/free): 3968/1216
UUID: 48f7d7e0-295b9248-7710-0015174a4432
Partitions spanned (on "lvm"):
         vmhba1:0:36:1
         vmhba1:0:37:1
         vmhba1:0:35:1
         vmhba1:0:34:1
         vmhba1:0:33:1
         vmhba1:0:20:1
         vmhba1:0:21:1
         vmhba1:0:22:1
         vmhba1:0:23:1

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

1 2 3 4 5

1 2 3 4 5

Actions

Bookmark Document

Did this article help you?
This article resolved my issue. This article did not resolve my issue. This article helped but additional information was required to resolve my issue.
What can we do to improve this information? (4000 or fewer characters)