Freeing ESXi inodes

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:
ESXi hosts that exhibit these symptoms may have low inodes:

The ESXi host disconnects from vCenter Server
The services.sh script does not restart services
The output of stat -f / indicates that inodes are low
The host management agent, the hostd process becomes unresponsive
The esxcfg-swiscsi -q command (or any command that interacts with esx.conf, such as esxcfg-info) produces errors similar to:
- Error interacting with configuration file /etc/vmware/esx.conf: Write failed during Unlock.This is likely due to a full or read-only filesystem
- Error interacting with configuration file /etc/vmware/esx.conf:
  Unable to write to file /etc/vmware/esx.conf.bXQvyt while saving /etc/vmware/esx.conf operation aborted.
- Unlock of ConfigFileLocker failed : Error interacting with configuration file /etc/vmware/esx.conf
On the summary page for a host, you see a message similar to:

Unable to apply DRS resource settings on host xxx.xxx.xxx.xxx in <ClusterName>. A general system error occurred: vmodl.fault.SystemError. This can significantly reduce the effectiveness of DRS.

Environment

VMware vSphere ESXi 5.5
VMware vSphere ESXi 6.0
VMware vSphere ESXi 6.5
VMware vSphere ESXi 5.0
VMware vSphere ESXi 5.1

Resolution

If an ESXi host has low inodes, you may have to free inodes on the host.

To verify that a host has low inodes, run the command:

[root@esxhost root]# stat -f /

The output appears similar to:

File: "/"
ID: 0 Namelen: 255 Type: ext2/ext3
Blocks: Total: 1290707 Free: 718116 Available: 652551 Size: 4096
Inodes: Total: 656000 Free: 0

Note: The Free: field refers to the number of available free inodes. If it is zero (or close to 0), the amount of free inodes is insufficient.

To free inodes on an ESXi host:

On the ESXi host, enter Tech Support Mode. For more information, see Tech Support Mode for Emergency Support (1003677).
Delete the /var/log/vmware/aam/rule directory.
Delete any files in the /var/core/ directory.
Uninstall the vpxa and aam clients with the uninstall scripts in /opt/vmware/uninstallers (VMware-vpxa-uninstall.sh and VMware-aam-uninstall.sh, respectively).
Restart ESXi services with the /sbin/services.sh script.
Reconnect the ESXi host to VirtualCenter.
Investigate for the recent presence of more than usual small files, such as files ending .txt or .dmp. If they are present, delete the files, then check the inodes.

Additional Information

Tech Support Mode for Emergency Support
ESXi inode の解放