Knowledge Base

The VMware Knowledge Base provides support solutions, error messages and troubleshooting guides
 
Search the VMware Knowledge Base (KB)   View by Article ID
 

ESXi host is disconnected from vCenter Server due to sfcbd exhausting inodes (2037798)

Symptoms

  • ESXi 5.x host gets disconnected from vCenter Server
  • Cannot connect the ESXi 5.x host to vCenter Server
  • Cannot connect to ESXi 5.x host using the vSphere Client 
  • The hostd.log file indicates that the host is out of space:

    VmkCtl Locking (/etc/vmware/esx.conf) : Unable to create or open a LOCK file. Failed with reason: No space left on device
    VmkCtl Locking (/etc/vmware/esx.conf) : Unable to create or open a LOCK file. Failed with reason: No space left on device

  • Running the vdf -h command shows that there is sufficient space in the root file system
  • The vmkernel.log file indicates that it is out of inodes:

    cpu4:1969403)WARNING: VisorFSObj: 893: Cannot create file /etc/vmware/esx.conf.LOCK for process python because the visorfs inode table is full.
    cpu11:1968837)WARNING: VisorFSObj: 893: Cannot create file /etc/vmware/esx.conf.LOCK for process hostd because the visorfs inode table is full.
    cpu5:1969403)WARNING: VisorFSObj: 893: Cannot create file /etc/vmware/esx.conf.LOCK for process python because the visorfs inode table is full.
    cpu11:1968837)WARNING: VisorFSObj: 893: Cannot create file /etc/vmware/esx.conf.LOCK for process hostd because the visorfs inode table is full.

  • The output of stat -f / indicates that inodes are low

Cause

This issue is caused by the hardware monitoring service (sfcdb) that populates the /var/run/sfcb directory with over 5000 files.

Resolution

To resolve this issue, you must free up inodes.
 
To free up inodes:
  1. Connect to the ESXi host using SSH.
  2. Run this command to stop the sfcbd service:

    /etc/init.d/sfcbd-watchdog stop

  3. Manually delete the files in /var/run/sfcb directory to free inodes.

    To remove these files, run this command:

    # cd /var/run/sfcb
    # rm [0-2]*
    # rm [3-6]*
    # rm [7-9]*
    # rm [a-c]*
    # rm [d-f]*

  4. Run this command to restart the sfcbd service:

    /etc/init.d/sfcbd-watchdog start

See Also

Update History

11/21/2012 - Added ESXi 5.1 to Products

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 2 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.
What can we do to improve this information? (4000 or fewer characters)
  • 2 Ratings
Actions
KB: