Search the VMware Knowledge Base (KB)
View by Article ID

ESXi 5.x host is disconnected from vCenter Server due to sfcbd exhausting inodes (2037798)

  • 15 Ratings

Symptoms

  • The ESXi 5.x host gets disconnected from vCenter Server
  • It cannot connect the ESXi 5.x host to vCenter Server
  • It cannot connect to the ESXi 5.x host using the vSphere Client
  • The /var/log/hostd.log file indicates that the host is out of space:

    VmkCtl Locking (/etc/vmware/esx.conf) : Unable to create or open a LOCK file. Failed with reason: No space left on device
    VmkCtl Locking (/etc/vmware/esx.conf) : Unable to create or open a LOCK file. Failed with reason: No space left on device


  • Running the vdf -h command shows that there is sufficient space in the root file system
  • The /var/log/vmkernel.log file indicates that it is out of inodes:

    cpu4:1969403)WARNING: VisorFSObj: 893: Cannot create file /etc/vmware/esx.conf.LOCK for process python because the visorfs inode table is full.
    cpu11:1968837)WARNING: VisorFSObj: 893: Cannot create file /etc/vmware/esx.conf.LOCK for process hostd because the visorfs inode table is full.
    cpu5:1969403)WARNING: VisorFSObj: 893: Cannot create file /etc/vmware/esx.conf.LOCK for process python because the visorfs inode table is full.
    cpu11:1968837)WARNING: VisorFSObj: 893: Cannot create file /etc/vmware/esx.conf.LOCK for process hostd because the visorfs inode table is full.


  • Running the stat -f / command on ESXi 5.0 indicates that inodes are low
  • Running the localcli system visorfs ramdisk list command on ESXi 5.1 and 5.5 indicates that inodes are low

Cause

This issue occurs by the hardware monitoring service (sfcdb) that populates the /var/run/sfcb directory with over 5000 files.

Resolution

This issue is resolved in:

To work around this issue, free up inodes.
 
To free up inodes:

Note:  This process is a temporary workaround. The inodes can reach exhaustion even after adding the RAM disk for the sfcb tickets.
  1. Connect to the ESXi host using SSH.
  2. Run this command to stop the sfcbd service:

    /etc/init.d/sfcbd-watchdog stop

  3. Manually delete the files in the var/run/sfcb directory to free inodes.

    Run these commands to remove the files:

    cd /var/run/sfcb
    rm [0-2]*
    rm [3-6]*
    rm [7-9]*
    rm [a-c]*
    rm [d-f]*


  4. Run this command to restart the sfcbd service:

    /etc/init.d/sfcbd-watchdog start

    Note: Restart management agents after step 4. For more information, see Restarting the Management agents on an ESXi or ESX host (1003490).
To prevent root file system from getting impacted on ESXi 5.1 and 5.5:

  1. Connect to the ESXi host using SSH.
  2. Run this command to stop the sfcbd service:

    /etc/init.d/sfcbd-watchdog stop

  3. Enable this command to be run at boot time:

    esxcli system visorfs ramdisk add --name sfcbtickets --min-size 0 --max-size 1024 --permissions 0755 --target /var/run/sfcb

    For more information, see Modifying the rc.local or local.sh file in ESX/ESXi to execute commands while booting (2043564).

  4. Run this command to start the sfcbd service:

    /etc/init.d/sfcbd-watchdog start

  5. Run this command and verify if the new ramdisk exists:

    esxcli system visorfs ramdisk list

Note: This issue may also occur if the SNMP agent is enabled. For more information, see vCenter Server and vmkernel logs report the message: The ramdisk 'root' is full (2033073).

See Also

Update History

11/21/2012 - Added ESXi 5.1 to Products 12/02/2013 - Added issue resolved in ESXi 5.0 Update 3

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 15 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.

What can we do to improve this information? (4000 or fewer characters)




Please enter the Captcha code before clicking Submit.
  • 15 Ratings
Actions
KB: