Knowledge Base

The VMware Knowledge Base provides support solutions, error messages and troubleshooting guides
 
Search the VMware Knowledge Base (KB)   View by Article ID
 

VMware ESXi 5.x host becomes unresponsive when attempting a vMotion migration or a configuration change (2040707)

Symptoms

  • VMware ESXi 5.x host becomes unresponsive after attempting to migrate a virtual machine from VMware vCenter Server.
  • Making a configuration change to the ESXi host renders the host unresponsive.
  • Migration fails at 13%.
  • Some of the virtual machines in the inventory become invalid. 
  • vpxa fails to start.
  • You are unable to power on a virtual machine.
  • In the vpxa.log file, you see entries similar to:

    YYYY-MM-DDT09:00:11.615Z [FF8AD6D0 warning 'Libs'] vmware-vpxa: CnxUnix_UseConfigFile lock of /var/run/vmware/f4a0dbedb2e0fd30b80f90123fbe40f8 failed: No space left on device.
    YYYY-MM-DDT09:00:11.615Z [FF8AD6D0 error 'provisioningvpxNfcServer'] Cnx_UseConfigFile(vmware-vpxa, vpxa-nfc) failed
    YYYY-MM-DDT09:00:11.615Z [FF8AD6D0 error 'provisioningvpxNfcServer'] Error setting up NFC server listener

  • When you complete tasks such as a vMotion migration or powering on a virtual machine, you see the error:

    A general system error occurred: Timed out waiting for vpxa to start

  • The vSphere HA agent for the host reports the error:

    The vSphere HA agent is not reachable from vCenter Server.

  • Deploying an OVF template fails with the error:

    A general system error occurred. Failed to create journal file provider

  • In the /var/log/vmkernel.log file of the host, you see entries similar to:

    WARNING: VisorFSObj: 1954: Cannot create file /var/spool/snmp/1351589546_6_3_6876.trp for process hostd-worker because the inode table of its ramdisk (root) is full.
    WARNING: VisorFSObj: 1954: Cannot create file /var/run/vmware/tickets/vmtck-520db3d3-3305-37 for process hostd-worker because the inode table of its ramdisk (root) is full.
    WARNING: VisorFSObj: 1954: Cannot create file /var/spool/snmp/1351589576_6_4_6876.trp for process hostd-worker because the inode table of its ramdisk (root) is full.

    ...

    WARNING: VisorFSRam: 233: Cannot extend visorfs file /var/spool/snmp/1381771804_6_401_6876.trp because its ramdisk (snmptraps) is full.
    WARNING: VisorFSRam: 233: Cannot extend visorfs file /var/spool/snmp/1381772106_6_401_6876.trp because its ramdisk (snmptraps) is full.

    ...

    User: 2742: wantCoreDump : snmpd-enabled : 0



  • You observe similar warnings in the Events tab when connected directly to the ESXi host via the vSphere Client.
  • Opening a virtual machine console from the vSphere Client fails with the error:

    Unable to contact the MKS

  • When you connect to the ESXi host using SSH and run the stat -f filesystem_name command, you see that the host is out of inodes.
  • In the Event viewer of vCenter Server, you see the error:

    Cannot install the vCenter agent service. Cannot upload agent.

  • Running the stat -f filesystem_name command displays output similar to:

    Inodes: Total: 0 Free: 0

    Note: For more information on connecting to an ESXi host via SSH, see Using ESXi Shell in ESXi 5.0 and 5.1 (2004746).

  • Storage vMotion fails and vCenter Server reports an error indicating that the RAM disk is full.
  • In the /var/log/vmkernel.log file, you see an error similar to:

    VisorFSRam: 233: Cannot extend visorfs file /var/log/hpHelper.log because its ramdisk (root) is full.

Cause

This issue occurs when SNMPD is enabled and the /var/spool/snmp folder is filled with Simple Network Management Protocol (SNMP) trap files.

Similar symptoms may also occur on HP ProLiant Gen8 servers running ESXi 5.x, where the /var/log/hpHelper.log file fills the ESXi 5.x RAMDisk.
 

Note: To help identify which files are consuming space, see ESX error: No free space left on device (1007638)


Resolution

This is a known issue affecting VMware ESXi 5.x. Currently, there is no resolution.
 
To work around this issue:
  1. Connect to the ESXi host using SSH. For more information, see Using ESXi Shell in ESXi 5.0 and 5.1 (2004746).
  2. Check if SNMP is creating too many .trp files in the /var/spool/snmp directory on the ESXi host by running the command:

    ls /var/spool/snmp | wc -l

    Note: If the output indicates that the value is 2000 or more, this may be causing the full inodes.

  3. Delete the .trp files in the /var/spool/snmp/ directory by running the commands:

    # cd /var/spool/snmp
    # for i in $(ls | grep trp); do rm -f $i;done


  4. Change directory to /etc/vmware/ and back up the snmp.xml file by running the commands:

    # cd /etc/vmware
    # mv snmp.xml snmp.xml.bkup


  5. Create a new file named snmp.xml and open it using a text editor. For more information, see Editing files on an ESX host using vi or nano (1020302).
  6. Copy and paste these contents to the file:

    <?xml version="1.0" encoding="ISO-8859-1"?>
    <config>
    <snmpSettings><enable>false</enable><port>161</port><syscontact></syscontact><syslocation></syslocation>
    <EnvEventSource>indications</EnvEventSource><communities></communities><loglevel>info</loglevel><authProtocol></authProtocol><privProtocol></privProtocol></snmpSettings>
    </config>


  7. Save and close the file.
  8. Reconfigure SNMP on the affected host by running the command:

    # esxcli system snmp set –-enable=true

  9. To confirm the SNMP services are running normally again, run the command:

    # esxcli system snmp get

    Here is an example of the output:
    /etc/vmware # esxcli system snmp get
    Authentication: Communities: Enable: true Engineid: 00000063000000a10a0121cf Hwsrc: indications Loglevel: info Notraps: Port: 161 Privacy: Remoteusers: Syscontact: Syslocation: Targets: Users: V3targets:

To ensure that the issue does not recur, you can temporarily disable snmpd to stop logging. To stop the snmpd service, run this command:

# /etc/init.d/snmpd stop


Additional Information

Note: In the event that the host has run out of inodes, attempt to stop vpxa on the host to free up an inode:
  1. Connect to the host with the vSphere Client.
  2. Click Configuration > Security Profile.
  3. Under Services, click Properties.
  4. Click vpxa, then click Options.
  5. Under Startup Policy, click Stop.
  6. Click OK.
  7. Click OK.
After vpxa has stopped, an inode is freed up so that you can enable the ESXi Shell and obtain command line access to remove the .trp files as described in the Resolution section. For more information on using the ESXi Shell, see Using ESXi Shell in ESXi 5.0 and 5.1 (2004746).
 
For translated versions of this article, see:

• 日本語: ESXi5.1ホストがvMotion実行または構成変更後に応答しなくなる (2048005)

To be alerted when this article is updated, click Subscribe to Document in the Actions box.

See Also

Update History

12/17/2012 - Added additional symptom 02/06/2013 - Added additional symptom on deploying OVF template 03/26/2013 - Added additional symptoms 3/29/2013 - Added HP known issues and the visorfs symptom. 04/17/2014 - Added external HP link under Cause: Also, see HP Proliant Gen8 Servers - ESXi 5: The /var/log/hpHelper.log File Can Grow Very Large and Fill the ESXi 5 RAMDisk.

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 21 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.
What can we do to improve this information? (4000 or fewer characters)
  • 21 Ratings
Actions
KB: