Knowledge Base

The VMware Knowledge Base provides support solutions, error messages and troubleshooting guides
 
Search the VMware Knowledge Base (KB)   View by Article ID
 

ESXi 5.1 host becomes unresponsive when attempting a vMotion migration or a configuration change (2040707)

Symptoms

  • An ESXi 5.1 host becomes unresponsive when you attempt a vMotion migration of a virtual machine from vCenter Server.
  • Making a configuration change to the ESXi host renders the host unresponsive.
  • vMotion fails at 13% (13 percent).
  • Some of the virtual machines in inventory become invalid. 
  • vpxa fails to start.
  • In the vpxa.log file, you see entries similar to:

    013-06-12T09:00:11.615Z [FF8AD6D0 warning 'Libs'] vmware-vpxa: CnxUnix_UseConfigFile lock of /var/run/vmware/f4a0dbedb2e0fd30b80f90123fbe40f8 failed: No space left on device.
    2013-06-12T09:00:11.615Z [FF8AD6D0 error 'provisioningvpxNfcServer'] Cnx_UseConfigFile(vmware-vpxa, vpxa-nfc) failed
    2013-06-12T09:00:11.615Z [FF8AD6D0 error 'provisioningvpxNfcServer'] Error setting up NFC server listener

  • When you complete tasks such as a vMotion migration or powering on a virtual machine, you see the error:

    A general system error occurred: Timed out waiting for vpxa to start

  • The vSphere HA agent for the host reports the error:

    The vSphere HA agent is not reachable from vCenter Server.
  • Deploying an OVF template fails with the error:

    A general system error occurred. Failed to create journal file provider

  • In the /var/log/vmkernel.log file of the host, you see entries similar to:

    WARNING: VisorFSObj: 1954: Cannot create file /var/spool/snmp/1351589546_6_3_6876.trp for process hostd-worker because the inode table of its ramdisk (root) is full.
    WARNING: VisorFSObj: 1954: Cannot create file /var/run/vmware/tickets/vmtck-520db3d3-3305-37 for process hostd-worker because the inode table of its ramdisk (root) is full.
    WARNING: VisorFSObj: 1954: Cannot create file /var/spool/snmp/1351589576_6_4_6876.trp for process hostd-worker because the inode table of its ramdisk (root) is full.

  • When connected directly to the ESXi host via the vSphere Client, you see similar warnings in the Events tab.
  • Opening a virtual machine console from the vSphere Client fails with the error:

    Unable to contact the MKS

  • When you connect to the ESXi host using SSH and run the stat -f filesystem_name command, you see that the host is out of inodes.
  • In the Event viewer of vCenter Server, you see the error:

    Cannot install the vCenter agent service. Cannot upload agent.
  • Running the stat -f filesystem_name command displays output similar to:

    Inodes: Total: 0 Free: 0

    Note: For more information on connecting to an ESXi host via SSH, see Using ESXi Shell in ESXi 5.x (2004746).

  • Storage vMotion fails and vCenter Server reports an error indicating that the RAM disk is full.
  • In the /var/log/vmkernel.log file, you see an error similar to:

    VisorFSRam: 233: Cannot extend visorfs file /var/log/hpHelper.log because its ramdisk (root) is full.

Cause

This issue occurs when SNMPD is enabled and the /var/spool/snmp folder is filled with Simple Network Management Protocol (SNMP) trap files.

This issue may also occur on HP ProLiant Gen8 servers running ESXi 5.x where the /var/log/hpHelper.log file fills the ESXi 5.x RAMDisk. For more information, see The /var/log/hpHelper.log File Can Grow Very Large and Fill the ESXi 5 RAMDisk.
 
Note: The preceding link was correct as of June 11, 2013. If you find the link is broken, provide feedback and a VMware employee will update the link.

Resolution

This issue is resolved in ESXi 5.1 Update 1. To download ESXi 5.1 Update 1, see the vSphere Download Center.

To work around this issue when you do not want to upgrade:
  1. Connect to the ESXi host using SSH. For more information, see Using ESXi Shell in ESXi 5.x (2004746).
  2. Check if SNMP is creating too many .trp files in the /var/spool/snmp directory on the ESXi host by running the command:

    ls /var/spool/snmp | wc -l

    Note: If the output indicates that the value is 2000 or more, then this may be causing the full inodes.

  3. Delete the .trp files in the /var/spool/snmp/ directory by running the commands:

    # cd /var/spool/snmp
    # for i in $(ls | grep trp); do rm -f $i;done

  4. Change directory to /etc/vmware/ and back up the snmp.xml file by running the commands:

    # cd /etc/vmware
    # mv snmp.xml snmp.xml.bkup

  5. Create a new file named snmp.xml and open it using a text editor. For more information, see Editing files on an ESX host using vi or nano (1020302).
  6. Copy and paste these contents to the file:

    <?xml version="1.0" encoding="ISO-8859-1"?>
    <config>
    <snmpSettings><enable>false</enable><port>161</port><syscontact></syscontact><syslocation></syslocation>
    <EnvEventSource>indications</EnvEventSource><communities></communities><loglevel>info</loglevel><authProtocol></authProtocol><privProtocol></privProtocol></snmpSettings>
    </config>

  7. Save and close the file.
  8. Reconfigure SNMP on the affected host by running the command:

    # esxcli system snmp set –-enable=true

  9. To confirm the SNMP services are running normally again, run the command:

    # esxcli system snmp get

    Here is an example of the output:
    /etc/vmware # esxcli system snmp get
    Authentication: Communities: Enable: true Engineid: 00000063000000a10a0121cf Hwsrc: indications Loglevel: info Notraps: Port: 161 Privacy: Remoteusers: Syscontact: Syslocation: Targets: Users: V3targets:

To ensure that the issue does not recur, you can temporarily disable snmpd to stop logging. To stop the snmpd service, run this command:

# /etc/init.d/snmpd stop

Additional Information

Note: In the event that the host has run out of inodes, attempt to stop vpxa on the host to free up an inode:
  1. Connect to the host with the vSphere Client.
  2. Click Configuration > Security Profile.
  3. Under Services, click Properties.
  4. Click vpxa, then click Options.
  5. Under Startup Policy, click Stop.
  6. Click OK.
  7. Click OK.
Once vpxa has stopped, this frees up an inode so that you can enable the ESXi Shell and obtain command line access to remove the .trp files as described in the Resolution section. For more information on using the ESXi Shell, see Using ESXi Shell in ESXi 5.x (2004746).

See Also

Update History

12/17/2012 - Added additional symptom 02/06/2013 - Added additional symptom on deploying OVF template 03/26/2013 - Added additional symptoms 3/29/2013 - Added HP known issues and the visorfs symptom. 04/29/2013 - Added that the issue is resolved in ESXi 5.1 Update 1 and added the download link

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 10 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.
What can we do to improve this information? (4000 or fewer characters)
  • 10 Ratings
Actions
KB: