VMware
 

Knowledge Base

Search the Knowledge Base:
Products:
Search In:
 

Powering off an unresponsive virtual machine on an ESX host

Symptoms

  • You cannot power off an ESX host virtual machine
  • A virtual machine is unresponsive and cannot be killed or stopped
  • You cannot access or unlock files on a virtual machine. For more information, see Virtual machine does not power on because of missing or locked files (10051).
  • After shutting down a virtual machine, Virtual Center shows the virtual machine as up and running
  • There is no indication a virtual machine is shutdown
  • You cannot edit properties in the virtual machine
  • The following error(s) are displayed:
    • Soap error 999. The operation is not allowed in current state.    
    • The attempted operation cannot be performed in the current state (Powered Off)

Purpose

This article provides the correct sequence for shutting down or (when necessary) killing an unresponsive virtual machine.
 
Note: This article pertains to ESX and does not apply to ESXi. For ESXi hosts, see Powering off a virtual machine on an ESXi host (1014165).

Resolution

Warning: Follow the sections and steps in this article in order. Do not skip a section or step, as each step may have an impact on the virtual machine.
 
Powering off the virtual machine
 
To determine if you must use the command line, attempt to power off the virtual machine:
  1. Connect VMware Infrastructure (VI) Client to the Virtual Center Server. Right-click on the virtual machine and click Power off.
  2. Connect VI Client directly to the ESX host. Right-click on the virtual machine and click Power off.

    If this does not work, you must use the command line method.
Determining the virtual machine's state
  1. Determine the host on which the virtual machine is running. This information is available in the virtual machine's Summary tab when viewed in the VI Client page.
  2. Log in as root to the ESX host using an SSH client. 
  3. Run the following command to verify that the virtual machine is running on this host:

    # vmware-cmd -l

    The output of this command returns the full path to each virtual machine running on the ESX host. Verify that the virtual machine is listed, and record the full path for use in this process. For example:

    # /vmfs/volumes/<UUID>/<VMDIR>/<VMNAME>.vmx
  4. Run the following command to determine the state in which the ESX host believes the virtual machine to be operating:

    # vmware-cmd <path.vmx> getstate

    If the output from this command is getstate() = on, the Virtual Center Server may not be communicating with the host properly. This issue must be addressed in order to complete the shutdown process.

    If the output from this command is getstate() = off, the ESX host may be unaware it is still running the virtual machine. This article provide additional assistance in addressing this issue.
Using the ESX command line tool to power off the virtual machine
 
This procedure uses the ESX command line tool, and attempts to gracefully power off the virtual machine. It works if the virtual machine's process is running properly and is accessible. If unsuccessful, the virtual machine's process may not be running properly and may require further troubleshooting.
  1. From the Service Console of the ESX host, run the following command:

    vmware-cmd <path.vmx> stop

    Note: <path.vmx> is the complete path to the configuration file, as determined in the previous section. To verify that it is stopped, run the command:

    # vmware-cmd <path.vmx> getstate
  2. From the Service Console of the ESX host, run the command:

    # vmware-cmd <path.vmx> stop hard

    Note: <path.vmx> is the complete path to the configuration file, as determined in the previous section. To verify that it is stopped, run the command:

    # vmware-cmd <path.vmx> getstate
  3. If the virtual machine is still inaccessible, proceed to the next section. 
Using alternate ESX command line methods to power off the virtual machine
 
This procedure uses additional ESX command line tools to stop the virtual machine. Perform these steps in order, as they are listed in order of potential impact to the system if performed incorrectly.
 
Perform these steps first: 
  1. Determine the WorldID with the command:

    # vm-support -x

  2. Kill the virtual machine by using the following command in the home directory of the virtual machine:

    # vm-support -X <world_ID>

    This can take upwards of 30 minutes to terminate the virtual machine.

    Note    : If the vm-support command is slow with the -X switch, add -d0 to speed up the process.
If the preceding steps fail, perform the following steps for an ESX 3.x host:
  1. List all running virtual machines to find the VMID of the affected virtual machine with the command:

    # cat /proc/vmware/vm/*/names
  2. Determine the master world ID with the command:

    # less -S /proc/vmware/vm/####/cpu/status

  3. Scroll to the right with the arrow keys until you see the group field. It appears similar to:

    Group
    vm.####

  4. Run the following command to shut the virtual machine down with the group ID:

    # /usr/lib/vmware/bin/vmkload_app -k 9 ####
Using the ESX command line to kill the virtual machine
 
If the virtual machine does not power off using the steps in this article, it has likely lost control of its process. You need to manually kill the process at the command line.
 
Caution: This procedure is potentially hazardous to the ESX host. If you do not identify the appropriate process id (PID), and kill the wrong process, it may have unanticipated results. If you are not comfortable with the following procedure, contact VMware Technical Support and open a Service Request. Please refer to this article when you create the SR.
  1. To determine if the virtual machine process is running on the ESX host, run the command:

    # ps -auxwww |grep -i <VMNAME>.vmx

    The output of this command appears similar to the following if the .vmx process is running:

    root 3093 0.0 0.3 2016 860 ? S< Jul30 0:17 /usr/lib/vmware/bin/vmkload_app /usr/lib/vmware/bin/vmware-vmx -ssched.group=host/user -# name=VMware ESX Server;version=3.5.0;licensename=VMware ESX Server;licenseversion=2.0 build-158874; -@ pipe=/tmp/vmhsdaemon-0/vmx569228e44baf49d1; /vmfs/volumes/49392e30-162037d0-17c6-001f29e9abec/<VMDIR>/<VMNAME>.vmx

    The process ID (PID) for this process is in bold. In this example, the PID is 3093. Take note of this number for use in the following steps.

    Caution: Ensure that you identify the line specific only to the virtual machine you are attempting to repair. If you continue this process for another virtual machine the one in question, you can cause downtime for the other virtual machine.

    If the .vmx process is listed, it is possible that the virtual machine has lost control of the process and that it must be stopped manually.
  2. To kill the process, run the command:

    # kill <PID>
  3. Wait 30 seconds and check for the process again.
  4. If it is not terminated, run the command:

    # kill -9 <PID>
  5. Wait 30 seconds and check for the process again.
  6. If it is not terminated, the ESX host may need to be rebooted to clear the process. This is a last resort option, and should only be attempted if the preceding steps in this article are unsuccessful.

Feedback

Rating: 1 - Lowest 2 3 4 5 - Highest (12 Ratings)   

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.
What can we do to improve this information? (2000 or fewer characters)
Submit
Rating: 1 - Lowest 2 3 4 5 - Highest (12 Ratings)   
Actions
  • KB Article: 1004340
  • Updated: Nov 2, 2009
  • Products:
    VMware ESX
    VMware VirtualCenter
  • Product Versions:
    VMware ESX 3.0.x
    VMware ESX 3.5.x
    VMware ESX 4.0.x
    VMware VirtualCenter 2.0.x
    VMware VirtualCenter 2.5.x