Knowledge Base

The VMware Knowledge Base provides support solutions, error messages and troubleshooting guides
 
Search the VMware Knowledge Base (KB)   View by Article ID
 

Powering off an unresponsive virtual machine on an ESX host (1004340)

Symptoms

  • You cannot power off virtual machine on an ESX host.
  • A virtual machine is unresponsive and cannot be killed or stopped.
  • You cannot access or unlock files on a virtual machine.
  • After shutting down a virtual machine, vCenter Server shows the virtual machine as up and running.
  • There is no indication that a virtual machine is shut down.
  • You cannot edit properties in the virtual machine.
  • You see one or more of these errors:
    • Soap error 999. The operation is not allowed in current state.
    • The attempted operation cannot be performed in the current state (Powered Off).
    • The request refers to an object that no longer exists or has never existed

Purpose

This article provides the correct sequence for shutting down or (when necessary) killing an unresponsive virtual machine.
 
Note: This article covers ESX and does not apply to ESXi. For ESXi hosts, see Powering off a virtual machine on an ESXi host (1014165).

Resolution

 
Warning: Follow the sections and steps in this article in order. Do not skip a section or step, as each step may have an impact on the virtual machine.
 
Powering off the virtual machine
 
To determine if you must use the command line, attempt to power off the virtual machine:
  1. Connect VMware Infrastructure/vSphere Client to the vCenter Server. Right-click the virtual machine and click Power off.
  2. Connect vSphere Client directly to the ESX host. Right-click the virtual machine and click Power off.

    If this does not work, you must use the command line method.
Determining the virtual machine's state
  1. Determine the host on which the virtual machine is running. This information is available in the virtual machine's Summary tab when viewed in the vSphere Client page.
  2. Log in as root to the ESX host using an SSH client.
  3. Run this command to verify that the virtual machine is running on this host:

    # vmware-cmd -l

    The output of this command returns the full path to each virtual machine running on the ESX host. Verify that the virtual machine is listed, and record the full path for use in this process. For example:

    # /vmfs/volumes/<UUID>/<VMDIR>/<VMNAME>.vmx

  4. Run this command to determine the state in which the ESX host believes the virtual machine to be operating:

    # vmware-cmd <path.vmx> getstate

    If the output from this commandisgetstate() = on, the vCenter Server may not be communicating with the host properly. This issue must be addressed in order to complete the shutdown process.

    If the output from this command is getstate() = off, the ESX host may be unaware it is still running the virtual machine. This article provides additional assistance in addressing this issue.


Powering off the virtual machine using the vmware-cmd command
 
Caution: If you want to collect the virtual machine logs to assist in troubleshooting, do not perform the steps in this section.
 
This procedure uses the ESX command line tool and attempts to gracefully power off the virtual machine. It works if the virtual machine's process is running properly and is accessible. If unsuccessful, the virtual machine's process may not be running properly and may require further troubleshooting.
  1. From the Service Console of the ESX host, run the command:

    vmware-cmd <path.vmx> stop

    Note: <path.vmx>is the complete path to the configuration file, as determined in the previous section. To verify that it is stopped, run the command:

    # vmware-cmd <path.vmx> getstate

  2. From the Service Console of the ESX host, run the command:

    # vmware-cmd <path.vmx> stop hard

    Note:<path.vmx>is the complete path to the configuration file, as determined in the previous section. To verify that it is stopped, run the command:

    # vmware-cmd <path.vmx> getstate

  3. If the virtual machine is still inaccessible, proceed to the next section.


Powering off the virtual machine while collecting diagnostic information using the vm-support script
 
Use this procedure when you want to investigate the cause of the issue. This command attempts to power off the virtual machine while collecting diagnostic information. Perform these steps in order, as they are listed in order of potential impact to the system if performed incorrectly.
 
Perform these steps first:
  1.  Determine the WorldID of the virtual machine with the command:

    # vm-support -x

    1. Kill the World_ID of the virtual machine by using this command in the root directory of the host:

      # vm-support -X <world_ID>


      It can take upwards of 30 minutes to terminate the virtual machine. Exercise patience to avoid corruption.

      Note: This command uses several different methods to stop the virtual machine. When attempting each method, the command waits for a predetermined amount of time. The timeout value can be configured to be 0 by adding the -d0 switch to the vm-support command.

    If the preceding steps fail, perform these steps for an ESX 3.x host:

    1. List all running virtual machines to find the VMID of the affected virtual machine with the command:

         
      # cat /proc/vmware/vm/*/names

    2. Determine the master world ID with the command:

      # cat /proc/vmware/vm/####/cpu/status | less

    3. Scroll to the right with the arrow keys until you see the group field. It appears similar to:

      Group
      vm.####

    4. Run this command to shut the virtual machine down with the group ID:

      # /usr/lib/vmware/bin/vmkload_app -k 9 ####
     
     If the preceding steps fail, perform these steps for an ESX 4.x host:

    1. List all running virtual machines to find the vmxCartelID of the affected virtual machine with the command:

      # /usr/lib/vmware/bin/vmdumper -l


    2. Scroll through the list until you see your virtual machine's name. The output appears similar to:

      vmid=5151 pid=-1 cfgFile="/vmfs/volumes/4a16a48a-d807aa7e-e674-001e4ffc52e9/mdineeen_test/vm_test.vmx" uuid="56 4d a6 db 0a e2 e5 3e-a9 2b 31 4b 69 29 15 19" displayName="vm_test" vmxCartelID=####

    3. Run this command to shut the virtual machine down with the vmxCartelID:

      # /usr/lib/vmware/bin/vmkload_app -k 9 ####


    Using the ESX command line to kill the virtual machine
     
    If the virtual machine does not power off using the steps in this article, it has likely lost control of its process. You must manually kill the process at the command line.
     
    Caution: This procedure is potentially hazardous to the ESX host. If you do not identify the appropriate process ID (PID), and kill the wrong process, it may have unexpected results. If you are not comfortable with these procedures, contact VMware Technical Support and open a Service Request. Refer to this article when you create the SR.

    1. To determine if the virtual machine process is running on the ESX host, run the command:

      # ps auxwww | grep -i <VMNAME>.vmx

      The output of this command appears similar to this if the .vmx process is running:

      root 3093 0.0 0.3 2016 860 ? S< Jul30 0:17 /usr/lib/vmware/bin/vmkload_app /usr/lib/vmware/bin/vmware-vmx -ssched.group=host/user -# name=VMware ESX Server;version=3.5.0;licensename=VMware ESX Server;licenseversion=2.0 build-158874; -@ pipe=/tmp/vmhsdaemon-0/vmx569228e44baf49d1; /vmfs/volumes/49392e30-162037d0-17c6-001f29e9abec/<VMDIR>/<VMNAME>.vmx

      The process ID (PID) for this process is in bold. In this example, the PID is 3093. Take note of this number for use in these steps.


      Caution: Ensure that you identify the line specific only to the virtual machine you are attempting to repair. If you continue this process for a virtual machine other than the one in question, you can cause downtime for the other virtual machine.

      If the .vmx process is listed, it is possible that the virtual machine has lost control of the process and that it must be stopped manually.

    2. To end the process, run the command:

      # kill <PID>

    3. Wait 30 seconds and check for the process again.
    4. If it is not terminated, run the command:

      # kill -9 <PID>

    5. Wait 30 seconds and check for the process again.
    6. Where possible, it may be necessary to migrate the virtual machine to another host and attempt the kill instruction from that host.
    7. If it is not terminated, the ESX host may need to be rebooted to clear the process. This is a last resort option, and should only be attempted if the preceding steps in this article are unsuccessful.


Additional Information

Tags

not-responding  stops-responding  vm-not-responding  vm-unresponsive

See Also

This Article Replaces

1000674, 1002459, 1487776, 1002350, 10060

Update History

11/02/2011 - Changed:vm-support -X command in the home directory of the virtual machine, to the root directory. 08/15/2012 - Added link for ESXi. 05/03/2013 - Added English language and changed misformatted fonts in resolution section.

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 184 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.
What can we do to improve this information? (4000 or fewer characters)
  • 184 Ratings
Actions
KB: