Powering off an unresponsive virtual machine on an ESXi host

Products

VMware vSphere ESXi

Issue/Introduction

This article provides steps for shutting down and when necessary, force on an unresponsive virtual machine.

Symptoms:

You cannot power off virtual machine on an ESXi host
A virtual machine is unresponsive and cannot be forced off or stopped
You cannot access or unlock files on a virtual machine
After shutting down a virtual machine, vCenter Server shows the virtual machine as running
There is no indication that a virtual machine is shut down
You cannot edit properties in the virtual machine
You see one or more of these errors:
- Soap error 999. The operation is not allowed in current state
- The attempted operation cannot be performed in the current state (Powered Off)
- The request refers to an object that no longer exists or has never existed

Environment

VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.7
VMware vSphere ESXi 7.0.0

Resolution

If Powering off the virtual machine from vSphere Web Client or Host Client does not work , you must use the command line method.

Determining the virtual machine's location

Determine the host on which the virtual machine is running. This information is available in the virtual machine's Summary tab when viewed in vCenter web client.

There are several methods to power off the non-responsive virtual machine using command line as the following:

Using the ESXi command-line
Using ESXi esxcli commands

Warning: Follow the sections and steps in this article in order.

Using the ESXi command-line

Log in as root to the ESXi using SSH.
Get a list of all registered virtual machines, identified by their VMID and Display Name by running this command:
```
# vim-cmd vmsvc/getallvms
```

Example output:

Take note of the impacted virtual machine ID VMID
Get the current state of a virtual machine by running this command:
```
# vim-cmd vmsvc/power.getstate VMID
```

Example output:

Check if there hanging task on the impacted virtual machine that prevent the machine from powering on by running this command:
```
# vim-cmd vmsvc/get.tasklist VMID
```

Example output:

(ManagedObjectReference) [
'vim.Task:haTask-2-vim.VirtualMachine.createSnapshot-182550283',
'vim.Task:haTask-2-vim.VirtualMachine.consolidateDisks-182550274'
]

Run the vim-cmd vimsvc/task_info task_id command to see the task in more detail.

For example using the data from step 5:

$ vim-cmd vimsvc/task_info haTask-2-vim.VirtualMachine.createSnapshot-182550283

vim.TaskInfo) {
dynamicType = <unset>,
key = "haTask-2-vim.VirtualMachine.createSnapshot-182550283",
task = 'vim.Task:haTask-2-vim.VirtualMachine.createSnapshot-182550283',
description = (vmodl.LocalizableMessage) null,
name = "vim.VirtualMachine.createSnapshot",
descriptionId = "VirtualMachine.createSnapshot",
entity = 'vim.VirtualMachine:2',
entityName = "VMware vCenter Server Appliance5-1",
state = "running",
cancelled = false,
cancelable = false,
error = (vmodl.MethodFault) null,
result = <unset>,
progress = 50,
reason = (vim.TaskReasonUser) {
dynamicType = <unset>,
userName = "vpxuser",
},
queueTime = "2014-11-09T17:49:01.171933Z",
startTime = "2014-11-09T17:49:01.175521Z",
completeTime = <unset>,
eventChainId = 182550283,
changeTag = <unset>,
parentTaskKey = <unset>,
rootTaskKey = <unset>,
}

The task is hung and needs to be cancelled, use the command:
```
# vim-cmd vimsvc/task_cancel task_id
```

Note: Please note that not all tasks can be canceled. For instance, in the example above we see the snapshot task has the cancelable flag set to false (cancelable = false).

Shutdown the virtual machine using the VMID found in Step 3 and run this command:
```
# vim-cmd vmsvc/power.shutdown VMID
```

Note: If the virtual machine fails to shut down, run this command:

# vim-cmd vmsvc/power.off VMID

Example output:

Using the ESXi esxcli command

Log in as root to the ESXi using SSH.
Get a list of running virtual machines, identified by World ID and Display Name by running this command:
```
# esxcli vm process list
```

Example output:

Take a note of the world ID of the non-responsive virtual machine. For example in step 3 the World ID is 2750434.

Power off the virtual machine from the list by running one of these commands:

# esxcli vm process kill -t=soft -w=WorldID
  or
# esxcli vm process kill -t=hard -w=WorldID
  or
# esxcli vm process kill -t=force -w=WorldID

Note: These command provide different levels of force to stop the virtual machine:

Soft is the most graceful
Hard performs an immediate shutdown
Force should be used as a last resort

If the VM fails to stop even after running the "esxcli vm process kill" command, evacuate all other VMs off the ESXi host and restart the ESXi itself.