• Powering on the virtual machine results in the power on task remaining at 95% indefinitely.
• Adding an existing virtual machine disk (VMDK) to a powered on virtual machine fails with the error:
  • Failed to add disk scsi0:1. Failed to power on scsi0:1
  • Cannot power on the virtual machine after deploying it from a template.
  • Powering on a virtual machine fails with an error:
  • Unable to open Swap File
  • Unable to access a file since it is locked
  • Unable to access a file filename since it is locked
  • Unable to access Virtual machine configuration
• In the /var/log/vmkernel log file, you see entries similar to:

• When opening a console to the virtual machine, you may receive the error:

• The virtual machine reports conflicting power states between vCenter Server and the ESXi host UI.
• Attempting to view or open the .vmx file using a text editor (ex: cat or vi), reports an error similar to:

The Purpose of File Locking

To prevent concurrent changes to critical virtual machine files and file systems, ESXi hosts establish locks on these files. In certain circumstances, these locks may not be released when the virtual machine is powered off. As such, the files cannot then be accessed by other ESXi hosts while locked and the virtual machine fails to power on.

Virtual machine files locked during runtime include:

• VMNAME.vswp
• DISKNAME-flat.vmdk
• DISKNAME-ITERATION-delta.vmdk
• VMNAME.vmx
• VMNAME.vmxf
• vmware.log

Initial Quick Test

• Set DRS in manual mode and attempt to power on the virtual machine on all hosts in the cluster. If the virtual machine powers on, that host help the lock.

Identifying the Locked File

1. Power on the virtual machine, this process should fail and display an error message. Make note of the this error.
2. Connect to the ESXi host the virtual machine is on with an SSH session
3. Find the IP address of the host holding the lock by running vmfsfilelockinfo on the VMDK flat, delta, or sesparse file for VMFS, or the .UUID.lck file for vSAN.

• File to be tested
• Username and password for accessing VMware vCenter Server (when tracing MAC address to ESX host.)

For example:

vmfsfilelockinfo -p /vmfs/volumes/iscsi-lefthand-2/VM1/VM1_1-000001-delta.vmdk -v 192.168.1.10 -u [email protected]

vmfsflelockinfo Version 1.0
Looking for lock owners on "VM1_1-000001-delta.vmdk"
"VM1_1-000001-delta.vmdk" is locked in Exclusive mode by host having mac address ['xx:xx:xx:xx:xx:xx']
Trying to make use of Fault Domain Manager
----------------------------------------------------------------------
Found 0 ESX hosts using Fault Domain Manager.
----------------------------------------------------------------------
Could not get information from Fault domain manager
Connecting to 192.168.1.10 with user [email protected]
Password: xXxXxXxXxXx
----------------------------------------------------------------------
Found 3 ESX hosts from Virtual Center Server.
----------------------------------------------------------------------
Searching on Host 192.168.1.178
Searching on Host 192.168.1.179
Searching on Host 192.168.1.180
MAC Address : xx:xx:xx:xx:xx:xx

Host owning the lock on the vmdk is 192.168.1.180, lockMode : Exclusive

Total time taken : 0.27 seconds.

Note: During the life-cycle of a powered on virtual machine, several of its files transitions between various legitimate lock states. The lock state mode indicates the type of lock that is on the file. The list of lock modes is:

• mode 0 = no lock
• mode 1 = is an exclusive lock (vmx file of a powered on virtual machine, the currently used disk (flat or delta), *vswp, and so on.)
• mode 2 = is a read-only lock (For example on the ..-flat.vmdk of a running virtual machine with snapshots)
• mode 3 = is a multi-writer lock (For example used for MSCS clusters disks or FT VMs)

4. To find the name of the process holding the lock, run lsof 'locked_file_name' on the host holding the lock and filter the output for the file name in question:

lsof | egrep 'Cartel|VM1_1-000001-delta.vmdk'

Cartel | World name | Type | fd | Description
36202 vmx FILE 80 /vmfs/volumes/556ce175-7f7bed3f-eb72-000c2998c47d/VM1/VM1_1-000001-delta.vmdk

This shows that the file is locked by a virtual machine having Cartel ID 36202.

5. Now display the list of active Cartel IDs by with this command:

esxcli vm process list

This displays information for active virtual machines grouped by virtual machine name and having a format similar to:

Alternate_VM27
World ID: 36205
Process ID: 0
VMX Cartel ID: 36202
UUID: 56 4d bd a1 1d 10 98 0f-c1 41 85 ea a9 dc 9f bf
Display Name: Alternate_VM27
Config File: /vmfs/volumes/556ce175-7f7bed3f-eb72-000c2998c47d/Alternate_VM27/Alternate_VM27.vmx
………

The virtual machine entry having VMX Cartel ID: 36202 shows the display name of the virtual machine holding the lock on file VM1_1-000001-delta.vmdk, which in this example, is Alternate_VM27.
 

6. If no processes are shown, the following script can search for VMs with the locked vmdk mounted.

for i in $(vim-cmd vmsvc/getallvms | grep -v Vmid | awk -F "/"  '{print $2}' | awk '{print $1}'); do echo $i && find ./ -iname $i | xargs grep vmdk | grep -Ei VMDKS_TO_LOOK_FOR ; done

Removing the Lock

1. Power off the virtual machine holding the lock
2. Reboot the ESXi host that holds the lock

Removing the .lck file (NFS only)

1. Power down the virtual machine and remove lck-#### files in the virtual machines directory.

Caution: Only remove the lck files of a powered off virtual machine

Note: VMFS volumes do not have lck files. The locking mechanism for VMFS volumes is handled within VMFS metadata on the volume.

Verify the integrity of the virtual machine configuration file (.vmx)

For more information on checking the integrity of the virtual machine configuration file, see Verifying ESX/ESXi virtual machine file integrity

For related information, see

• "Cannot open the disk" errors powering on a VM .
• "An error occurred while consolidating disks:" deleting one or more snapshots or consolidate disks
• Powering off an unresponsive virtual machine on an ESXi host

Opening a Support Request

If the problem persists after completing the steps in this article:

• Collecting diagnostic information for VMware products .
• Investigating virtual machine file locks on ESXi

• VMware Skyline Health Diagnostics for vSphere - FAQ
• Troubleshooting virtual machine power on task hangs
• Failed to power on virtual machine