Search the VMware Knowledge Base (KB)
View by Article ID

Finding the lock owners of a VMDK or file on a VMFS datastore in VMware ESXi 5.5 P05 and later (2110152)

  • 24 Ratings

Symptoms

During operations in a VMware ESXi 5.5 P05 environment, you experience these issues:
  • You cannot perform any virtual machine operations on an ESXi host, such as power-on or vMotion.

  • The files used by the virtual machines running on the host are locked by other processes or hosts.

Purpose

In some VMware ESXi host failure scenarios, such as PSOD, SAN failures, power outage, the file(s) used by the virtual machines registered on that host are locked and unavailable to other processes or hosts. This prevents operations, such as start-up or vMotion, that change the state of virtual machine files, and require tracing the lock owners where possible. This article provides steps to trace lock owners in ESXi 5.5 P05 using a new tool, vmfsfilelockinfo.

Cause

This issue occurs when another VMware ESXi host locks a file which is in use or required by the host experiencing problems.

Resolution

This is a known issue affecting all current ESXi versions. In ESXi versions earlier than 5.5 P05 and in ESXi 6.0, there is a manual procedure to locate the host and virtual machine holding locks. In ESXi 5.5 P05, there is a script in the /bin directory which finds the identity of the host(s) holding a lock on files.

Note: This script is only available in ESXi 5.5 P05 and later.

To work around this issue, run the vmfsfilelockinfo script from the host experiencing difficulties with one or more locked files:

  1. To find out the IP address of the host holding the lock, run the /bin/vmfsfilelockinfo Python script. The script takes these parameters:

    • File being tested
    • Username and password for accessing VMware vCenter Server (when tracing MAC address to ESX host.)

      For example:

      Run this command:

      ~ # vmfsfilelockinfo -p /vmfs/volumes/iscsi-lefthand-2/VM1/VM1_1-000001-delta.vmdk -v 192.168.1.10 -u administrator@vsphere.local

      You see ouput similar to:

      vmfsflelockinfo Version 1.0

      Looking for lock owners on "VM1_1-000001-delta.vmdk"
      "VM1_1-000001-delta.vmdk" is locked in Exclusive mode by host having mac address ['xx:xx:xx:xx:xx:xx']
      Trying to make use of Fault Domain Manager
      ----------------------------------------------------------------------
      Found 0 ESX hosts using Fault Domain Manager.
      ----------------------------------------------------------------------
      Could not get information from Fault domain manager
      Connecting to 192.168.1.10 with user administrator@vsphere.local
      Password: xXxXxXxXxXx
      ----------------------------------------------------------------------
      Found 3 ESX hosts from Virtual Center Server.
      ----------------------------------------------------------------------
      Searching on Host 192.168.1.178
      Searching on Host 192.168.1.179
      Searching on Host 192.168.1.180
      MAC Address : xx:xx:xx:xx:xx:xx

      Host owning the lock on the vmdk is 192.168.1.180, lockMode : Exclusive

      Total time taken : 0.27 seconds.

      Note
      : During the life-cycle of a powered on virtual machine, several of its files transitions between various legitimate lock states. The lock state mode indicates the type of lock that is on the file. The list of lock modes is:

    • mode 0 = no lock
    • mode 1 = is an exclusive lock (vmx file of a powered on virtual machine, the currently used disk (flat or delta), *vswp, and so on.)
    • mode 2 = is a read-only lock (For example on the ..-flat.vmdk of a running virtual machine with snapshots)
    • mode 3 = is a multi-writer lock (For example used for MSCS clusters disks or FT VMs)


  2. To get the name of the process holding the lock, run the lsof command on the host holding the lock and filter the output for the file name in question:

    ~ # lsof | egrep 'Cartel|VM1_1-000001-delta.vmdk'

    You see output similar to:

    Cartel | World name | Type | fd | Description
    36202 vmx FILE 80 /vmfs/volumes/556ce175-7f7bed3f-eb72-000c2998c47d/VM1/VM1_1-000001-delta.vmdk


    This shows that the file is locked by a virtual machine having Cartel ID 36202. Now display the list of active Cartel IDs by executing this command:

    ~ # esxcli vm process list

    This displays information for active virtual machines grouped by virtual machine name and having a format similar to:

    Alternate_VM27
    World ID: 36205
    Process ID: 0
    VMX Cartel ID: 36202
    UUID: 56 4d bd a1 1d 10 98 0f-c1 41 85 ea a9 dc 9f bf
    Display Name: Alternate_VM27
    Config File: /vmfs/volumes/556ce175-7f7bed3f-eb72-000c2998c47d/Alternate_VM27/Alternate_VM27.vmx
    Alternate_VM20
    World ID: 36207
    Process ID: 0
    VMX Cartel ID: 36206
    UUID: 56 4d bd a1 1d 10 98 0f-c1 41 85 ea a5 dc 94 5f
    Display Name: Alternate_VM20
    Config File: /vmfs/volumes/556ce175-7f7bed3f-eb72-000c2998c47d/Alternate_VM20/Alternate_VM20.vmx
    ...

    The virtual machine entry having VMX Cartel ID 36202 shows the display name of the virtual machine holding the lock on file VM1_1-000001-delta.vmdk, which in this example, is Alternate_VM27.

  3. Shut down the virtual machine holding the lock to release the lock.

Additional Information

This script performs these actions in this sequence:
  1. Identifies locked state Exclusive, Read-Only, not locked.
  2. Identifies MAC address of locking host ['xx:xx:xx:xx:xx:xx'].
  3. Queries the Fault Domain Manager (HA) for information on discovered MAC address.
  4. Queries vCenter Server for information on discovered MAC address.
  5. Outputs final status.
    For example:

    Host owning the lock on the vmdk is 192.168.1.180, lockMode : Exclusive.

  6. The script outputs total execution time when it terminates.

Notes:

  • The script does not attempt to break/remove locks. The script only identifies the potential ESX host which holds the lock.
  • If not run with vCenter Server username and password, it prompts for the same, after querying the Fault Domain Manager.
  • This script works on a single file parameter, without wildcards. If multiple queries are required, you must execute the script repeatedly in a wrapper script.
For more information on investigating virtual machine files that are locked, see Investigating virtual machine file locks on ESXi/ESX (10051).

See Also

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 24 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.

What can we do to improve this information? (4000 or fewer characters)




Please enter the Captcha code before clicking Submit.
  • 24 Ratings
Actions
KB: