Search the VMware Knowledge Base (KB)
View by Article ID

Manipulating a file on a Datastore fails with the error: device or resource busy / file is locked (1008728)

  • 18 Ratings

Symptoms

When attempting to manipulate a file on a datastore:

  • Manipulating (move, copy, or delete) a file fails
  • You receive the error:

    device or resource busy

  • You see that the file is locked

Resolution

You receive this error when the file you are manipulating is locked by another process or task. When the other process is identified , you must wait for the process to complete or stop it.

Warning: Stopping storage related processes abruptly may lead to data corruption.
 
To identify processes that are locking files:
  1. Run the command:

    vmkfstools -D path

    where path is the full path to the file you are attempting to manipulate.

    Note: This command does not generate output to the console. The output is sent to vmkernel logs.

  2. View the output of vmkfstools:
    • For ESX hosts, run the command:

      tail /var/log/vmkernel
       
    • For ESXi hosts, run the command:

      tail /var/log/messages

       
  3. Record the MAC address in the output of vmkfstools. For example:

    vmkernel: 11:04:31:50.251 cpu1:1035)FS3: 130: <START rdmtest-flat.vmdk>                
    vmkernel: 11:04:31:50.251 cpu1:1035)Lock [type 10c00001 offset 60088320 v 160, hb offset 3646464               
    vmkernel: gen 80, mode 1, owner 479f5a1a-0855d3ff-7288-0017a446ae31 mtime 532684]               
    vmkernel: 11:04:31:50.251 cpu1:1035)Addr <4, 130, 156>, gen 157, links 1, type reg, flags 0x0, uid 0, gid 0, mode 100600               
    vmkernel: 11:04:31:50.251 cpu1:1035)len 16106127360, nb 15360 tbz 0, zla 3, bs 1048576                
    vmkernel: 11:04:31:50.251 cpu1:1035)FS3: 132: <END rdmtest-flat.vmdk>  


    Note: The MAC address 00:17:A4:46:AE:31 is represented as 0017a446ae31.


     
  4. Determine which host has created the lock by comparing the MAC address from the output of vmkfstools with the MAC address from the management interface of all other ESX or ESXi hosts.
    • ESX/ESXi lists the MAC address of the service console interface with the commands:
      • # esxcfg-info -y | grep  "System UUID"

        For example:

        # esxcfg-info -y | grep  "System UUID"
           |----System UUID.................................................4b97b79b-dda7-eede-
        0017a446ae31

      • # grep uuid /etc/vmware/esx.conf

        For example
        :

        # grep uuid /etc/vmware/esx.conf
        /system/uuid = "4b97b79b-dda7-eede-617f-0017a446ae31

    • For ESX and VMware Infrastructure (VI) Client, select the host and click Configuration tab > Networking > Properties... >  Ports tab > Service Console. The MAC address is on the right.
    • For ESXi and VI Client, select the host and click Configuration tab > Networking > Properties... >  Ports tab > Management Network. The MAC address is on the right.
    • For verifying if the existing NICs having the lock and need to find out the MAC address of the NICs use:

      # esxcfg-nics -l | grep -i "xx:xx:xx:xx :xx:xx"

  5. Run this command to verify that there are no virtual machines that could be using the file in question:
     
    # find -H /vmfs/volumes/*/*/*.vmx  -print0 | xargs -0 grep "filename"

    Note: In ESXi remove the option "-H". Doing so you would get 2 lines for every resulting vmx. One with the Datastore easy name and another with the Datastore UUID.

    For e
    xample:

    # find /vmfs/volumes/*/*/*.vmx -print0 | xargs -0 grep "ag-test.vmdk"
    /vmfs/volumes/7e2d0213-7530bdba/ag-test/ag-test.vmx:scsi0:0.fileName = "ag-test.vmdk"
    /vmfs/volumes/dc03-nfs/ag-test/ag-test.vmx:scsi0:0.fileName = "ag-test.vmdk"

    This lists all instances of filename within the configuration files of all virtual machines on all the Datastores.


     
  6. For ESX hosts, run the following command to verify that there are no service console processes accessing the file:

    lsof | grep -i filename

      
  7. Verify that all processes accessing the file complete or are stopped.

  8. Verify that long running processes accessing the file have not timed out. For more information, see vCenter operation times out with the error: Operation failed since another task is in progress (1004790).

If these steps do not identify the process that has locked the file, reboot your host or virtual machine.

Warning: Rebooting a host during a storage operation may result in data loss.

See Also

Update History

05/13/2013 - Added ESXi 5.x to Products

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 18 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.

What can we do to improve this information? (4000 or fewer characters)




Please enter the Captcha code before clicking Submit.
  • 18 Ratings
Actions
KB: