"Device 'Bootstrap' is not available" error powering on a virtual machine

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

Symptoms:

Powering on a virtual machine, you see the error:

Error An error was received from the ESX host while powering on VM vmname. Device 'Bootstrap' is not available.

In the hostd.log, you see similar to:

017-02-02T17:07:56.590Z info hostd[3F581B70] [Originator@6876 sub=Libs opID=AC471949-0000012E-d85e user=root] OBJLIB-FILEBE : FileBEOpen: can't open '/vmfs/volumes/39580756-532be087/vmpath/vm.vmdk' : Insufficient permission to access the file (458754)

Environment

VMware vCenter Server 7.0.x
VMware vCenter Server 6.5.x
VMware vSphere ESXi 6.0
VMware vCenter Server 6.0.x
VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.7
VMware vCenter Server Appliance 6.5.x
VMware vCenter Server Appliance 6.0.x
VMware vCenter Server 6.7.x
VMware vCenter Server Appliance 6.7.x

Cause

This issue can be caused by:

ESXi host that does not hold the lock on the VM files.
ESXi host has insufficient access to underlying storage
Bad data in the VM descriptor file (*.vmx)

Resolution

Rule out file lock problems:
1. Open an SSH session into the ESXi host where the virtual machine is registered to
2. Follow this link to find out which ESXi host is holding the lock on any of the files that compose that virtual machine: Investigating virtual machine file locks on ESXi hosts
3. Login to vCenter Server Web Client and set DRS to "Manual"
4. Migrate the VM to the host identified in previous steps
5. Power on VM
6. Set DRS back to previous the Automation Level
If there are no file lock issues found, or if you see insufficient permission to access the file in hostd.log, rule out datastore access problems:
1. For NFS storage, check the NFS array manager software, to find out whether the following options are enabled:

root_squash

all_squash

If so, these settings should be disabled. You may need your storage array vendor to assist with this
For other storage types:
1. Check that there are no errors on storage array manager
2. Check whether you see input/output errors in the vmware.log, or if you get those errors when trying to access the VM's directory
  1. If there are I/O errors, restore or rebuild VM onto a healthier datastore
3. Check that permissions to the host are adequate such as read/write is enabled
- May need to reset the affected LUN
If you see evidence of permission problems but can't find the cause you may need to consult with your storage array vendor

If there are no datastore access problems found, rule out bad data in the VM descriptor file (*.vmx)
1. Manually rebuild a fresh VM:
  1. Right-click the VM and select Edit Settings...
  2. Take note of all VM settings
  3. Create a new VM with the same settings
  4. Attach the affected VM's virtual disks to the new VM
2. Alternatively, you may follow Rebuilding the virtual machine's .vmx file from vmware.log which provides a script to automate rebuilding the *.vmx file

Additional Information

Unable to perform operations on a virtual machine with a locked disk
Failed to power on virtual machine

Impact/Risks:
Do not turn DRS off if you have a vCloud Director instance pointed to the vCenter Server you are working on.