Moving vSAN VMDKS from one Namespace Folder to another does not update the Object Path Attribute
search cancel

Moving vSAN VMDKS from one Namespace Folder to another does not update the Object Path Attribute

book

Article ID: 315539

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

The purpose of this article is to provide a detailed example of how this behavior can occur, along with corrective measures available using ESXi's vSAN Object Tool, or objtool.

Symptoms:
The output of esxcli vsan debug object list can give VMDKS that have been moved between Namespaces the appearance of being (Missing), thus making them seem Orphaned or Unassociated.

For example:

[root@vSAN-1:~] esxcli vsan debug object list
Object UUID: 19b10b5d-48fd-733d-a2f0-005056263753
   Version: 7
   Health: healthy
   Owner: vSAN-1.gsslabs.org
   Size: 10.00 GB
   Used: 0.01 GB
   Policy:
      CSN: 2
      stripeWidth: 1
      cacheReservation: 0
      proportionalCapacity: 0
      spbmProfileName: vSAN Default Storage Policy
      forceProvisioning: 0
      spbmProfileId: aa6d5a82-1c88-45da-85d3-3d74b91a5bad
      hostFailuresToTolerate: 1
      spbmProfileGenerationNumber: 0

   Type: vdisk
   Path: /vmfs/volumes/vsan:529e00851e70c3b9-215f67dee5e0391b/10b10b5d-3071-b7cb-a2be-005056263753/Virtual Machine 1.vmdk (Missing)
   Group UUID: 10b10b5d-3071-b7cb-a2be-005056263753
   Directory Name: N/A

Environment

VMware vSAN 6.x
VMware vSAN 7.0.x
VMware vSAN 5.5.x
VMware vSAN 8.0.x

Cause

When moving vSAN VMDK Descriptor Files from one Namespace directory to a different Namespace directory, the Object Path Attribute is not automatically updated.

By default, vSAN does not move VMDK Descriptor Files from one Namespace to a different Namespace.

This is expected product behavior. The Object Path Attribute of vSAN Objects must be updated explicitly with no automated methods currently available.

Resolution

Currently there is no mechanism to prevent this behavior from happening as vSAN does not move VMDK Descriptor files between Namespace Directories. However, the need to move files between Namespaces should be a relatively rare scenario.

Workaround:
See step 5 in the Related Information section.

Additional Information

This behavior can be reproduced when VMDK Descriptor files are moved manually using the Datastore Browser in the vSphere Client UI, when using "mv" commands in ESXi's command line interface, and when VMDKS are moved using vSphere API's such as MoveDatastoreFile_Task. 

In the example scenario below, we'll walk through the process of reproducing and correcting this behavior with two example Virtual Machines: "Virtual Machine 1" & "Virtual Machine 2": 

"Virtual Machine 1" Path: /vmfs/volumes/vsan:529e00851e70c3b9-215f67dee5e0391b/10b10b5d-3071-b7cb-a2be-005056263753

"Virtual Machine 2" Path: /vmfs/volumes/vsan:529e00851e70c3b9-215f67dee5e0391b/6fb10b5d-5bd1-f2aa-219a-005056263753


1. First, we can obtain the Object UUID of "Virtual Machine 1.vmdk" by simply cat'ing the VMDK Descriptor file - 

[root@vSAN-1:~] cat "/vmfs/volumes/vsan:529e00851e70c3b9-215f67dee5e0391b/10b10b5d-3071-b7cb-a2be-005056263753/Virtual Machine 1.vmdk"
# Disk DescriptorFile
version=4
encoding="UTF-8"
CID=fffffffe
parentCID=ffffffff
createType="vmfs"

# Extent description
RW 20971520 VMFS "vsan://19b10b5d-48fd-733d-a2f0-005056263753"

# The Disk Data Base
#DDB

ddb.adapterType = "lsilogic"
ddb.geometry.cylinders = "1305"
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
ddb.longContentID = "0046fa2bcb13ce19e6ba05cffffffffe"
ddb.thinProvisioned = "1"
ddb.uuid = "60 00 C2 92 46 38 6d 04-ae 07 eb 8b a2 fe af 7a"
ddb.virtualHWVersion = "14"



2. Next, we can move "Virtual Machine 1.vmdk" to Virtual Machine 2's Directory using a "mv" command: 

[root@vSAN-1:~] mv "/vmfs/volumes/vsanDatastore/10b10b5d-3071-b7cb-a2be-005056263753/Virtual Machine 1.vmdk" "/vmfs/volumes/vsanDatastore/6fb10b5d-5bd1-f2aa-219a-005056263753"


3. Using the vSphere Client UI, we can attach "Virtual Machine 1.vmdk" to Virtual Machine 2: 






4. Reviewing the output of esxcli vsan debug object list, we can now see that the VMDK Descriptor is reported as (Missing), even though it still exists:

[root@vSAN-1:~] esxcli vsan debug object list
Object UUID: 19b10b5d-48fd-733d-a2f0-005056263753
   Version: 7
   Health: healthy
   Owner: vSAN-1.gsslabs.org
   Size: 10.00 GB
   Used: 0.01 GB
   Policy:
      CSN: 2
      stripeWidth: 1
      cacheReservation: 0
      proportionalCapacity: 0
      spbmProfileName: vSAN Default Storage Policy
      forceProvisioning: 0
      spbmProfileId: aa6d5a82-1c88-45da-85d3-3d74b91a5bad
      hostFailuresToTolerate: 1
      spbmProfileGenerationNumber: 0

   Type: vdisk
   Path: /vmfs/volumes/vsan:529e00851e70c3b9-215f67dee5e0391b/10b10b5d-3071-b7cb-a2be-005056263753/Virtual Machine 1.vmdk (Missing)
   Group UUID: 10b10b5d-3071-b7cb-a2be-005056263753
   Directory Name: N/A


This is because the Path Attribute for this Object still references the previous directory from Virtual Machine 1. This can be confirmed via an "objtool getAttr" (Object Tool Get Attribute) command: 

* Command Syntax: /usr/lib/vmware/osfs/bin/objtool getAttr -u <Object UUID>

[root@vSAN-1:~] /usr/lib/vmware/osfs/bin/objtool getAttr -u 19b10b5d-48fd-733d-a2f0-005056263753
Object Attributes --

UUID:19b10b5d-48fd-733d-a2f0-005056263753

Object type:vsan

Object size:10737418240

User friendly name:(null)

HA metadata:(null)

Allocation type:Thin

Policy:((\"stripeWidth\" i1) (\"cacheReservation\" i0) (\"proportionalCapacity\" i0) (\"hostFailuresToTolerate\" i1) (\"forceProvisioning\" i0) (\"spbmProfileId\" \"aa6d5a82-1c88-45da-85d3-3d74b91a5bad\") (\"spbmProfileGenerationNumber\" l+0) (\"spbmProfileName\" \"vSAN Default Storage Policy\"))

Object class: vdisk

Object capabilities: NONE

Object path: /vmfs/volumes/vsan:529e00851e70c3b9-215f67dee5e0391b/10b10b5d-3071-b7cb-a2be-005056263753/Virtual Machine 1.vmdk

Group uuid: 10b10b5d-3071-b7cb-a2be-005056263753

Container uuid: (null)


5. We must update the Path Attribute using an "objtool setAttr" (Object Tool Set Attribute) command. 

* Command Syntax: /usr/lib/vmware/osfs/bin/objtool setAttr -u <Object UUID> -d <Path to VMDK>

[root@vSAN-1:~] /usr/lib/vmware/osfs/bin/objtool setAttr -u 19b10b5d-48fd-733d-a2f0-005056263753 -d "/vmfs/volumes/vsan:529e00851e70c3b9-215f67dee5e0391b/6fb10b5d-5bd1-f2aa-219a-005056263753/Virtual Machine 1.vmdk"
Object set attribute succeeded


Re-running esxcli vsan debug object list, we can now confirm the VMDK Descriptor File is no longer reported as (Missing):

[root@vSAN-1:~] esxcli vsan debug object list
Object UUID: 19b10b5d-48fd-733d-a2f0-005056263753
   Version: 7
   Health: healthy
   Owner: vSAN-1.gsslabs.org
   Size: 10.00 GB
   Used: 0.01 GB
   Policy:
      stripeWidth: 1
      spbmProfileGenerationNumber: 0
      cacheReservation: 0
      spbmProfileName: vSAN Default Storage Policy
      forceProvisioning: 0
      proportionalCapacity: 0
      spbmProfileId: aa6d5a82-1c88-45da-85d3-3d74b91a5bad
      hostFailuresToTolerate: 1
      CSN: 2

   Type: vdisk
   Path: /vmfs/volumes/vsan:529e00851e70c3b9-215f67dee5e0391b/6fb10b5d-5bd1-f2aa-219a-005056263753/Virtual Machine 1.vmdk (Exists)
   Group UUID: 10b10b5d-3071-b7cb-a2be-005056263753
   Directory Name: N/A

NOTE: When using  "esxcli vsan debug object list" in certain vSAN versions it shows only 100 Objects and it may happen that you don't see the one you are looking for. You will need to add the flag "--all" however, this could impact the host functionality so, it is recommended to run the command and send the output to a txt file. Here it is "esxcli vsan debug object list --all > /tmp/objects.txt". Then you can "less" the file to search for your specific object.
If the vSAN DOM Object UUID is known you can use the following syntax: esxcli vsan debug object list -u <DOM Object UUID>