Using vSphere On-disk Metadata Analyzer (VOMA) to check VMFS metadata consistency
Article ID: 318894
Updated On:
Products
VMware vSphere ESXi
Issue/Introduction
This article provides instructions on performing a file system metadata check using the vSphere On-disk Metadata Analyzer (VOMA).
The VOMA can be used to check the following:
Symptoms:
Attempting to modify the partition table fails with the error:
Error: Read-only file system during write on /dev/disks/naa.600601603b503400de93e0a87e9ce211
SetPtableGpt: Unable to commit to disk
In the /var/log/vmkernel file, you see entries similar to:
vmkernel: 25:21:39:57.861 cpu15:1047)FS3: 130: <START termserv2-5160fe37.vswp>
vmkernel: 25:21:39:57.861 cpu15:1047)Lock [type 10c00001 offset 52076544 v 69, hb offset 4017152
vmkernel: gen 109, mode 1, owner 4a15b3a2-fd2f4020-3625-001a64353e5c mtime 3420]
vmkernel: 25:21:39:57.861 cpu15:1047)Addr <4, 1011, 10>, gen 36, links 1, type reg, flags 0x0, uid 0, gid 0, mode 600
vmkernel: 25:21:39:57.861 cpu15:1047)len 3221225472, nb 3072 tbz 0, zla 3, bs 1048576
vmkernel: 25:21:39:57.861 cpu15:1047)FS3: 132: <END termserv2-5160fe37.vswp>
vmkernel: 0:00:20:51.964 cpu3:1085)WARNING: Swap: vm 1086: 2268: Failed to open swap file '/volumes/4730e995-faa64138-6e6f-001a640a8998/mule/mule-560e1410.vswp': Invalid metadata
vmkernel: 0:00:20:51.964 cpu3:1085)WARNING: Swap: vm 1086: 3586: Failed to initialize swap file '/volumes/4730e995-faa64138-6e6f-001a640a8998/mule/mule-560e1410.vswp': Invalid metadata
cpu11:268057)WARNING: HBX: 599: Volume 50fd60a3-3aae1ae2-3347-0017a4770402 ("<Datastore_name>") may be damaged on disk. Corrupt heartbeat detected at offset 3305472: [HB state 0 offset 6052837899185946624 gen 15439450 stampUS 5 $
Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
The VOMA can be used to check the following:
- Partition table update
- Reports of metadata errors in the vmkernel.log file
- Unable to access files on the VMFS volume that are not in use by any other host
Symptoms:
- You have problems accessing certain files on a VMFS datastore.
- You cannot modify or erase files on a VMFS datastore.
- Attempting to read files on a VMFS datastore may fail with the error:
Attempting to modify the partition table fails with the error:
Error: Read-only file system during write on /dev/disks/naa.600601603b503400de93e0a87e9ce211
SetPtableGpt: Unable to commit to disk
In the /var/log/vmkernel file, you see entries similar to:
vmkernel: 25:21:39:57.861 cpu15:1047)FS3: 130: <START termserv2-5160fe37.vswp>
vmkernel: 25:21:39:57.861 cpu15:1047)Lock [type 10c00001 offset 52076544 v 69, hb offset 4017152
vmkernel: gen 109, mode 1, owner 4a15b3a2-fd2f4020-3625-001a64353e5c mtime 3420]
vmkernel: 25:21:39:57.861 cpu15:1047)Addr <4, 1011, 10>, gen 36, links 1, type reg, flags 0x0, uid 0, gid 0, mode 600
vmkernel: 25:21:39:57.861 cpu15:1047)len 3221225472, nb 3072 tbz 0, zla 3, bs 1048576
vmkernel: 25:21:39:57.861 cpu15:1047)FS3: 132: <END termserv2-5160fe37.vswp>
vmkernel: 0:00:20:51.964 cpu3:1085)WARNING: Swap: vm 1086: 2268: Failed to open swap file '/volumes/4730e995-faa64138-6e6f-001a640a8998/mule/mule-560e1410.vswp': Invalid metadata
vmkernel: 0:00:20:51.964 cpu3:1085)WARNING: Swap: vm 1086: 3586: Failed to initialize swap file '/volumes/4730e995-faa64138-6e6f-001a640a8998/mule/mule-560e1410.vswp': Invalid metadata
cpu11:268057)WARNING: HBX: 599: Volume 50fd60a3-3aae1ae2-3347-0017a4770402 ("<Datastore_name>") may be damaged on disk. Corrupt heartbeat detected at offset 3305472: [HB state 0 offset 6052837899185946624 gen 15439450 stampUS 5 $
Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
Environment
VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.0
VMware vSphere ESXi 5.5
VMware vSphere ESXi 7.0.0
VMware vSphere ESXi 5.1
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.0
VMware vSphere ESXi 5.5
VMware vSphere ESXi 7.0.0
VMware vSphere ESXi 5.1
Resolution
The vSphere On-disk Metadata Analyzer (VOMA) scans the VMFS volume metadata and highlights any inconsistencies to which you may be required to open a support request.
To perform a VOMA check on a VMFS datastore and send the results to a specific log file, the command syntax is:
voma -m vmfs -d /vmfs/devices/disks/naa.00000000000000000000000000 -s /tmp/analysis.txt
voma -m vmfs -d /vmfs/devices/disks/naa.00000000000000000000000000:1 -s /tmp/analysis.txt
Where: naa.00000000000000000000000000:1 is the LUN NAA ID and partition to be checked. For a VMFS6 multi extent volume, LUN NAA ID represents the head extent. Note the ":1" at the end of device. This is the partition number containing the datastore. Partition number is optional when the datastore resides on the first partition, i.e., VOMA cannot be run against the device if datastore resides on a partition other than the first partition.
Note:
Error: Missing LVM Magic. Disk doesn't have a valid LVM Device
Error: Failed to Initialize LVM Metadata
In circumstances where the corruption is irreversible, VMware recommends that you restore from backup or consult with a data recovery organization as VMware does not perform data recovery. For more information, see Data recovery services for data not recoverable by VMware Technical Support (1015413).
Before running VOMA you must ensure that:
VOMA command syntax options
To perform a VOMA check on a VMFS datastore and send the results to a specific log file, the command syntax is:
voma -m vmfs -d /vmfs/devices/disks/naa.00000000000000000000000000 -s /tmp/analysis.txt
voma -m vmfs -d /vmfs/devices/disks/naa.00000000000000000000000000:1 -s /tmp/analysis.txt
Where: naa.00000000000000000000000000:1 is the LUN NAA ID and partition to be checked. For a VMFS6 multi extent volume, LUN NAA ID represents the head extent. Note the ":1" at the end of device. This is the partition number containing the datastore. Partition number is optional when the datastore resides on the first partition, i.e., VOMA cannot be run against the device if datastore resides on a partition other than the first partition.
Note:
1. VOMA is not supported on VMFS5 multi extent volume.
2. On running VOMA against a device, where datastore does not reside on the first partition, it produces an error similar to:
2. On running VOMA against a device, where datastore does not reside on the first partition, it produces an error similar to:
Error: Missing LVM Magic. Disk doesn't have a valid LVM Device
Error: Failed to Initialize LVM Metadata
In circumstances where the corruption is irreversible, VMware recommends that you restore from backup or consult with a data recovery organization as VMware does not perform data recovery. For more information, see Data recovery services for data not recoverable by VMware Technical Support (1015413).
Before running VOMA you must ensure that:
- All virtual machines on the affected datastore are powered off or migrated to another datastore. See Impact/Risks section.
- For VMFS-5/VMFS-6 datastores, the datastore is unmounted on all ESXi 6.x/7.x hosts.
If you attempt to run VOMA against a volume that is still being accessed by one or more ESXi hosts, VOMA produces a message similar to:
voma -m vmfs -d /vmfs/devices/disks/naa.00000000000000000000000000:1
Checking if device is actively being used by other hosts
Found 1 actively heartbeating hosts on device '/vmfs/devices/disks/naa.00000000000000000000000000:1'
1): MAC Address xx:xx:xx:xx:xx:xx
voma -m vmfs -d /vmfs/devices/disks/naa.00000000000000000000000000:1
Checking if device is actively being used by other hosts
Found 1 actively heartbeating hosts on device '/vmfs/devices/disks/naa.00000000000000000000000000:1'
1): MAC Address xx:xx:xx:xx:xx:xx
VOMA command syntax options
| Option | Description |
| -m or --module | Specify whether the module to run is vmfs or lvm. When vmfs is specified, the lvm check is also performed. The default check is vmfs. |
| -d or --device | Device/disk to be inspected. |
| -s or --logfile | Used to specify the output log file. |
| -v or --version | Display the VOMA version. |
| -h or --help | Displays the VOMA command help message. |
Additional Information
VMware Skyline Health Diagnostics for vSphere - FAQ
Unable to access certain files on a VMFS datastore
Data recovery services for data not recoverable by VMware Technical Support
vSphere On-disk Metadata Analyzer（VOMA）を使用して、VMFS メタデータの整合性をチェックする
使用 vSphere On-disk Metadata Analyzer (VOMA) 检查 VMFS 元数据的一致性
Using VMware vSphere On-disk Metadata Analyzer to re-create missing partition tables on VMware ESXi
I/O access on Virtual Machine File System returns invalid metadata error
Impact/Risks:
Shutting down a virtual machine running on files having certain types of corrupt metadata by the vSphere On-disk Metadata Analyzer, may make the virtual machine and its data permanently unavailable. Because of this it is always advisable to have current backups of the virtual machines in the environment. If you suspect that the virtual machine may become unavailable, because for example, there are read/write errors in the guest OS, or the virtual machine is unresponsive, you should open a support request.
Unable to access certain files on a VMFS datastore
Data recovery services for data not recoverable by VMware Technical Support
vSphere On-disk Metadata Analyzer（VOMA）を使用して、VMFS メタデータの整合性をチェックする
使用 vSphere On-disk Metadata Analyzer (VOMA) 检查 VMFS 元数据的一致性
Using VMware vSphere On-disk Metadata Analyzer to re-create missing partition tables on VMware ESXi
I/O access on Virtual Machine File System returns invalid metadata error
Impact/Risks:
Shutting down a virtual machine running on files having certain types of corrupt metadata by the vSphere On-disk Metadata Analyzer, may make the virtual machine and its data permanently unavailable. Because of this it is always advisable to have current backups of the virtual machines in the environment. If you suspect that the virtual machine may become unavailable, because for example, there are read/write errors in the guest OS, or the virtual machine is unresponsive, you should open a support request.
Feedback
Yes
No
Powered by
