There are different scenarios on how a vSAN disk group (DG) will respond to a certain failure:

In case were "Deduplication and Compression" is enabled:

• Cache disk failure --> The whole disk group will be down.
• Capacity disk failure --> The whole disk group will be down.

In case were "Deduplication and Compression" is not enabled:

• Cache disk failure --> The whole disk group will be down.
• Capacity disk failure --> Only the failed disk will be down.

In this article, we will be addressing the following:

• How to identify if your DG is using deduplication and compression or not?

• How to identify the failed disks?

• How to verify the disk status?

• How to identify the physical location of the failed disks?

• How to respond?

• How to identify the time when the disk failed and if there are any SCSI errors reported on the failed disk?

• Common disk failure reasons.

• From the vCenter Web Client (Cluster --> Monitor --> vSAN --> Capacity), if deduplication is enabled, you will see "Deduplication and compression savings" ratio

• Or from Cluster > Configure > vSAN > Services > Data Services > Space efficiency
• Or from ESXi CLI, open a putty/ssh session to any host in the cluster and run the following command: 

esxcli vsan storage list | grep -i dedup
   Deduplication: true
   Deduplication: true
   Deduplication: true
   Deduplication: true
   Deduplication: true
   Deduplication: true

• In case deduplication & compression is not enabled, go to vCenter and check "Operation health" under "Skyline Health" [Cluster --> Monitor --> vSAN --> Skyline Health --> Physical disk --> Operation health]

• In case deduplication & compression is enabled, any disk failure will cause the whole DG will be offline. In this situation, ensure to scroll to the right to view the "Operational State Description" to see which disk is failed. The disk marked as Permanent disk failure is the failed disk.

• (Reference KB vSAN Deduplication enabled -- Identifying Failed Disk (2149067)).

• To confirm if the disk or disk group is currently mounted or still down, go to vCenter and check "Disk Management" [Cluster --> Configure --> vSAN --> Disk Management]

• Run the following command using the device identifier of the failed disk typically naa.xxxx, eui.xxx, or mpx.xxx

esxcli storage core device physical get -d naa.xxxxx
   Physical Location: enclosure 1, slot 0
 

• Or click on "TURN ON LED" for the failed disk [Connect to vCenter, click on the host with the failed disk --> Configure --> Storage devices --> Mark the failed disk -->  "TURN ON LED"]

• Follow KB vSAN Deduplication enabled -- Identifying Failed Disk (2149067)
• If for some reason the above KB doesn't help with identifying the failed disk you can identify the physical slot in this state by clicking on "TURN ON LED" for all the working disks or identify the physical slots (esxcli storage core device physical get -d "naa.xxxx") of all the working disks, so we can eliminate the physical location of all the working disks, hence, conclude the physical location of the failed disk.

• First, we need to make sure there is no inaccessible object. You can check it from the vCenter under "Virtual Objects" [Cluster --> Monitor --> vSAN --> Virtual Objects]

You can also check it from CLI using the following command: esxcli vsan debug object health summary get.
In case there are inaccessible objects, open a case with VMware vSAN support.
 

• Based on the cause of the disk failure, an activity might need to be done on the host (Example: reboot the host, replace/re-insert the failed disk, or recreate DG). As a best practice, place the host in maintenance mode with ensure accessibility before proceeding with any activity (Note: run the evacuation precheck, to understand if it will require to migrate data or not).

• Important note: If you placed the host in maintenance mode with ensure accessibility for more than the the default repair time of 60 minutes, vSAN will start a re-sync operation to rebuild the data residing on all DGs on that host. In case the activity will take more than the configured repair time, you can increase the configured repair time temporarily to avoid unneeded re-sync operation. (To change the repair time, please reference KB Changing the default repair delay time for a host failure in vSAN (2075456).

• Open SSH/Putty session to the ESXi host and run the following command to identify the time-stamp when the disk failed:

• Then run the following command to identify any read/write commands failing at the same time stamp collected from the previous step: 

Note: If the disk hasn't failed recently or if vmkernel is chatty you may need to look at older logs. To do this run zcat /var/run/log/vmkernel.*.gz|grep <disk device>

Common disk failure reasons:

(1) Disk is soft failed:
 
Troubleshooting steps:

• Check KVM (iDrac, iLo) for any issues with disks/controller
• Check the logs for any SCSI error codes
• Check controller driver/firmware to ensure they're not down rev or in an unsupported combination by running esxcli vsan debug controller list -v=true then check the vSAN HCL if the drivers/firmware are down rev or not in a supported combination then upgrade them.
• If no issues are found in the KVM or no SCSI error codes then this is a soft fail of a disk and reboot of the host may bring the disk(s)/DG back online place the host into maintenance mode with ensure accessibility, then reboot the host (If the disk(s)/DG comes back great, otherwise if it doesn't engage the hardware vendor for replacement/further assistance). 

(2) Hardware issue: Valid sense data: 0x4 0x0 0x0

Example of SCSI error from log file: /var/run/log/vmkernel.log:
2021-01-05T08:37:16.337Z cpu26:2098033)ScsiDeviceIO: 3047: Cmd(0x45a3e27a1700) 0x2a, CmdSN 0x2238d from world 2960707 to dev "naa.xxxx" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x0 0x0.

Troubleshooting steps:

• Disk needs to be replaced by the Hardware vendor.

(3) Medium error: Valid sense data: 0x3 0x11 0x0

• Unrecovered Read Error (URE) is a type of a medium error that occurs when the esxi host try to read from a bad block on the disk. For more information about URE, please reference the following document "Improving vSAN's Resilience against Unrecovered Read Errors on Devices".
• URE can occur in the metadata region or the data region.
• If URE occurs in the data region of the disk open case with VMware vSAN support for further assistance.
• If URE occurred in the metadata region, as of  ESXi/vSAN 6.7 P03 and 7.0 Update 1 and newer a feature called autoDG Creation was introduced for All Flash DG and vSAN Skyline Health reports that the disk is unhealthy and will reallocate the blocks marking the bad blocks for non-use. See KB vSAN Disk Or Diskgroup Fails With Medium Errors (81121) for more details.

Example of scsi error from log file : /var/run/log/vmkernel.log:
2022-10-12T19:36:55.253Z cpu11:2098330)ScsiDeviceIO: 4325: Cmd(0x45bea479ec40) 0x28, CmdSN 0xfaf from world 0 to dev "naa.xxxx" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x0

Troubleshooting steps:

In case "Hybrid vSAN" is used:

• Disks are HDDs, then the bad disk will need to be replaced by the Hardware vendor.

 
DDH feature in vSAN continuously monitors the health of disks and disk groups in order to detect an impending disk failure or a poorly performing disk group (For more information about DDH, please reference the following:  KB Dying Disk Handling (DDH) in vSAN (2148358)  and the following document "vSAN Degraded Device Handling").
DDH unmounts the disk or reports it unhealthy in the following situations :

• High write IO latency on one of the vSAN disks.
• Maximum log congestion threshold reached on one of the DG.
• IMPENDING FAILURE reported on one of the vSAN disks (We can see the health status of the disk using the following command : localcli storage core device smart get -d naa.xxx), 

• Check if the failed disk is facing any hardware or medium errors (reference the above steps).
• Run the following command: esxtop, option "u" on the host with the failed disk, and check the "DAVG" for the failed disk to see if there is any high latency reported on that disk. If there's high latency seen, engage the hardware vendor. For more information on how to check DAVG using esxtop, reference the following KB Using esxtop to identify storage performance issues for ESX / ESXi (multiple versions) (1008205).
• Check the compatibility of the controller driver and firmware, and also check if the disk is a vSAN supported device and if its firmware version is supported (Reference vSAN HCL Link).
• If there are no compatibility issues, engage the hardware vendor to check for any firmware issues on the controller or disks.

(5) Read/write commands failing with Aborts/RETRY: H:0x5 & H:0xc

Example from log file: /var/run/log/vmkernel.log:
2022-10-21T02:50:51.069Z cpu0:2098435)ScsiDeviceIO: 3501: Cmd(0x45a203564900) 0x28, cmdId.initiator=0x45223c91a7f0 CmdSN 0xaa97f from world 0 to dev "naa.xxxx" failed H:0x5 D:0x0 P:0x0 Aborted at driver layer. Cmd count Active:2 Queued:0

2022-10-21T04:41:13.494Z cpu0:2098435)ScsiDeviceIO: 3463: Cmd(0x45aa8ffdedc0) 0x28, CmdSN 0x2 from world 2102512 to dev "naa.xxxxx" failed H:0xc D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0.

Troubleshooting steps:

• Check the compatibility of the controller driver and firmware, and also check if the disk is a vSAN supported device and if its firmware version is supported (Reference vSAN HCL Link).
• If there are no compatibility issues, engage the hardware vendor to check for any firmware issues on the controller or disks.