Understanding SCSI device/target NMP errors/conditions in ESX/ESXi 4.x and ESXi 5.x/6.0
search cancel

Understanding SCSI device/target NMP errors/conditions in ESX/ESXi 4.x and ESXi 5.x/6.0

book

Article ID: 338435

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

This article provides information about the SCSI device status messages that you may observe when reviewing NMP errors in the vmkernel log.

Symptoms:



Environment

VMware vSphere ESXi 6.0
VMware vSphere ESXi 5.1
VMware vSphere ESXi 5.5

Resolution

These are SCSI device/target status messages that are defined, updated, and maintained by t10.org.

VMK_SCSI_DEVICE_GOOD = 0x0

vmkernel: 116:03:44:19.039 cpu4:4100)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x4100020e0b00) to NMP device "sym.029010111831353837" failed on physical path "vmhba2:C0:T0:L152" H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

This status is returned when there is no error from the device or target side. This is when you will see if there is a status for Host or Plugin.

For more information on host side NMP errors, see Understanding SCSI host-side NMP errors/conditions in VMware ESX 4.x (1029039).

VMK_SCSI_DEVICE_CHECK_CONDITION = 0x2

vmkernel: 76:23:45:36.239 cpu11:22687)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x41000b10f000) to NMP device "naa.600601601f70190016361881f3b4de11" failed on physical path "vmhba3:C0:T2:L10" H:0x0 D:0x2 P:0x0 Valid sense data: 0x2 0x4 0x3.

This status is returned when a command fails for a specific reason. When a CHECK CONDITION is received, the ESX storage stack will send out a SCSI command 0x3 (REQUEST SENSE) in order to get the SCSI sense data (Sense Key, Additional Sense Code, ASC Qualifier, and other bits). The sense data is listed after Valid sense data in the order of Sense Key, Additional Sense Code, and ASC Qualifier. When decoding Additional Sense Code/ASC Qualifier bits, they must be read together. Here is a breakdown of the above message:

Host Status = 0x0 = OK
Device Status = 0x2 = Check Condition
Plugin Status = 0x0 = OK

Sense Key = 0x2 = NOT READY
Additional Sense Code/ASC Qualifier = 0x4/0x3 = LOGICAL UNIT NOT READY - MANUAL INTERVENTION REQUIRED
For a complete list of possible Sense Keys, see SCSI Sense Keys
For a complete list of Additional Sense Code/ASC Qualifier pairings, see ASC-NUM.TXT


VMK_SCSI_DEVICE_CONDITION_MET = 0x4

vmkernel: 0:00:42:28.062 cpu5:4112)ScsiDeviceIO: SCSIAsyncDeviceCommand:2077: Error Inject: Command 0x2a to device "naa.600601600f6021002cfb75ea2bc6de11" failed H:0x0 D:0x4 P:0x0

This status is returned for successful completion of conditional commands, such as PREFETCH.

VMK_SCSI_DEVICE_BUSY = 0x8

vmkernel: 1:02:02:02.206 cpu3:4099)NMP: nmp_CompleteCommandForPath: Command 0x28 (0x410005078e00) to NMP device "naa.6001e4f000105e6b00001f14499bfead" failed on physical path "vmhba1:C0:T0:L100" H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0.

This status is returned when a LUN cannot accept SCSI commands at the moment. As this should be a temporary condition, the command is tried again.

VMK_SCSI_DEVICE_INTERMEDIATE = 0x10

Never seen in ESX.
This status is the same as 0x0 (GOOD) and is returned for a command that was part of a series of linked commands.

VMK_SCSI_DEVICE_INTERMEDIATE_CONDITION_MET = 0x14

Never seen in ESX.

This status is the combination of device statuses CONDITION MET (0x4) and INTERMEDIATE (0x10).

VMK_SCSI_DEVICE_RESERVATION_CONFLICT = 0x18

vmkernel: 1:08:40:03.933 cpu5:4736)ScsiDeviceToken: 115: Completed IO with status H:0x0 D:0x18 P:0x0 after losingreservation on device naa.6006016026601d007c174a7aa292df11

This status is returned when a LUN is in a Reserved status and commands from initiators that did not place that SCSI reservation attempt to issue commands to it. Rarely will you ever see a device status of 0x18 in the format shown above. The most common form of a RESERVATION CONFLICT status message is seen as follows:

vmkernel: 12:17:46:05.000 cpu12:4108)ScsiCore: 1181: Sync CR (opcode 28) at 992 (wid 4121)

* The opcode or SCSI Command Operation Code referenced in this message is 28 or 0x28. This is a READ(10) command or a 10 byte READ, which is a command to be issued by a virtual machine.

For more information on SCSI Command Operation Codes, see OP-NUM.TXT

For more information on troubleshooting SCSI Reservation conflicts, see Analyzing SCSI Reservation conflicts on VMware Infrastructure 3.x and vSphere 4.x (1005009).

VMK_SCSI_DEVICE_COMMAND_TERMINATED = 0x22

Never seen in ESX.

Obsolete status code. Was originally returned as a result of a TERMINATE I/O message. That message is also obsolete.

VMK_SCSI_DEVICE_QUEUE_FULL (TASK SET FULL) = 0x28

vmkernel: 1:08:42:28.062 cpu3:8374)NMP: nmp_CompleteCommandForPath:2190: Command 0x16 (0x41047faed080) to NMP device "naa.600508b40006c1700001200000080000" failed on physical path "vmhba39:C0:T1:L16" H:0x0 D:0x28 P:0x0 Possible sense data: 0x0 0x0 0x0.

This status is returned when the LUN prevents accepting SCSI commands from initiators due to lack of resources, namely the queue depth on the array.

Adaptive queue depth code was introduced into ESX 3.5 U4 (native in ESX 4.x) that adjusts the LUN queue depth in the VMkernel. If configured, this code will activate when device status TASK SET FULL (0x28) is return for failed commands and essentially throttles back the I/O until the array stops returning this status.

For more information, see Controlling LUN queue depth throttling in VMware ESX/ESXi (1008113).

VMK_SCSI_DEVICE_ACA_ACTIVE = 0x30

vmkernel: 0:00:14:08.673 cpu2:4098)NMP: nmp_CompleteCommandForPath: Command 0x25 (0x41013f5b4a40) to NMP device "naa.60060160f1c01400d2731d49473edd11" failed on physical path "vmhba2:C0:T1:L0" H:0x0 D:0x30 P:0x0 Possible sense data: 0x0 0x0 0x0.

This status is returned when an Auto Contingent Allegiance (ACA) has been aborted by another SCSI initiator. This status has been observed on arrays running older/unsupported firmware with ESX 4.1, specifically because the firmware does not support VAAI commands.

For information on VAAI, see vStorage APIs for Array Integration FAQ (1021976).

VMK_SCSI_DEVICE_TASK_ABORTED = 0x40

vmkernel: 1:01:32:19.519 cpu6:4102)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x4100062e7d00) to NMP device "naa.60a9800043346534645a43396a6c524d" failed on physical path "vmhba3:C0:T0:L23" H:0x0 D:0x40 P:0x0 Possible sense data: 0x0 0x0 0x0.

This status is returned when a command has been aborted by another SCSI initiator and the TASK ABORT STATUS bit is set to '1'. So far this status has only been returned by iSCSI arrays.

Any device status value not covered in this list would be vendor specific. If a vendor specific condition is hit, please contact the vendor support team for clarification on the status.


Additional Information

Note: The information provided in this article is provided as is and VMware does not guarantee the accuracy or applicability of this information. The preceding links were correct as of February 23, 2015. If you find the links are broken, please provide feedback and a VMware employee will update the links.

To be alerted when this document is updated, click the Subscribe to Article link in the Actions box
Analyzing SCSI Reservation conflicts on VMware Infrastructure 3.x, vSphere 4.x, vSphere 5.x and vSphere 6.0
Controlling LUN queue depth throttling in VMware ESX/ESXi
Frequently Asked Questions for vStorage APIs for Array Integration
了解 ESX/ESXi 4.x 和 ESXi 5.x/6.0 中的 SCSI 设备/目标 NMP 错误/状况
ESX/ESXi 4.x および ESXi 5.x における SCSI デバイス/ターゲットの NMP エラー/状態について
Interpreting SCSI sense codes in VMware ESXi and ESX