Using esxtop to identify storage performance issues for ESX / ESXi (multiple versions)

Products

VMware vSphere ESXi

Issue/Introduction

This article provides information about esxtop and latency statistics that can be used when troubleshooting performance issues with SAN-connected storage (Fibre Channel or iSCSI).

Note: In ESXi 5.x and later, you may see messages indicating that performance has deteriorated. For more information, see Storage device performance deteriorated (2007236).

Environment

VMware ESXi 4.0.x Embedded
VMware vSphere ESXi 5.0
VMware ESX Server 3.5.x
VMware vSphere ESXi 5.1
VMware vSphere ESXi 6.5
VMware vSphere ESXi 5.5
VMware vSphere ESXi 6.0
VMware ESX 4.1.x
VMware ESX 4.0.x
VMware ESXi 4.1.x Embedded
VMware vSphere ESXi 6.7
VMware ESXi 4.0.x Installable
VMware vSphere ESXi 7.0.0
VMware ESXi 3.5.x Installable
VMware ESX Server 3.0.x
VMware ESXi 4.1.x Installable
VMware ESXi 3.5.x Embedded

Resolution

The interactive esxtop utility can be used to provide I/O metrics over various devices attached to a VMware ESX host.

Configuring monitoring using esxtop

To monitor storage performance per HBA:

Start esxtop by typing esxtop at the command line.
Press d to switch to disk view (HBA mode).
To view the entire Device name, press SHIFT + L and enter 36 in Change the name field size.
Press f to modify the fields that are displayed.
Press b, c, d, e, h, and j to toggle the fields and press Enter.
Press s and then 2 to alter the update time to every 2 seconds and press Enter.
See Analyzing esxtop columns for a description of relevant columns. For more information, see Interpreting esxtop Statistics.

Note: These options are available only in VMware ESX 3.5 and later.

To monitor storage performance on a per-LUN basis:

Start esxtop by typing esxtop from the command line.
Press u to switch to disk view (LUN mode).
Press f to modify the fields that are displayed.
Press b, c, f, and h to toggle the fields and press Enter.
Press s and then 2 to alter the update time to every 2 seconds and press Enter.
See Analyzing esxtop columns for a description of relevant columns. For more information, see Interpreting esxtop Statistics.

To increase the width of the device field in esxtop to show the complete naa id:

Start esxtop by typing esxtop at the command line.
Press u to switch to the disk device display.
Press L to change the name field size.

Note: Ensure to use uppercase L.
Enter the value 36 to display the complete naa identifier.

To monitor storage performance on a per-virtual machine basis:

Start esxtop by typing esxtop at the command line.
Type v to switch to disk view (virtual machine mode).
Press f to modify the fields that are displayed.
Press b, d, e, h, and j to toggle the fields and press Enter.
Press s and then 2 to alter the update time to every 2 seconds and press Enter.
See Analyzing esxtop columns for a description of relevant columns. For more information, see Interpreting esxtop Statistics.

Note: If the number of lines displayed exceeds the window size you can press 2 to select and navigate between the lines you wish to remove. Press 4 on a highlighted line to remove it.

Analyzing esxtop columns

Refer to this table for relevant columns and descriptions of these values:

Column	Description
CMDS/s	This is the total amount of commands per second and includes IOPS (Input/Output Operations Per Second) and other SCSI commands such as SCSI reservations, locks, vendor string requests, unit attention commands etc. being sent to or coming from the device or virtual machine being monitored. In most cases, CMDS/s = IOPS unless there are a lot of metadata operations (such as SCSI reservations)
DAVG/cmd	This is the average response time in milliseconds per command being sent to the device.
KAVG/cmd	This is the amount of time the command spends in the VMkernel.
GAVG/cmd	This is the response time as it is perceived by the guest operating system. This number is calculated with the formula: DAVG + KAVG = GAVG

These columns are for both reads and writes, whereas xAVG/rd is for reads and xAVG/wr is for writes. The combined value of these columns is the best way to monitor performance, but high read or write response time it may indicate that the read or write cache is disabled on the array. All arrays perform differently, however, DAVG/cmd, KAVG/cmd, and GAVG/cmd should not exceed more than 10 milliseconds (ms) for sustained periods of time.

Note: VMware ESX 3.0.x does not include direct functionality to monitor individual LUNs or virtual machines using esxtop. Inactive LUNs lower the average for DAVG/cmd, KAVG/cmd, and GAVG/cmd. These values are also visible from the vCenter Server performance charts. For more information, see the Performance Charts section in the Basic System Administration Guide.

If you experience high latency times, investigate current performance metrics and running configuration for the switches and the SAN targets. Check for errors or logging that may suggest a delay in operations being sent to, received, and acknowledged. This includes the array's ability to process I/O from a spindle count aspect, or the array's ability to handle the load presented to it.

If the response time increases to over 5000 ms (or 5 seconds), VMware ESX will time out the command and abort the operation. These events are logged; abort messages and other SCSI errors can be reviewed in these logs:

ESX 3.5 and 4.x – /var/log/vmkernel
ESXi 3.5 and 4.x – /var/log/messages
ESXi 5.x and later - /var/log/vmkernel.log

The type of storage logging you may see in these files depends on the configuration of the server. You can find the value of these options by navigating to Host > Configuration > Advanced Settings > SCSI > SCSI.Log* or SCSI.Print*.

Additional Information

VMware Skyline Health Diagnostics for vSphere - FAQ

Troubleshooting LUN connectivity issues on ESXi hosts
You can also collect performance snapshots using vm-support. For more information, see:

For translated versions of this article, see: