Large-scale workloads with intensive I/O patterns might require queue depths significantly greater than Paravirtual SCSI default values

Products

VMware vSphere ESXi

Issue/Introduction

The large-scale workloads with intensive I/O patterns require adapter queue depths greater than the Paravirtual SCSI (PVSCSI) default values. Current PVSCSI queue depth default values are 64 (for device) and 254 (for adapter). You can increase PVSCSI queue depths to 254 (for device) and 1024 (for adapter) inside a Windows virtual machine or Linux virtual machine.

Environment

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.0.x
VMware vSphere ESXi 6.7
VMware vSphere ESXi 5.5
VMware vSphere ESXi 5.1
VMware vSphere ESXi 6.5

Resolution

To resolve this issue, increase the number of pages used by the PVSCSI adapter for the request ring from the default value of 8 pages to 32 pages and set the PVSCSI device queue depth to 254:

Step 1: Adjust the queue depth for the HBAs on the ESXi host on which the Windows virtual machine runs. Complete these steps for every host to which the virtual machine is migrated:

Verify which HBA module is currently loaded by running the appropriate command on the service console:
- For QLogic:
  # esxcli system module list | grep qla
- For Emulex:
  # esxcli system module list | grep lpfc
Set the queue depth to 128 by running one of these commands:
- For QLogic:
  # esxcli system module parameters set -p ql2xmaxqdepth=128 -m qla2xxx
- For Emulex:
  
  If all Emulex cards on the host need to be updated, apply the global parameter, lpfc_lun_queue_depth.
  # esxcli system module parameters set -p lpfc0_lun_queue_depth=128 -m lpfc820
Note: The above examples show the QLogic qla2xxx and Emulex lpfc820 modules. Use the appropriate module based on the result of the previous step.

Also, consider these:
- In the above commands, both ql2xmaxqdepth and lpfc0 use the lowercase letter L, and not the numeral 1.
- The HBAs represented by ql2x and lpfc0 have their LUN queue depths set to 128.
Reboot your host.
Run this command to verify your changes:

# esxcli system module parameters list -m driver

Where driverrepresents the QLogic or Emulex adapter driver module, such as lpfc820 or qla2xxx.

You see output similar to:

Name Type Value Description -------------------------- ---- ----- -------------------------------------------------- ..... ql2xmaxqdepth int 128 Maximum queue depth to report for target devices. .....

Step 2: Increase the PVSCSI queue depth inside the Windows or Linux virtual machine, as described in the following sections. This modification increases the number of pages used by the PVSCSI adapter for the request ring to 32. It also increases the queue depth of the devices attached to the PVSCSI adapter to 254.

Note: The default number of ring pages is 8, each of which is 4 KB in size. One entry of 128 bytes is needed for a queued I/O, which means that 32 entries on a single page of 4096 bytes, resulting in 256 entries (8x32). The Windows PVSCSI driver adapter queue was hard coded in previous releases, but it can be adjusted up to the maximum of 32 pages since the versions delivered with VMware Tools in ESXi 5.5, 5.1 U1, and 5.0 U3. The Linux PVSCSI driver's number of ring pages can be a adjusted with versions prior to the Windows ones.

For a Windows virtual machine:

From the command line of your Windows virtual machine, run this command:

REG ADD HKLM\SYSTEM\CurrentControlSet\services\pvscsi\Parameters\Device /v DriverParameter /t REG_SZ /d "RequestRingPages=32,MaxQueueDepth=254"
Reboot the virtual machine
To verify the successful creation of registry entry:
- Open the registry editor by running the REGEDIT command from the command line.
- Browse to HKLM\ SYSTEM\CurrentControlSet\services\pvscsi\Parameters\Device.
- Verify that the DriverParameter key exists with a value of RequestRingPages=32, MaxQueueDepth=254.

For a Linux virtual machine:

Create a file of any name in the /etc/modprobe.d/ directory with this line:

options vmw_pvscsi cmd_per_lun=254 ring_pages=32

Note: For RHEL5, edit /etc/modprobe.conf with the same line. Make a new initrd for the settings to take effect. You can do this either by using mkinitrd, or by re-running vmware-config-tools.pl. Starting in version 6, RHEL uses modprobe.d.
Alternatively, append these to kernel boot arguments (for example, on Red Hat Enterprise Linux edit /etc/grub.conf or on Ubuntu edit /boot/grub/grub.cfg).

vmw_pvscsi.cmd_per_lun=254
vmw_pvscsi.ring_pages=32
Reboot the virtual machine.
Verify the changed queue depth by using cat on those files:

$ cat /sys/module/vmw_pvscsi/parameters/cmd_per_lun
$ cat /sys/module/vmw_pvscsi/parameters/ring_pages

Additional Information

For translated versions of this article, see: