Knowledge Base

The VMware Knowledge Base provides support solutions, error messages and troubleshooting guides
 
Search the VMware Knowledge Base (KB)   View by Article ID
 

Changing the queue depth for QLogic, Emulex and Brocade HBAs (1267)

Details

If the performance of your hardware bus adapters (HBAs) is unsatisfactory, or your SAN storage processors or heads are over-utilized, you can adjust your ESXi/ESX hosts' maximum queue depth value. The maximum value refers to the queue depths reported for various paths to the LUN. When you lower this value, it throttles the ESXi/ESX host's throughput and alleviates SAN contention concerns if multiple hosts are over-utilizing the storage and are filling its command queue.

When one virtual machine is active on a LUN, you only need to set the maximum queue depth. When multiple virtual machines are active on a LUN, the Disk.SchedNumReqOutstanding value is also relevant. The queue depth value, in this case, is equal to whichever value is the lowest of the two settings: adapter queue depth or Disk.SchedNumReqOutstanding.

Note:
The per-host parameter Disk.SchedNumReqOutstanding is deprecated in vSphere 5.5 to accommodate modification on a per-LUN basis. For more information, see Setting the Maximum Outstanding Disk Requests for virtual machines (1268).
Note: The preceding links were correct as of November 14, 2013. If you find a link is broken, provide feedback and a VMware employee will update the link.

Solution

This table lists the default Queue Depth values for QLogic HBAs for various ESXi/ESX versions:

ESXi/ESX version Queue depth
3.5 with driver version 6.04 16
3.5 with driver version 6.07 & 7.x 32
4.0 32
4.1 32
5.0 64
5.1 64
5.5 64

The default Queue Depth value for Emulex adapters has not changed for all versions of ESXi/ESX released to date. The Queue Depth is 32 by default, and because 2 buffers are reserved, 30 are available for I/O data.

The default Queue Depth value for Brocade adapters is 32.

Notes:
  • VMware recommends keeping the HBA Vendors uniform across all hosts participating in a cluster. If you are using a mixed environment (for example, with QLogic and Emulex), VMware recommends keeping the Queue Depth value the same across all the hosts in the cluster.
  • If you are applying a host profile which includes Queue Depth settings and a host reboot is required after application, note that queue depth values can only be modified at boot time. Reboot the host and confirm that the Queue Depth values are set correctly. The capability to modify Queue Depth variables is reliant on a compatible driver, and some async drivers may not allow the Queue Depth to be set correctly.

ESXi 5.0, 5.1 and 5.5

To adjust the queue depth for an HBA:

  1. Verify which HBA module is currently loaded by entering one of these commands on the service console:

    • For QLogic:

      # esxcli system module list | grep qla

      For ESXi 5.5 QLogic native drivers:

      # esxcli system module list | grep qln

    • For Emulex:

      # esxcli system module list | grep lpfc

    • For Brocade:

      # esxcli system module list | grep bfa

  2. Run one of these commands:

    Note: The examples show the QLogic qla2xxx and Emulex lpfc820 modules. Use the appropriate module based on the outcome of the previous step.

    • For QLogic:

      # esxcli system module parameters set -p ql2xmaxqdepth=64 -m qla2xxx

      For ESXi 5.5 QLogic native drivers:

      # esxcli system module parameters set -p ql2xmaxqdepth=64 -m qlnativefc

    • For Emulex:

      # esxcli system module parameters set -p lpfc0_lun_queue_depth=64 -m lpfc820

      For ESXi 5.5 Emulex native drivers:

      # esxcli system module parameters set -p lpfc0_lun_queue_depth=64 -m lpfc

    • For Brocade:

      # esxcli system module parameters set -p bfa_lun_queue_depth=64 -m bfa

    Notes:
    • In these commands, both ql2xmaxqdepth and lpfc0 use the lowercase letter L, "l", and not the numeric digit 1.
    • In this case, the HBAs represented by ql2x and lpfc0 have their LUN queue depths set to 64.
    • If all Emulex cards on the host must be updated, apply the global parameter, lpfc_lun_queue_depth instead.

  3. Reboot your host.
  4. Run this command to confirm that your changes have been applied:

    # esxcli system module parameters list -m driver

    Where driver is your QLogic, Emulex, or Brocade adapter driver module, such as lpfc820, qla2xxx, or bfa.

    The output appears similar to:

    Name                        Type  Value  Description 
    --------------------------  ----  -----  --------------------------------------------------
    .....
    ql2xmaxqdepth               int   64     Maximum queue depth to report for target devices.
    .....


ESXi/ESX 4.x

To adjust the queue depth for an HBA:

  1. Verify which HBA module is currently loaded by entering one of these commands on the service console:

    • For QLogic:

      # vmkload_mod -l | grep qla

    • For Emulex:

      # vmkload_mod -l | grep lpfc

    • For Brocade:

      # vmkload_mod -l | grep bfa

  2. Run one of these commands:

    Note: The examples show the QLogic qla2xxx and Emulex lpfc820 modules. Use the appropriate module based on the outcome of the previous step.

    • For QLogic:

      # esxcfg-module -s ql2xmaxqdepth=64 qla2xxx

    • For Emulex:

      # esxcfg-module -s 'lpfc0_lun_queue_depth=64' lpfc820

    • For Brocade:

      # esxcfg-module -s 'bfa_lun_queue_depth=64' bfa

    In this case, the HBAs represented by ql2x and lpfc0 have their LUN queue depths set to 64.

    Note: For multiple instances of Emulex HBAs being presented to a system, use:

    # esxcfg-module -s 'lpfc0_lun_queue_depth=64 lpfc1_lun_queue_depth=64' lpfc820

  3. Reboot your host.
  4. Run this command to confirm if your changes are applied:

    # esxcfg-module -g driver

    Where driver is your QLogic, Emulex, or Brocade adapter driver module, such as lpfc820, qla2xxx, or bfa.

ESXi/ESX 3.5

To adjust the queue depth for a QLogic HBA:

  1. Log into the service console as root.
  2. Verify which QLogic HBA module is currently loaded:

    # vmkload_mod -l | grep qla

    Depending on the model of the HBA, the module can be one of these:

    • qla2300_707_vmw
    • qla4010
    • qla4022

  3. Run these commands:

    # esxcfg-module -s ql2xmaxqdepth=64 qla2300_707_vmw
    # esxcfg-boot –b


    The HBA represented by ql2x has its LUN queue depth set to 64.

    Note: The example shows the qla2300_707 module. Use the appropriate module based on the output of step 2.

  4. Reboot the host.
  5. Run this command to confirm your changes have been applied:

    # esxcfg-module -g driver

    Where driver is your QLogic, Emulex, or Brocade adapter driver module, such as lpfc820, qla2xxx, or bfa.

To adjust the queue depth of an Emulex HBA:

  1. Log into the service console as root.
  2. Verify which Emulex HBA module is currently loaded:

    # vmkload_mod -l | grep lpfc

  3. For a single instance of an Emulex HBA on the system, run these commands:

    Note: The example shows the lpfc_740 module. Use the appropriate module based on the output of step 2.

    # esxcfg-module -s lpfc0_lun_queue_depth=64 lpfc_740
    # esxcfg-boot –b


    In this case, the HBA represented by lpfc0 has its LUN queue depth set to 64.

  4. For multiple instances of an Emulex HBA present on the system, run these commands:

    # esxcfg-module -s "lpfc0_lun_queue_depth=64 lpfc1_lun_queue_depth=64" lpfc_7xx
    # esxcfg-boot –b


    In this case, both HBAs lpfc0 and lpfc1 have their LUN queue depths set to 64.

  5. Reboot the host.
  6. Run this command to confirm your changes have been applied:

    # esxcfg-module -g driver

    Where driver is your QLogic, Emulex, or Brocade adapter driver module, such as lpfc820, qla2xxx, or bfa.

ESX 2.x

To adjust the queue depth of the QLogic adapter on ESX 2.x:

  1. In a text editor, edit the /etc/vmware/hwconfig file and locate the line for the QLogic adapter. It looks similar to:

    device.3.12.0.name = "QLogic Corp QLA2312/2340 (rev 02)"

    Make note of the string of three numbers between device and name.

  2. In the /etc/vmware/hwconfig file, add one of these lines to the end of the file:

    • For ESX Server 2.0, add this line:

      device.vmnix.3.12.0.options = "ql2xmaxqdepth=nn"

    • For ESX Server 2.1 and later versions, add this line:

      device.esx.3.12.0.options = "ql2xmaxqdepth=nn"

      In these examples, 3.12.0 is the string of numbers obtained from the QLogic line located in the previous step, and nn is the desired queue depth maximum. Be sure to save your changes.

      Note: The second character in ql2xmaxqdepth is a lowercase L, "l".

  3. Reboot the host.

Tags

queue-depth  Qdepth

Keywords

Fibre Channel storage urlz

Update History

12/04/2012 - Added link to article 1008113. 02/08/2013 - Added link to article 1027901. 11/14/2013 - Added ESXi 5.5 to Product Versions. 11/15/2013 - Added the ESXi 5.5 commands to manipulate the queue depth in the QLogic native driver.

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 41 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.
What can we do to improve this information? (4000 or fewer characters)
  • 41 Ratings
Actions
KB: