Knowledge Base

The VMware Knowledge Base provides support solutions, error messages and troubleshooting guides
 
Search the VMware Knowledge Base (KB)   View by Article ID
 

iSCSI and Jumbo Frames configuration on ESX/ESXi (1007654)

Symptoms

  • You see large queues when monitoring Jumbo Frames
  • Packet fragmentation is observed when viewing a packet capture using the tcpdump utility

Purpose

This article provides information related to setting up a NAS and an iSCSI environment with Jumbo Frames.
 
Note: Jumbo Frames on software iSCSI for ESX/ESXi 3.5 is an experimental feature and is not a fully supported configuration. Jumbo Frames are fully supported in ESX/ESXi 4.x and ESXi 5.x.

Resolution

Ensure that you read this important information about Jumbo Frames before working with them:
  • ESX/ESXi supports a maximum MTU size of 9000.

    Note: Some switch configurations for Jumbo Frames need to have an MTU set higher than 9000. For more information, see the Cisco Nexus 5000 Series NX-OS Software Configuration Guide.

  • Any packet larger than 1500 MTU is a Jumbo Frame. ESX/ESXi supports frames up to 9 Kb (9000 bytes).
  • Jumbo Frames are limited to data networking only (virtual machines and the vMotion network).
  • It is possible to configure Jumbo Frames for an iSCSI network. It is not a fully supported configuration in ESX 3.5, but it is supported in ESX/ESXi 4.x and ESXi 5.0.
  • You can enable Jumbo Frames for each vSwitch or VMkernel interface through the command line interface on your ESX host.
  • To allow an ESX host to send larger frames out onto the physical network, the network must support Jumbo Frames end to end.
  • Ensure that your NIC or LOM supports Jumbo Frames.
  • For experimental support of Jumbo Frames in ESX 3.5, these NICs are supported:

    • Intel (82546, 82571)
    • Broadcom (5708, 5706, 5709, 57710, 57711)
    • Netxen (NXB-10GXxR, NXB-10GCX4)
    • Neterion (Xframe, Xframe II, Xframe E)

  • For ESX/ESXi 4.x and ESXi 5.0, contact your NIC hardware vendor regarding support for Jumbo Frames.
  • You cannot use Jumbo Frames on a Broadcom card that is configured as a hardware initiator performing iSCSI Offload functions. You can either use Jumbo Frames or iSCSI Offload and you cannot use both together with the Broadcom adapters.

Jumbo Frames in ESXi 5.1 and later

Jumbo frames for all iSCSI adapters in vSphere 5.1 can be configured using the UI. This applies to Software iSCSI,Dependent Hardware iSCSI and Independent Hardware iSCSI adapters.
 
To enable Jumbo Frames for software and dependent hardware iSCSI adapters in the vSphere Web Client, change the default value of the MTU parameter:
  1. Browse to the host in the vSphere Web Client navigator.
  2. Click the Manage tab, and click Networking.
  3. Click Virtual Switches, and select the vSphere switch that you want to modify from the list.
  4. Click Edit Settings.
  5. On the Properties page, change the MTU parameter.

    This step sets the MTU for all physical NICs on that standard switch. The MTU value should be set to the largest MTU size among all NICs connected to the standard switch. ESXi supports the MTU size up to 9000 Bytes.
 
To enable Jumbo Frames for independent hardware iSCSI adapters in the vSphere Web Client, change the default value of the MTU parameter:
 
Use the Advanced Options settings to change the MTU parameter for the iSCSI HBA.
  1. Browse to the host in the vSphere Web Client navigator.
  2. Click the Manage tab, and click Storage.
  3. Click Storage Adapters, and select the independent hardware iSCSI adapter from the list of adapters.
  4. Under Adapter Details, click the Advanced Options tab and click Edit.
  5. Change the value of the MTU parameter.

Jumbo Frame 5.0 and earlier

Creating a Jumbo Frames-enabled vSwitch

To create a Jumbo Frames-enabled vSwitch:

  1. Log in to the ESX host console directly.
  2. To set the MTU size for the vSwitch:

    • Run this command for ESX 3.5 and ESX/ESXi 4.x:

      # esxcfg-vswitch -m MTU vSwitch#

    • Run this command for ESXi 5.0:

      # esxcli network vswitch standard set -m MTU -v vSwitch#

      Note: This command sets the MTU for all uplinks on that vSwitch. Set the MTU size to the largest MTU size among all the virtual network adapters connected to the vSwitch.

  3. To display a list of vSwitches on the host, and to check that the configuration of the vSwitch is correct:

    • Run this command for ESX 3.5 and ESX/ESXi 4.x:

      # esxcfg-vswitch -l

    • Run this command for ESXi 5.0:

      # esxcli network vswitch standard list

Configuring Jumbo Frames on a vSphere Standard Switch

To configure Jumbo Frames on a vSphere Standard Switch:

  1. Log into the vSphere Client and select the Hosts and Clusters inventory view.
  2. On the host Configuration tab, click Networking.
  3. Click Properties for the vSphere standard switch associated with the VMkernel to modify.
  4. On the Ports tab, select the VMkernel interface and click Edit.
  5. Set the MTU to 9000, and click OK.
Note: To create a Jumbo Frames-enabled vNetwork Distributed Switch and its associated VMkernel interfaces, see Enabling Jumbo Frames for VMkernel ports in a virtual distributed switch (1038827).

To create a Jumbo Frames-enabled VMkernel interface on a vNetwork Standard Switch:

  1. Log directly into the ESX host console.
  2. Obtain the current vSwitch and portgroup configuration with the esxcfg-vswitch command:

    # esxcfg-vswitch -l

  3. To create a VMkernel interface with Jumbo Frames support, we first need to create a portgroup on an existing vSwitch:

    # esxcfg-vswitch -A vmkernel_port_group_name vSwitch#

    Note: If you plan to have a vSwitch which just contains the iSCSI port group, you can specify the MTU for the vSwitch to be 9000 and need to specify the MTU of 9000 when creating the VMkernel port as well in the next step. To configure a vSwitch to use Jumbo Frames (MTU 9000):

    # esxcfg-vswitch -m 9000 vSwitch#

  4. To create a VMkernel connection with Jumbo Frame support:

    • Run this command for ESX 3.5 and ESX/ESXi 4.x:

      # esxcfg-vmknic -a -i ip_address -n netmask -m MTU portgroup_name

      Note: If the vmnic port is already created, use the command for ESX/ESXi 4.1 only:

      # esxcfg-vmknic -m 9000 portgroup_name

    • Run this command for ESXi 5.x:

      # esxcli network ip interface set -m 9000 -i vmk_interface

  5. To display a list of VMkernel interfaces, and to check that the configuration of the Jumbo Frame‐enabled interface is correct:

    • Run this command for ESX 3.5 and ESX/ESXi 4.x:

      # esxcfg-vmknic -l

    • Run this command for ESXi 5.x:

      # esxcli network ip interface list

Additional Information

Tags

enable-IOAT-Jumbo-Frames

See Also

Update History

06/23/2010 - Mentioned Broadcom 57710 and 57711 in the list of supported cards. 10/14/2010 - Added a note on the command to use when vmnic port is already created. 02/06/2013 - Added ESXi 5.1.x to Products

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 52 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.
What can we do to improve this information? (4000 or fewer characters)
  • 52 Ratings
Actions
KB: