This article provides information about Partner Verified and Supported Products (PVSP) for the Internet SCSI (iSCSI) over Data Center Bridging (DCB) Solution for ESX from partner Dell, Inc. The iSCSI over DCB solution for ESX consists of Dell™ EqualLogic™ PS Series storage arrays supporting 10GbE in conjunction with host adapters and switch components that provide Enterprise iSCSI over DCB functionality.

Note: The Partner Verified and Supported Products (PVSP) policy implies that the solution is not directly supported by VMware. For issues with this configuration, contact Dell directly. See the Support Workflow on how partners can engage with VMware. It is Dell's responsibility to verify that the configuration functions with future vSphere major and minor releases, as VMware does not guarantee that compatibility with future releases is maintained.

Disclaimer: The partner products referenced in this article are hardware devices that are developed and supported by stated partners. Use of these products are also governed by the end user license agreements of the partners. You must obtain the application, support, and licensing for using these products from the partners. For more information, see Support Information in this article.

Solution Overview

• Converged network adapters (CNAs) that have the ability to manage all of the DCB functionality within its hardware engine without any assistance from ESX. See the EqualLogic Compatibility Matrix for specific vendors and models.
• Switches that support these DCB standard features:
  • Per Priority Flow Control (PFC)
  • Enhanced Transmission Selection (ETS)
  • Data Center Bridging Exchange (DCBx)
  • iSCSI TLV (Enterprise iSCSI over DCB)
    
    Note: See the EqualLogic Compatibility Matrix for specific vendors and models.
     
• EqualLogic PS-Series 10Gb Ethernet arrays Array Firmware 5.1.x or later

Solution Details

Host Configuration

• Simple NIC mode, where all of the functionality for the TCP/IP and iSCSI functions are provided for by the host OS
• Hardware offload mode for TCP and/or iSCSI, where there is no management of these functions by the host OS, but is completely managed by the CNA hardware device.

The first method requires specific kernel based code to manage the different types of traffic (LAN/Client, SAN, Cluster, Management) and based on the source or destination information, modify the Ethernet frame with the proper tags to allow the host converged network adapters and switches to properly manage the frame from the host to the eventual destination. Since this code does not currently exist in the VMware kernel for iSCSI, this method is not possible with current VMware versions.

The second method requires that the CNA manage all of the traffic in proxy to the host operating system. This means that the host operating system is not required to have any knowledge of the DCB environment for it to be able to participate in the DCB converged network. To do this, the CNA must have a way of properly routing the traffic between the network and the host operating system. In this method, that would require that the CNA provide multiple personalities to the host operating system via its in-box driver. These personalities are represented by pseudo-devices. A CNA with this functionality has the ability to present to the OS a standard Ethernet NIC pseudo-device and a separate SCSI Storage Controller pseudo-device. Additionally, the CNA may offer some form of network partitioning that allows the Ethernet NIC device to be further partitioned into multiple virtual kernel NIC ports.

To be able to function in this second mode of operation, the CNA must have the ability to operate in either Host Bus Adapter mode or iSCSI Hardware Offload mode. Today, there are several CNAs that meet this criteria. For details on how to determine which CNAs have been validated by Dell’s Interoperability Test Lab for use within this solution, see the EqualLogic Compatibility Matrix.

• The host CNAs can be manually configured for these settings
  
  Or
   
• The host CNAs can accept the DCB settings from the switch(es) to which the CNA is connected.

By industry ad-hoc agreement, it is customary for DCB settings to be maintained within the switching domain and that all edge devices (such as CNAs and arrays) be in DCB Willing mode. In this mode, the edge devices receive their DCB settings from the switch upon initial port link up. Any CNA used within this solution must be configured in DCB willing mode. Since the VMware environment is unaware of the DCB settings, it may be necessary that all iSCSI over DCB configuration of the selected CNAs be done via the CNA’s BIOS configuration interface available during server Power-On Self Test process or other pre-boot management processes like the Dell Lifecycle Manager. Depending on CNA vendor implementations, it may be possible to use the vendor’s vCenter Server plug-in module to configure the appropriate DCB settings.

Each CNA vendor has unique set of steps to properly configure the CNA into DCB willing mode and to enable support for iSCSI over DCB. See the vendor’s product documentation for the specific steps. In general, these steps include:

1. Enable iSCSI Offload or iSCSI HBA mode.
2. Select DCB Willing or Accept DCB Settings from Switch (or similar setting) to enable DCB willing mode.
3. Install appropriate iSCSI Offload/HBA VMware drivers provided by the vendor and located on the VMware Hardware Compatibility List. See Support Information to find out how to identify the supported network adapters and the tested driver versions.

Switches

Switch Functionality Requirements

• PFC – In this solution, iSCSI traffic must not be blocked by any other types (streams) of network traffic such as the LAN or other network use cases. Per Priority Flow Control prevents blocking of resources from other classes of service within the DCB domain. Any switch for this solution must allow the iSCSI traffic to be assigned to a unique class of service from all other traffic types.
   
• ETS – In this solution, iSCSI SAN traffic must be guaranteed a minimum amount of total bandwidth throughout the DCB infrastructure domain. Enhanced Transmission Selection is the mechanism that defines and enforces the bandwidth allocation as a percentage of the link bandwidth.
   
• DCBx with iSCSI TLV – The switch must be able to communicate to other DCB capable devices the unique iSCSI settings within the DCB domain. This is done by transmitting an iSCSI TLV Ethernet frame within the DCB Capabilities Exchange process.

Switch Configuration

Each switch has a unique configuration process to enable iSCSI over DCB. In general, the steps include:

1. Enable DCB functionality.
2. Identify iSCSI or TCP Port 3270 for a unique class of service (default generally accepted as CoS:4).
3. Assign iSCSI class of service to a DCB Priority Group.
4. If required, define the iSCSI assigned Priority Group as a no-drop priority group.
5. Define the ETS bandwidth allocation between all active DCB priority groups.

• ETS is configured based on percentage of link’s operating bandwidth. For example, an ETS setting of 30% would indicate 3Gb of bandwidth for a 10Gb link. It would also indicate 12Gb of bandwidth on a 40Gb link.
• ETS settings are a guaranteed minimum bandwidth allocation for the priority group. If additional bandwidth is available at any given time, a priority group may use additional bandwidth until it is required. Also, if classes assigned to the priority group are not transmitting, the bandwidth can be made available to other priority groups until such time as the priority group bandwidth is again required.
• Dell recommends isolating iSCSI traffic into its own VLAN to provide an additional layer of security.

Dell EqualLogic PS Series Storage Arrays

All PS Series 10Gb Ethernet storage arrays are DCB ready and can have DCB capabilities enabled when using Array Software version 5.1 or later. When DCB is enabled on a Storage Group, the arrays are placed into a DCB Willing mode whereby they will accept DCB domain settings from the DCB configured switch to which they are attached. This is the only method the arrays can use to be configured with the appropriate DCB settings for PFC, Class of Service, ETS, Priority Groups, etc. that make up the definition of the DCB domain.

To configure DCB on the EqualLogic SAN Group:

1. Ensure that Array Software 5.1 or later is installed on all PS arrays in the Storage Group 2. Ensure that DCB is enabled on the PS Array Group:
   1. Click Group Manager > Group Configuration > Advanced tab > Network Management.
   2. Select Enable DCB:
      
      
      

   3. Enter the VLAN ID that is being used for iSCSI or Converged network traffic on the switch infrastructure.
      
      Note: The default VLAN ID is listed as VLAN 0, but that VLAN should not be used in production environments. iSCSI traffic should be isolated to a separate, non-default VLAN from other VLANs.

2. Verify DCB settings

   1. Select a Group Member array.

   2. Click the Network tab.

   3. Right-click on any array member Ethernet port and select DCB Details.

      Notes:

      • iSCSI should be listed in a Group and be listed as Lossless.
      • DCB VLAN ID should not be the default VLAN (typically VLAN0 or VLAN1 depending on switch vendor). Many switch vendors ignore or strip off the VLAN ID Tag within the Ethernet frame on the default VLAN.
      • VLAN ID must match the VLAN assigned to iSCSI SAN traffic within the switch infrastructure. If this VLAN ID is not properly set, DCB will not work and host-target connections may not be possible.

Dell Verification Details

Over the last three years Dell has attended numerous industry plugfests sponsored by the Ethernet Alliance’s Data Center Bridging Consortium, the University of New Hampshire (UNH) iSCSI and DCB consortium plugfests, and is listed in the UNH IOL Interoperability Integrators Matrix.

The UNH IOL iSCSI over DCB test matrix can be obtained here.

Support Information

The first line of support for this solution is Dell’s EqualLogic Customer Support (https://support.equallogic.com/). All Dell customers are required to have active support agreements for all components within the solution (Servers, OS, CNA, Switch, and PS Arrays).

A minimum of ProSupport Next Day Response (or vendor equivalent) is required for all components in the SAN/LAN converged solution defined by the knowledge base article and support will be provided based on the limits of that contractual agreement.

• CNAs and Switches must be listed on the Dell EqualLogic Compatibility Matrix
• CNAs must indicate support for DCB for the VMware version being deployed (5.1 or later).
• Switches must indicate support for DCB with the deployed version of PS Series Array Software
• CNAs must be listed on the VMware Hardware Compatibility List

EqualLogic VMware Integration Software and Documentation can be obtained here. (Login and product registration required)

VMware Release To Release Support

Because DCB support is self-contained, resident in the CNA hardware, switch hardware and the iSCSI array, release to release compatibility with ESXi is expected.

Additional Resources

• VMware IO Compatibility Guide
• Dell EqualLogic Configuration Guide
• Dell Switch Configuration Guides
• Dell Rapid EqualLogic Configuration Portal
• Dell EqualLogic Whitepapers

VMware Release to Release compatibility

Dell’s continuing interoperability test program includes integration of future VMware releases. The Dell EqualLogic Compatibility Matrix will provide additional details about the suitability of future version of VMware ESXi.