Knowledge Base

This answer applies to ESX Server 2.5, as it includes changes dependent on new features. For corresponding information about ESX Server 2.1, refer to kb.vmware.com/kb/1325. For corresponding information about ESX Server 2.0, refer to kb.vmware.com/kb/1303.

These recommendations apply to cluster configurations that cross machine boundaries. This includes cluster nodes on two or more ESX Server machines that share an IBM FAStT storage server. It also includes clusters that pair an entire physical machine with a virtual machine passive node, using a raw LUN as the shared disk volume. For more information about clustering options, refer to www.vmware.com/support/esx25/doc/admin/esx25admin_cluster_setup_esx.html.

To configure an IBM FAStT storage array to use clustering as well as multipathing with HBA and storage port failover, make sure all the following conditions are met:

• Each ESX Server machine has four paths to the LUN, using redundant HBAs, switches and storage processors. (Details)
• The storage processors should be configured with AVT disabled. (Details)
• The host type for the storage processors should be set to LNXCL. (Details)
• The storage processors should return Not Ready sense data when quiesced. (Details)
• The multipathing policy for the LUN should be set to MRU. (Details)
• The VMkernel on each ESX Server machine should use LUN reset in place of bus reset. (Details)
• The virtual machines should use raw device mappings to access their virtual disks. (Details)

Note the following:

• This configuration is not compatible with boot from SAN.
• Virtual memory swap space should be on local storage rather than on a SAN. Accessing swap files on a SAN is subject to occasional delays; these can impact critical operations in ESX Server, leading to spurious failovers in Microsoft Cluster Services.

Configuring the Hardware for SAN Failover with FAStT Storage Servers

To configure a proper highly available SAN failover configuration with FAStT storage models equipped with two storage processors, you need these hardware components:

• Two Fibre Channel host bus adapters (HBAs), such as QLogic or Emulex, on each ESX Server machine. Each pair of adapters must be the same model.
• Two fibre switches connecting the HBAs to the SAN: SW1 and SW2.
• Two storage processors (SPs): SPA and SPB. Each storage processor must have at least two ports connected to the SAN.

• HBA0 and HBA1 on each ESX Server machine should be connected to switches SW1 and SW2, respectively.
• SW1 should be connected to port 1 on SPA and to port 2 on SPB. Connect SPA to a lower switch port number (than SPB) on SW1, so SPA is listed first.
• SW2 should be connected to port 2 on SPA and to port 1 on SPB. Connect SPA to a lower switch port number (than SPB) on SW2, so SPA is listed first.

This configuration provides two paths from each HBA, so each element of the connection is able to fail over to a redundant path.

Viewing the Storage Processor Port Configuration

To view storage processors and related ports on the preferred path, log on to the ESX Server machine as root and enter the following command:
wwpn.pl -v

The output lists the World Wide Port Names (WWPNs) of HBAs on the ESX Server machine and storage processors on the FAStT. Compare these to those listed in the FAStT profile. The target numbers and WWPNs must be identical as seen by each ESX Server machine connected to the same FAStT.

Disabling AVT with Host Type

To avoid the possibility of path thrashing, you should disable AVT (Auto Volume Transfer) on the SAN storage processors. If AVT is enabled, the two storage processors can alternately take ownership of the LUN from each other in certain situations, resulting in performance degradation. AVT may also be referred to as ADT (Auto Disk Transfer).

AVT is enabled by default for the LINUX host type. To disable AVT, set the host type to LNXCL in the FAStT Storage Manager for each host group containing HBAs for one or more ESX Server machines. LNXCL has AVT disabled by default.

You need to reboot the ESX Server machine after changing the AVT configuration.

Configuring Storage Processor Sense Data

If you are running Windows as a guest operating system, the FAStT storage processors should return Not Ready sense data when quiesced. The storage processors can be configured to return either Unit Attention or Not Ready. Returning Unit Attention may cause the Windows guest to crash during a failover.

To configure the storage processors to return Not Ready sense data, perform these steps:

1. Determine the index for the host type LNXCL, using your FAStT Storage Manager software or by running SMcli.exe directly. For SMcli.exe, use the following commands in a shell window:
   
   SMcli.exe <ip-addr-for-sp-A>
     show hosttopology;
     <EOF>
   
   SMcli.exe <ip-addr-for-sp-B>
     show hosttopology;
     <EOF>
    
   The following commands assume 13 is the index corresponding to LNXCL in the NVSRAM host type definitions. If your storage processors have LNXCL at a different index, substitute that index for 13 in the following commands.

   Warning: The following commands assume that the sense control bytes are at offsets 0x12 and 0x13 in the host type definition. These offsets are valid for firmware version 05.40. It is possible that a different firmware version could put the control bytes at a different offset.

2. Execute these commands for storage processor A:
   
   SMcli.exe <ip-addr-for-sp-A>
     set controller [a] HostNVSRAMBYTE [13,0x12]=0x01;
     set controller [a] HostNVSRAMBYTE [13,0x13]=0x00;
     reset Controller [a];
     <EOF>
   
   
3. Execute these commands for storage processor B:
   
   SMcli.exe <ip-addr-for-sp-B>
     set controller [b] HostNVSRAMBYTE [13,0x12]=0x01;
     set controller [b] HostNVSRAMBYTE [13,0x13]=0x00;
     reset Controller [b];
     <EOF>

   Note: If you use the FAStT Storage Manager GUI, you can paste the configuration commands for both storage processors into a single script and configure both storage processors at the same time. If you use SMcli.exe, you need to make individual connections to the two storage processors.

Viewing Multipath Information

To view your current multipathing configuration, log on to the ESX Server machine as root and enter:
vmkmultipath -q

The output lists all known paths to each SAN LUN. The asterisk (*) in the output indicates that the path is the current, active path, and the pound sign (#) indicates the preferred path from the server to the LUN. (Note that the preferred path is largely irrelevant when the Most Recently Used (MRU) multipathing policy is configured.) When ESX Server 2.5 is connected to IBM FAStT SANs, the default policy is MRU. To avoid the possibility of path thrashing, you should keep the MRU policy setting.

You can make changes to the multipathing policy from the VMware Management Interface, which causes them to persist across reboots of the ESX Server machine. To access the multipathing configuration in the management interface:

1. Click the Options tab.
2. Click the Storage Management link.
3. Click the Failover Paths tab.
4. Click the Edit link for the shared LUN.

Configuring LUN Reset

A Fibre Channel HBA failure causes ESX Server to stop sending SCSI commands from the failed HBA, and to send them from the backup HBA instead. The LUN receiving the SCSI commands would normally reject them, due to the previous SCSI reservation issued by Microsoft Cluster Services, which bound the LUN to the now-failed HBA.

You should configure ESX Server to do a LUN reset on HBA failover, so that the previous SCSI reservation is cancelled. Microsoft Cluster Services is then able to issue a new SCSI reservation from the backup HBA. To make this change, perform these steps in the VMware Management Interface:

1. Click the Options tab.
2. Click the Advanced Settings link.
3. Set Disk.ResetOnFailover to 1. (Click the value to open a new window where you can enter a replacement value.)

A failover detected in Microsoft Cluster Services causes the formerly passive node to take over for the failed active node. This process involves issuing a bus reset before taking ownership of the LUN. The effect of a bus reset on the SAN is to reset all LUNs accessed by the storage processor.

By default, ESX Server 2.5 translates a bus reset into a LUN reset. This restricts the reset to the LUN belonging to the virtual machine that is taking the active role in the cluster. If the setting on your system has been changed from the default, you should configure it to do a LUN reset. Perform these steps in the VMware Management Interface:

1. Click the Options tab.
2. Click the Advanced Settings link.
3. Set Disk.UseDeviceReset to 0. (Click the value to open a new window where you can enter a replacement value.)
4. Set Disk.UseLunReset to 1.

Configuring Raw Device Mappings

The recommended failover configuration uses raw device mappings to access raw LUNs acting as virtual disks. The primary advantage of raw device mapping in a failover situation is that the name of the virtual disk is the same for both the primary and the backup server. When a failover happens, ESX Server automatically resolves the new path to the LUN. There is no need to configure persistent bindings for device names.

The VMFS volume containing the mapping file should be configured for shared access mode, so all virtual machines in the cluster can open the mapping file simultaneously. In the case of a single virtual machine acting as a backup for a physical machine, a public VMFS volume can be used.

You should also configure the raw device mapping to use physical compatibility mode. This causes ESX Server to pass SCSI commands directly to the raw LUNs acting as the shared virtual disk for the virtual machines on each node.

When using raw device mappings in physical compatibility mode, you should also set the virtual SCSI controller bus sharing option to none or physical. The bus sharing option setting of virtual is not supported in this mode.

To choose the bus sharing option for a virtual device, perform the following steps in the VMware Management Interface:

1. Click the small triangle to the right of the virtual machine icon and choose Configure Hardware.
2. Click the Edit link for the SCSI controller to which the shared device is attached.
3. In the drop-down menu for Bus Sharing, choose none.

If you are unable to use raw device mappings, you need to determine the virtual disk device names and configure persistent bindings the same way as for ESX Server 2.1. In that case, refer to the recommendations at kb.vmware.com/kb/1325.

To configure raw device mappings, refer to www.vmware.com/pdf/esx25_rawdevicemapping.pdf.