Similar to VMFS and NFS datastores, VMware vSAN prevents multiple virtual machines from opening the same virtual disk (VMDK) in read-write mode. This is to protect the data stored on the virtual disk from corruption caused by multiple writers on the non-cluster-aware filesystems used by most guest operating systems.

To enable in-guest systems that leverage cluster-aware filesystems that have distributed write (multi-writer) capability, you must explicitly enable multi-writer support for all applicable virtual machines and VMDKs.

Virtual Use case:

• Red Hat Clustering Suite or SUSE Linux Enterprise HA, in which the application ensures that writes originating from two or more different virtual machines do not cause data loss.

Limitations and Requirements:

• When running a 3rd party Clustering Software like Red Hat Clustering Suite or SUSE Linux Enterprise HA on VMware infrastructure, contact the vendor for any Guest Operating System-level Clustering and Configuration issues.
• Support related to Multi-Writer Configuration and issues will be assisted by the VMware Technical Support team.
• To configure the Multi-Writer flag for Enabling Virtual Disk Sharing, see Enabling or disabling simultaneous write protection provided by VMFS using the multi-writer flag (1034165).
• RHEL 6.x and higher, or SUSE Linux 11 and higher, are the supported versions with multi-writer mode.
• As VMware vSAN does not support Raw Device Mappings (RDMs), this article applies only to virtual disks resident on the vSAN datastore.
• vSAN supports SCSI-3 persistent reservations only for Microsoft Windows Server Failover Clusters (WSFC) as of vSAN version 6.7 U3 and newer. Previous versions of vSAN do not support SCSI-3 persistent reservations.
• Prior to vSAN 6.7 Patch 01, when using the multi-writer mode, the virtual disk must be eager-zeroed thick (EZT). When creating EZT disk on vSAN, the disk is not zeroed automatically. If zeroing is required, you must use the vmkfstools -w command to zero out all blocks. For more information, see the following KB: Powering on the virtual machine fails with the error: Thin/TBZ disks cannot be opened in multiwriter mode
• [New in vSphere 6.7 Patch 01]  Starting in vSAN 6.7 Patch 01, EZT disk is no longer required for the multi-writer mode to be enabled on vSAN. Follow the steps that pertain to EZT disk below only if your vSAN version is prior to 6.7 Patch 01. Otherwise, you may skip over the steps to configure EZT disk and use a thin-provisioned disk instead.
• Sharing virtual disks in multi-writer mode is limited to 8 ESXi hosts, but there can be any number of supported ESXi hosts in that vSphere/vSAN cluster. The limitation is not on the number of virtual machines sharing the virtual disks.
• For other uses cases that require shared virtual disks with ready-only access, the above limitation does not apply (for View, linked clone, and fast provisioning use cases)

Note:

The eight hosts limit applies to multi-writer mode shared virtual disks only. Otherwise, the vSphere cluster size can be up to the maximum size supported by the license in use. However, from vSphere 6.7 Update 1 onwards, the virtual disk sharing support in multi-writer has been extended to more than 8 hosts. This applies to VMFS5 / VMFS6 as well. In order to enable this feature, you need to enable /VMFS3/GBLAllowMW advanced config option as shown below:

Log in to ESXi host and execute the below command:

#esxcli system settings advanced set -i 1 -o /VMFS3/GBLAllowMW

In a shared environment such as Raw Device Mapping (RDM) disks, you cannot use Multi-Writer Locks for virtual machines on more than eight ESXi hosts. If you migrate a virtual machine to a ninth host, it might fail to power on with the error message Could not open xxxx.vmdk or one of the snapshots it depends on. (Too many users). This fix makes the advanced configuration option /VMFS3/GBLAllowMW visible. You can manually enable or disable multi-writer locks for more than 8 hosts by using generation-based locking. This issue is resolved in this release. For more details, see VMware knowledge base article 1034165.

Enabling or disabling simultaneous write protection provided by VMFS using the multi-writer flag (1034165): https://kb.vmware.com/s/article/1034165

Overview

The process of configuring a Red Hat cluster or SUSE Linux Enterprise HA on a vSAN datastore needs to be done once per cluster at creation time. This requires these steps:

• Creating a VM Storage Policy that will be applied to the virtual disks that will be used as the cluster’s shared storage.
• Creating the shared virtual disks in eager-zeroed mode.
• Attaching the shared disks to one or more virtual machines.
• Enabling multi-writer mode for the VMs and disks.
• Applying the virtual machine Storage Policy to the shared disks.

In this example, the VM Storage Policy is named as RHCS

• Object Space Reservation: 100% which pre-allocates all object’s components on disk.
• Number of failures to tolerate: 1 is the default which will provide a mirror replica on another vSAN node. This provides RAID 1 protection against one host/hardware component failure. As the dialog shows, this will consume double the storage space on the vSAN datastore.
• Number of disk stripes per object: The default vSAN policy is 1. For this policy select the desired stripe width. This is similar to RAID 0. In this example, we are using the value of 2.

For more information about storage policy configuration options, see the VMware vSAN documentation.

Add a storage controller to the Red Hat Enterprise Linux virtual machines

Note: Create controllers of the same type and in the same position (SCSI address) on each Red Hat Enterprise Linux virtual machine.

1. Right-click the VM in the vSphere Web Client and select Edit Settings.
2. At the bottom of the subsequent window, select SCSI Controller from the New device: drop-down menu and then click Add.
3. Expand the New SCSI Controller entry.
4. Leave the Bus Sharing mode as None, and choose the desired type (LSI ParallelorLSI SASorParavirtual).
   
   Note: The Paravirtual controller type is generally recommended for Red Hat cluster shared disks.
   
   
5. Click OK.
6. Repeat this process on the remaining Red Hat Enterprise Linux virtual machines.

For vSphere 6.5 or newer

Creating shared disks on the first virtual machine

Starting with vSphere version 6.5, the vSphere Web Client has the option to create eager-zeroed thick (EZT) disks on the vSAN datastore.

Note: The virtual disks should be added to the same SCSI positions on each virtual machine. If a disk is in position 1:0 on one virtual machine, it should be in position 1:0 on all virtual machines in the Linux Cluster.

Using the vSphere Web Client

To create shared disks on the first virtual machine using the vSphere Web Client:

1. Right-click the appropriate virtual machine and select Edit Settings.
2. Select New Hard Disk from the New Device drop-down menu and then click Add.
3. Expand the New Hard Disk entry and select the Thick provision eager zeroed option in the Disk Provisioning drop-down menu.
4. In the Sharing drop-down menu, select the Multi-writer option.
5. Change the Disk Mode to Independent-Persistent.
   
   Note: While Independent-Persistent disk mode is not a hard requirement to enable the Multi-writer option, the default Dependent disk mode would cause the cannot snapshot shared disk error when a virtual machine snapshot is taken. The use of Independent-Persistent disk mode would allow taking a snapshot of the OS disk while the shared disk would need to be backed up separately by a third-party vendor software.
    
6. Modify the Virtual Device Node, as appropriate.
7. Click OK to save the changes.
8. (Optional) As EZT disks created on vSAN are not zeroed automatically, you must use the command vmkfstools -w <path-to-vmdk> to zero out all blocks if zeroing is required. Be aware of the additional IO workload on vSAN during zeroing.
9. Repeat this process for the remaining shared disks.
   
   

Adding shared disks to one or more virtual machines

To add shared disks to one or more virtual machines using the vSphere Web Client:

1. Right-click the appropriate virtual machine and select Edit Settings.
2. Select Existing Hard Disk from the New Device drop-down menu and then click Add.
3. Navigate to the applicable directory and select the disk.
4. Click OK.
5. Expand the New Hard Disk entry and modify the Virtual Device Node, as appropriate.
6. In the Sharing dropdown menu, select the Multi-writer option.
7. Change the Disk Mode to Independent-Persistent.
   
   Note: While Independent-Persistent disk mode is not a hard requirement to enable the Multi-writer option, the default Dependent disk mode would cause the cannot snapshot shared disk error when a virtual machine snapshot is taken. The use of Independent-Persistent disk mode would allow taking a snapshot of the OS disk while the shared disk would need to be backed up separately by a third-party vendor software.
    
8. Click OK to save the changes.
9. Repeat this process for the remaining virtual machines and shared disks.
   
   

For vSphere 6.0 Update 1 or newer, but below version 6.5

Creating Eager-Zeroed Disks (PowerCLI)

In vSphere version 6.0 Update 1 or later versions, (not later than version 6.5, Example: 6.0 Update 3), the vSphere Web Client cannot create eager-zeroed disks on the vSAN datastore. While the vSAN Datastore supports eager-zeroed disks, this functionality is not exposed in the vSphere Web Client. To accommodate this limitation in the current release, we must use either PowerCLI or the ESXi command line to create the eager-zeroed disks. This section outlines the PowerCLI method to create disks, and is the preferred method.

1. Launch PowerCLI and connect to vCenter with Connect-VIServer.
   
   For more guidance with PowerCLI conventions, login, and so on, see the VMware vSphere PowerCLI Documentation.
    
2. Create the new hard disk with the New-HardDiskcmdlet in the form of:
   
   New-HardDisk -VM VM_name -CapacityGBsize-DatastorevSAN_datastore_name-StorageFormat EagerZeroedThick -Controllercontroller_name -Persistence IndependentPersistent
   
   For Example, to create a 16 GB disk on the RHCS_0VM, attached to SCSI Controller 1 on the datastore vsanDatastore, run command:
   
   New-HardDisk -VM “RHCS_0” -CapacityGB 16 -Datastore “vsanDatastore” -StorageFormat EagerZeroedThick -Controller “SCSI Controller 1” -Persistence IndependentPersistent
   
   The name of the storage controller will match the name of the device we added to the virtual machines in the previous section.
   
   Note: While Independent-Persistent disk mode is not a hard requirement to enable Multi-writer option, the default Dependent disk mode would cause the cannot snapshot shared disk error when a virtual machine snapshot is taken. Use of Independent-Persistent disk mode would allow taking a snapshot of the OS disk while the shared disk would need to be backed up separately by a third-party vendor software.
    
3. (Optional) As EZT disks created on vSAN are not zeroed automatically, you must use the command vmkfstools -w <path-to-vmdk> to zero out all blocks if zeroing is required. Be aware of the additional IO workload on vSAN during zeroing.
4. Repeat this process for any remaining shared disks you wish to create.

Creating Eager-Zeroed Disks (ESXi command line)

In vSphere version 6.0 Update 1 or later, (not later than version 6.5, Example: 6.0 Update 3), the vSphere Web Client cannot create eager-zeroed disks on the vSAN datastore. While the vSAN Datastore supports eager-zeroed disks, this functionality is not currently exposed in the vSphere Web Client. To accommodate this limitation in the current release, we must use either PowerCLI or the ESXi command line to create the eager-zeroed disks. This section outlines the ESXi command line method to create disks.

Note: You must enable the local ESXi shell or SSH access to the host and log in as privileged (root) user to complete this procedure.

1. Navigate to the directory of the first virtual machine in the Red Hat cluster:
   
   cd /vmfs/volumes/vsan_datastore/VM_Name
   
   Example:
   
   cd /vmfs/volumes/vsanDatastore/RHCS_0
    
2. Create an Eager Zeroed Thick virtual disk to be shared using vmkfstools:
   
   vmkfstools -csize-W vsan -d eagerzeroedthick `pwd`/vmdk_file_name
   
   Example:
   
   vmkfstools -c 16G –W vsan –d eagerzeroedthick `pwd`/RHCS_0_1.vmdk
    
3. (Optional) As EZT disks created on vSAN are not zeroed automatically, you must use the command vmkfstools -w <path-to-vmdk> to zero out all blocks if zeroing is required. Be aware of the additional IO workload on vSAN during zeroing.

4. Repeat step 2 and 3 for as many shared disks as need to be created.

Adding shared disks to one or more virtual machines

After the eager-zeroed disks are created, you must add them to the remaining Red Hat Enterprise Linux virtual machines using either the vSphere Web Client or PowerCLI.

Note: The virtual disks should be added to the same SCSI positions on each virtual machine. If a disk is in position 1:0 on one virtual machine, it should be in position 1:0 on all virtual machines in the Red Hat cluster.

Using the vSphere Web Client

To add shared disks to one or more virtual machines using the vSphere Web Client:

1. Right-click the appropriate virtual machine and select Edit Settings.
2. Select Existing Hard Disk from the New Device drop-down menu and then click Add.
3. Navigate to the applicable directory and select the disk.
4. Click OK.
5. Expand the New Hard Disk entry and modify the Virtual Device Node, as appropriate.
6. In the Sharing drop down menu, select the Multi-writer option.
7. Change the Disk Mode to Independent-Persistent.
   
   Note: While Independent-Persistent disk mode is not a hard requirement to enable Multi-writer option, the default Dependent disk mode would cause the cannot snapshot shared disk error when a virtual machine snapshot is taken. Use of Independent-Persistent disk mode would allow taking a snapshot of the OS disk while the shared disk would need to be backed up separately by a third-party vendor software.
    
8. Click OK to save the changes.
   
   
9. Repeat this process for the remaining virtual machines and shared disks.

Using PowerCLI

1. Launch PowerCLI and connect to vCenter Server with Connect-VIServer.
2. Add the existing disk with the New-HardDisk cmdlet in the form of:
   
   New-HardDisk -VMVM_Name-DiskPath “[datastore_name]folder/disk_file” -Controllercontroller_name-Persistence IndependentPersistent
   
   For example, to add the disk RHCS_0_1.vmdk on the datastore vsanDatastore in the folder RHCS_0 to the VMRHCS_1 and attach it to SCSI Controller 1, run command:
   
   New-HardDisk -VM RHCS_1 -DiskPath “[vsanDatastore]RHCS_0/RHCS_0_1.vmdk” -Controller “SCSI Controller 1” -Persistence IndependentPersistent
   
   Note: While Independent-Persistent disk mode is not a hard requirement to enable Multi-writer option, the default Dependent disk mode would cause the cannot snapshot shared disk error when a virtual machine snapshot is taken. Use of Independent-Persistent disk mode would allow taking a snapshot of the OS disk while the shared disk would need to be backed up separately by a third-party vendor software.
    
3. Repeat this process for the remaining VMs and shared disks.

For vSphere version prior to 6.0 Update 1

Enabling multi-writer mode for the VMs and Disks using the ESXi shell

The addition of the multi-writer flag is not possible via the vSphere Web Client version prior to 6.0 Update 1. If vSphere 6.0 Update 1 or newer is not installed, use the ESXi shell to enable multi-writer mode on the applicable virtual machines and disks.

Note: As this involves modifying and loading virtual machine configurations, it is recommended that all Red Hat Enterprise Linux virtual machines be registered to the same ESXi host for the purposes of making these changes so we do not have to log in to multiple hosts. The VMs can be distributed throughout the vSphere cluster after making these changes.

1. To enable SCSI bus sharing for a particular disks, run this command:
   
   echo ‘scsi<N>:<D>.sharing = “multi-writer”’ >>path_to_VMX_file
   
   For example, to enable multi-writer for SCSI device 1:0 for the VMRHCS_0 on the datastore vsanDatastore, run this command:
   
   echo ‘scsi1:0.sharing = “multi-writer”’ >> /vmfs/volumes/vsanDatastore/RHCS_0/RHCS_0.vmx
    
2. Complete step 2 for all applicable disks and virtual machines, modifying the SCSI position as appropriate. For example, to share four disks, add all of these entries to the applicable VMX files:
   
   scsi1:0.sharing = "multi-writer"
   scsi1:1.sharing = "multi-writer"
   scsi1:2.sharing = "multi-writer"
   scsi1:3.sharing = "multi-writer"
    
3. Refresh the virtual machine configuration to apply the changes.
   1. Identify the virtual machine to refresh using this command:
      
      vim-cmd vmsvc/getallvms |grep -iVM_name
      
      For example, to list all virtual machine with RAC in the name registered to the host, run command:
      
      vim-cmd vmsvc/getallvms |grep -i RHCS
      
      You see output with the virtual machine registration that includes the VMID in the left-most column. The VMID is a simple integer number.
       
   2. Refresh the configuration:
      
      vim-cmd vmsvc/reloadVMID
      
      For example, if the VMID of one of the Red Hat Enterprise Linux virtual machines is 24, run command:
      
      vim-cmd vmsvc/reload 24
       
4. Reload all applicable virtual machines.

Applying the VM Storage Policies to the shared disks

After the shared disks are created and added to all virtual machines using one of the three methods mentioned above, you must apply the storage policy created for the Red Hat cluster shared disks. The policy must be applied to all applicable disks on all Red Hat Enterprise Linux virtual machines.

1. In vSphere Web Client, right-click the virtual machine and click Edit Settings.
2. Expand the appropriate hard disk entry.
3. Choose the desired VM storage policy from the dropdown list.
4. Verify that the disk mode is Independent – Persistent.
   
   Note: While Independent-Persistent disk mode is not a hard requirement to enable Multi-writer option, the default Dependent disk mode would cause the cannot snapshot shared disk error when a virtual machine snapshot is taken. Use of Independent-Persistent disk mode would allow taking a snapshot of the OS disk while the shared disk would need to be backed up separately by a third-party vendor software.
    
5. Click OK.
   
   
6. Repeat this process for all applicable virtual machines and disks.
   
   Note: If you want to apply a different storage policy to the Red Hat cluster shared disks later, the policy change must be executed against all shared disks and on all virtual machines sharing that disk.

For more information on configuring and managing Red Hat cluster or SUSE Linux Enterprise HA, see:

• Red Hat Enterprise Linux 7 High Availability Add-on Administration
• SUSE Linux Enterprise High Availability Extension 12 Administration Guide