Frequently Asked Questions for vStorage APIs for Array Integration
search cancel

Frequently Asked Questions for vStorage APIs for Array Integration

book

Article ID: 318941

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

This article provides answers to frequently asked questions about vStorage APIs for Array Integration (VAAI).

Environment

VMware ESXi 4.1.x Embedded
VMware vSphere ESXi 6.0
VMware vSphere ESXi 5.0
VMware vSphere ESXi 5.1
VMware vSphere ESXi 5.5
VMware ESXi 4.1.x Installable
VMware ESX 4.1.x

Resolution

vStorage APIs for Array Integration is a feature introduced in ESXi/ESX 4.1 that provides hardware acceleration functionality. It enables your host to offload specific virtual machine and storage management operations to compliant storage hardware. With the storage hardware assistance, your host performs these operations faster and consumes less CPU, memory, and storage fabric bandwidth.

For more information about vStorage APIs for Array Integration and when it is used, see the Storage Hardware Acceleration section in the Configuration Guide for your version of ESXi/ESX.

What does VAAI require?

VAAI requires:
  • ESXi/ESX 4.1 or later
  • Standard, Enterprise, Enterprise Plus or Remote Office Branch Office licensing for ESXi/ESX hosts
  • Storage arrays that support VAAI storage-based hardware acceleration

    Notes:
    • ESXi/ESX 4.1 does not support hardware acceleration with NAS storage devices.
    • Support for NAS storage devices was introduced in ESXi 5.x.

How do I know if my storage array supports VAAI?

To determine if your storage array supports VAAI, see the Hardware Compatibility List or consult your storage vendor. To enable the hardware acceleration on the storage array, check with your storage vendor. Some storage arrays require explicit activation of hardware acceleration support.

Note:
  • In VMware HCL version 2, there are options under the Features Category and Features fields where arrays can be searched against to verify if they are VAAI-certified.

What are the fundamental operations used by VAAI and what are their corresponding Advanced Settings?

VAAI uses these fundamental operations:
  • Atomic Test & Set (ATS), which is used during creation and locking of files on the VMFS volume
  • Clone Blocks/Full Copy/XCOPY, which is used to copy or migrate data within the same physical array
  • Zero Blocks/Write Same, which is used to zero-out disk regions
  • Thin Provisioning in ESXi 5.x and later hosts, which allows the ESXi host to tell the array when the space previously occupied by a virtual machine (whether it is deleted or migrated to another datastore) can be reclaimed on thin provisioned LUNs.
  • Block Delete in ESXi 5.x and later hosts, which allows for space to be reclaimed using the SCSI UNMAP feature. For more information on Block Delete/SCSI UNMAP, see Using vmkfstools to reclaim VMFS deleted blocks on thin-provisioned LUNs (2014849).
The fundamental operations are controlled by these advanced settings:

Advanced Parameter nameDescription
HardwareAcceleratedLockingAtomic Test & Set (ATS), which is used during creation of files on the VMFS volume
HardwareAcceleratedMoveClone Blocks/Full Copy/XCOPY, which is used to copy data
HardwareAcceleratedInitZero Blocks/Write Same, which is used to zero-out disk regions

Note: For Thin Provisioning, enabling/disabling occurs on the array and not on the ESXi host.

What improvements have been made to VAAI for ESXi 5.x?

To encourage customers to use larger and fewer datastores, support for Thin Provisioning VAAI primitive has been added in ESXi 5.0.

In ESXi 5.x, support for NAS Hardware Acceleration is included with support for these primitives:
  • Full File Clone – Like the Full Copy VAAI primitive provided for block arrays, this Full File Clone primitive enables virtual disks to be cloned by the NAS device.
  • Native Snapshot Support – Allows creation of virtual machine snapshots to be offloaded to the array.
  • Extended Statistics – Enables visibility to space usage on NAS datastores and is useful for Thin Provisioning.
  • Reserve Space – Enables creation of thick virtual disk files on NAS.

    Note: Previously, the only supported VMDK type that could be created on NAS was thin.
Support for Block Delete to reclaim space use on thin provisioned LUNs on support arrays is also included in ESXi 5.x. For more information on Block Delete/SCSI UNMAP, see Using vmkfstools to reclaim VMFS deleted blocks on thin-provisioned LUNs (2014849).


How do I know if VAAI is enabled?

To determine if VAAI is enabled using the vSphere Client:
  1. In the vSphere Client inventory panel, click the host.
  2. Click the Configuration tab, then click Advanced Settings under Software.
  3. Check that these options are set to 1 (enabled):

    DataMover.HardwareAcceleratedMove
    DataMover.HardwareAcceleratedInit
    VMFS3.HardwareAcceleratedLocking


    Note: These options are enabled by default.
On ESXi/ESX 4.x hosts, to determine if VAAI is enabled using the service console in ESX or the vCLI in ESXi, use the esxcfg-advcfg command to check if the options are set to 1 (enabled):

# esxcfg-advcfg -g /DataMover/HardwareAcceleratedMove
# esxcfg-advcfg -g /DataMover/HardwareAcceleratedInit
# esxcfg-advcfg -g /VMFS3/HardwareAcceleratedLocking


On ESXi 5.x hosts, to determine if VAAI is enabled, run these commands and check if Int Value is set to 1 (enabled):

# esxcli system settings advanced list -o /DataMover/HardwareAcceleratedMove
# esxcli system settings advanced list -o /DataMover/HardwareAcceleratedInit
# esxcli system settings advanced list -o /VMFS3/HardwareAcceleratedLocking


You see output similar to:

Path: /VMFS3/HardwareAcceleratedLocking
Type: integer
Int Value: 1 <-- set to 1 if enabled
Default Int Value: 1
Min Value: 0
Max Value: 1
String Value:
Default String Value:
Valid Characters:
Description: Enable hardware accelerated VMFS locking (requires compliant hardware)


Note: For Thin Provisioning, enabling/disabling occurs on the array and not on the ESXi host.


When enabling or disabling VAAI settings, is a reboot required?

No, a host reboot is not required. The changes made may or may not be effective immediately depending on use case. For details please see below table:.

Use Case

Effective

ESXi Host reboot Needed

Volume mount/unmount Needed

Remark

Disabling Hardware Accelerated Move (XCOPY) in ESXi

Immediate

No

No

Disabling Hardware Accelerated Init (WRITESAME) in ESXi

Immediate

No

No


Disabling ATS Usage at ESXi host level - For upgraded VMFS datastores a.k.a Non ATS-Only VMFS datastores

Immediate

No

No

Disabling ATS Usage at ESXi host level - for ATS-Only VMFS datastores

No

No

Yes

Not effective unless the datastore is unmounted and mounted back.
If you reboot the ESXi host to achieve the mount/unmount you will potentially end up in a mixture i.e. ATS+SCSI locking till all the host are rebooted which is not recommended

Disabling ATS usage per VMFS datastore - For upgraded VMFS datastores a.k.a Non ATS-Only VMFS datastores

NA

NA

NA

There is no option/need to disable ATS on such datastore.

Disabling ATS usage per VMFS datastore - For ATS-Only VMFS datastores

No

No

Yes

Datastore must be unmounted before executing command to disable ATS usage.

Disabling ATS Usage only for VMFS heartbeats

Immediate

No

No



What happens if I have VAAI enabled on the host but some of my disk arrays do not support it?

When storage devices do not support or provide only partial support for the host operations, the host reverts to its native methods to perform the unsupported operations.

Are there any caveats that I should be aware of in ESXi 5.x?

Use of the Atomic Test-and-Set (ATS) primitive depends on the type of filesystem:

On VAAI HardwareNew VMFS-5Upgraded VMFS-5VMFS-3
Single-extent datastoreATS only [1]ATS, but fall back to SCSI-2 reservationsATS, but fall back to SCSI-2 reservations
Multi-extent datastoreOnly allow spanning on ATS hardware [2]ATS, except when locks on non-headATS, except when locks on non-head

Notes:
  • If a new VMFS-5 is created on a non-ATS storage device, SCSI-2 reservations will be used.
  • When creating a multi-extent datastore where ATS is used, the vCenter Server will filter out non-ATS devices, so that only devices that support the ATS primitive can be used.

Are there any caveats that I should be aware of in ESXi/ESX 4.x?

There are a few reasons why a hardware-accelerated operation may fail.

For any primitive that the array does not implement, the array returns an error, which triggers the ESX data mover to attempt the operation using software data movement. In the case of ATS, it reverts to using SCSI Reservations.

VAAI hardware offload cannot be used when:
  • The source and destination VMFS volumes have different block sizes
  • The source file type is RDM and the destination file type is non-RDM (regular file)
  • The source VMDK type is eagerzeroedthick and the destination VMDK type is thin
  • The source or destination VMDK is any kind of sparse or hosted format
  • Cloning a virtual machine that has snapshots because this process involves consolidating the snapshots into the virtual disks of the target virtual machine.
  • The logical address and/or transfer length in the requested operation is not aligned to the minimum alignment required by the storage device (all datastores created with the vSphere Client are aligned automatically)
  • The VMFS datastore has multiple LUNs/extents spread across different arrays
Notes:
  • When using enhanced vMotion (move both VM and datastore simultaneously), the source and destination hosts must both see the source and destination datastores, or the file transfer defaults to NFC over the the management network, instead of VAAI.
  • Hardware cloning between arrays (even if within the same VMFS datastore) does not work.
  • For more information on supportability with Horizon View, see View Composer API for Array Integration (VCAI) support in VMware Horizon View (2061611).

What do I need to know about the Hardware Acceleration Support Status?

If you go to Host > Configuration > Storage, you can see the Hardware Acceleration Status in the panel on the right side.

For each storage device and datastore, the vSphere Client displays the hardware acceleration support status in the Hardware Acceleration column of the Devices view and the Datastores view.

The status values are Unknown, Supported, and Not Supported. The initial value is Unknown. The status changes to Supported after the host successfully performs the offload basic operations. If the offload operation fails, the status changes to Not Supported.

To determine if your storage device supports VAAI, test the Full Copy VAAI primitive:

  1. Using the vSphere Client, browse the datastore and locate a virtual disk (VMDK) of at least 4 MB that is not in use.
  2. Copy the virtual disk to a new file.
  3. Check the Hardware Acceleration status to verify that it changes from Unknown to either Supported or Not Supported.
Note: VAAI primitives can also be tested by creating a virtual machine with at least one new virtual disk, or cloning a virtual machine.


Can I check the VAAI status from the command line?
  • On ESXi 5.x

    To check the VAAI status, run this command:

    # esxcli storage core device vaai status get

    You see output similar to:

    mpx.vmhba1:C0:T1:L0
    VAAI Plugin Name:
    ATS Status: unsupported
    Clone Status: unsupported
    Zero Status: unsupported
    Delete Status: unsupported

    mpx.vmhba1:C0:T0:L0
    VAAI Plugin Name:
    ATS Status: unsupported
    Clone Status: unsupported
    Zero Status: unsupported
    Delete Status: unsupported


  • On ESXi/ESX 4.x

    To check the VAAI status, run this command:

    # esxcfg-scsidevs -l | egrep "Display Name:|VAAI Status:"

    You see output similar to:

    Display Name: Local VMware Disk (mpx.vmhba2:C0:T0:L0)
    VAAI Status: unsupported
    Display Name: Local USB CD-ROM (mpx.vmhba32:C0:T0:L0)
    VAAI Status: unknown
    Display Name: Local TEAC CD-ROM (mpx.vmhba3:C0:T0:L0)
    VAAI Status: unknown
    Display Name: DGC Fibre Channel Disk (naa.6006016030801c00f8506073d0d7de11)
    VAAI Status: unknown
    Display Name: DGC Fibre Channel Disk (naa.6006016030801c00f9506073d0d7de11)
    VAAI Status: unknown

Are changes to VAAI settings logged?

Changes to VAAI advanced settings (for example, enabling or disabling VAAI) are recorded in the VMkernel logs at /var/log/vmkernel (for ESXi 5.0 vmkernel.log) or /var/log/messages.

After enabling VAAI using the advanced settings, the VMkernel log contains entries similar to:

cpuN:1234)Config: 297: "HardwareAcceleratedMove" = 1, Old Value: 0, (Status: 0x0)
cpuN:1234)Config: 297: "HardwareAcceleratedInit" = 1, Old Value: 0, (Status: 0x0)
cpuN:1234)Config: 297: "HardwareAcceleratedLocking" = 1, Old Value: 0, (Status: 0x0)


After disabling VAAI using the advanced settings, the VMkernel log contains entries similar to:

cpuN:1234)Config: 297: "HardwareAcceleratedMove" = 0, Old Value: 1, (Status: 0x0)
cpuN:1234)Config: 297: "HardwareAcceleratedInit" = 0, Old Value: 1, (Status: 0x0)
cpuN:1234)Config: 297: "HardwareAcceleratedLocking" = 0, Old Value: 1, (Status: 0x0)

VAAI For NAS Storage

In ESXi 5.x, NAS Hardware Acceleration was introduced with support for these primitives:

· Full File Clone – Like the Full Copy VAAI primitive provided for block arrays, this Full File Clone primitive enables virtual disks to be cloned by the NAS device.

· Native Snapshot Support – Allows creation of virtual machine snapshots to be offloaded to the array.

· Extended Statistics – Enables visibility to space usage on NAS datastores and is useful for Thin Provisioning.

· Reserve Space – Enables creation of thick virtual disk files on NAS.

With NAS storage devices, the hardware acceleration integration is implemented through vendor-specific NAS plug-ins. These plug-ins are typically created by vendors and are distributed as VIB packages through a website. No claim rules are required for the NAS plug-ins to function.

Installing the NAS Plugin


1. Place the host into maintenance mode
2. Set the host acceptance level:
# esxcli software acceptance set–-level=value

The host acceptance level must be the same or less restrictive than the acceptance level of any VIB you want to add to the host. The value can be one of the following:

· VMwareCertified

· VMwareAccepted

· PartnerSupported

· CommunitySupported

3. Install the VIB package:

# esxcli software vib install –-viburl=URL

The URL specifies the URL to the VIB package to install. http:, https:, ftp:, and file: are supported.

4. Restart the host for the installation to take effect

Verify Hardware Acceleration Status for NAS

You can verify the NAS hardware acceleration status using the following command:

# esxcli storage nfs list

The Hardware Acceleration column in the output shows the status

Disabling VAAI when using NAS storage

To disable NAS VAAI functionality the plugin must be removed using the below commands.

1. Log into ESXi using SSH/DCUI and execute command:

# esxcli software vib remove –-vibname=name
For Example: # esxcli software vib remove –-vibname=NetAppNasPlugin


2. Verify that the plugin has been removed using command:

esxcli software vib list

3. Reboot the ESXi host to take effect.

Note: The advanced configuration options for Block VAAI mentioned above (DataMover.HardwareAcceleratedMove, DataMover.HardwareAcceleratedInit, VMFS3.HardwareAcceleratedLocking) do not affect NAS VAAI functionality


Additional Information

For more information, see the vSphere Storage Guide for your release of vSphere:
This is a list of commonly used SCSI opcodes related to VAAI operations:

0x93WRITE SAME(16)
0x41WRITE SAME(10)
0x42UNMAP
0x89SCSI COMPARE and WRITE - ATS
0x83EXTENDED COPY

Occasionally, failures during VAAI operations would be logged in the vmkernel.log file under /var/log as shown in this example:

Cmd(0x41248092e240) 0x42, CmdSN 0x13bb23 to dev "naa.xxxxxxxx" failed H:0x0 D:0x2

How to reclaim VMFS deleted blocks on thin-provisioned LUNs
vStorage APIs for Array Integration 常见问题
esxtop コマンドを使用して、 VAAI の監視および状態確認をする方法