VMware vSphere ESXi can use locally attached SSDs (Solid State Disk) and flash devices in multiple ways. Since SSDs offer much higher throughput and much lower latency than traditional magnetic hard disks the benefits are clear. While offering lower throughput and higher latency, flash devices such as USB or SATADOM can also be appropriate for some use cases. The potential drawback to using SSDs and flash device storage is that the endurance can be significantly less than traditional magnetic disks and it can vary based on the workload type as well as factors such as the drive capacity, underlying flash technology, etc.

 

This KB outlines the minimum SSD and flash device recommendations based on different technologies and use case scenarios.

A non-exhaustive survey of various usage models in vSphere environment are listed below.

• Host swap cache
  • This usage model has been supported since vSphere 5.1 for SATA and SCSI connected SSDs. USB and low end SATA or SCSI flash devices are not supported.
  • The workload is heavily influenced by the degree of host memory over commitment.

• Regular datastore
  • A (local) SSD is used instead of a hard disk drive.
  • This usage model has been supported since vSphere 6.0 for SATA and SCSI connected SSDs.
  • There is currently no support for USB connected SSDs or for low end flash devices regardless of connection type.
     
• vSphere Flash Read Cache (aka Virtual Flash)
  • This usage model has been supported since vSphere 5.5 for SATA and SCSI connected SSDs.
  • There is no support for USB connected SSDs or for low end flash devices.
     
• vSAN
  • This usage model has been supported since vSphere 5.5 for SATA and SCSI SSDs. For more information, see the vSAN Hardware Quick Reference Guide.
• vSphere ESXi Boot Disk
  • A USB flash drive or SATADOM or local SSD can be chosen as the install image for ESXi, the vSphere hypervisor, which then boots from the flash device.
  • This usage model has been supported since vSphere 3.5 for USB flash devices and vSphere 4.0 for SCSI/SATA connected devices.
  • Installation to SATA and SCSI connected SSD, SATADOM and flash devices creates a full install image which includes a logging partition (see below) whereas installation to a USB device creates a boot disk image without a logging partition.
     
• vSphere ESXi Coredump device
  • The default size for the coredump partition is 2.5 GiB which is about 2.7 GB and the installer creates a coredump partition on the boot device device for vSphere 5.5 and above. After installation the partition can be resized if necessary using partedUtil. For more information, see the vSphere documentation.
  • Any SATADOM or SATA/SCSI SSD may be configured with a coredump partition.
  • This usage model has been supported from vSphere 3.5 for boot USB flash devices and since vSphere 4.0 for any SATA or SCSI connected SSD that is local.
  • This usage model also applies to Autodeploy hosts which have no boot disk.
     
• vSphere ESXi Logging device
  • A SATADOM or local SATA/SCSI SSD is chosen as the location for the vSphere logging partition (/scratch partition). This partition may be but need not be on the boot disk and this applies to Autodeploy hosts which lack a boot disk.
  • This usage model has been supported since vSphere 6.0 for any SATA or SCSI connected SSD that is local. SATADOMs that meet the requirement set forth in Table 1 are also supported.
  • This usage model can be supported in a future release of vSphere for USB flash devices that meet the requirement set forth in Table 1.

 

The flash industry often uses Tera Bytes Written (TBW) as a benchmark for SSD endurance. TBW is the number of terabytes that can be written to the device over its useful life. Most devices have distinct TBW ratings for sequential and random IO workloads, with the latter being much lower due to Write Amplification Factor (WAF) (defined below). Other measures of endurance commonly used are DWPD (Drive Writes Per Day) and P/E (Program/Erase) cycles.

 

Conversion formulas are provided here:

• Converting DWPD (Drive Writes Per Day) to TBW (Terabytes Written):
  • TBW = DWPD * Warranty (in Years) * 365 * Capacity (in GB) / 1,000 (GB per TB)
     
• Converting Flash P/E Cycles per Cell to TBW (Terabytes Written):
  • TBW = Capacity (in GB) * (P/E Cycles per Cell) / (1,000 (GB per TB) * WAF)

WAF is a measure of the induced writes caused by inherent properties of flash technology. Due to the difference between the storage block size (512 bytes), the flash cell size (typically 4KiB or 8KiB bytes) and the minimum flash erase size of many cells one write can force a number of induced writes due to copies, garbage collection, etc. For sequential workloads typical WAFs fall in the range of single digits while for random workloads WAFs can approach or even exceed 100. Table 1 contains workload characterization for the various workloads excepting the Datastore and vSphere Flash Read Cache workloads which depend on the characteristics of the Virtual Machines workloads being run and thus cannot be characterized here. A WAF from the table can be used with the above P/E to TBW formula.

Performance and endurance are critical factors when selecting SSDs. For each of the above use cases, the amount and frequency of data written to the SSD or flash device determines the minimum requirement for performance and endurance by ESXi. In general, SSDs can be deployed in all of the above use cases, but (low end) flash devices including SATADOM can only be deployed in some. In the table below: ESXi write endurance requirements are stated in terms of Terabytes written (TBW) for a JEDEC random workload. There are no specific ESXi performance requirements, but products built on top of ESXi such as vSAN may have their own requirements.

Important: ALL of the TBW requirements in Table 1 are stated in terms of the JEDEC Enterprise Random Workload² because vendors commonly publish only a single endurance number, the random TBW. Vendors may provide a sequential number if asked and such a number together with a measured or worst case WAF can be used to calculate an alternative sequential TBW if the total workload writes in 5 years are known³. Failure of the boot or coredump device is catastrophic for vSphere so VMware requires use of random TBW requirement for boot and coredump use cases.

1. Co-located refers to the case where 2 use cases are partitions on the same device, thereby sharing flash cells.
2. See JESD218A and JESD219 for the Endurance Test Method and Enterprise Workload definitions, respectively.
3. Contact VMware for detailed instructions to compute the alternative sequential TBW for the logging use case.

The size requirement for the ESXi coredump device scales with the size of the host DRAM and also usage of vSAN. vSphere ESXi installations with an available local datastore are advised to use dump to file which automatically reserves the needed space on the local datastore but flash media in general and installations using vSAN in particular will often lack a local datastore and thus require a coredump device. While the default size of 2560 MiB suffices for a host with 1 TiB of DRAM not running vSAN, if vSAN is in use the default size is almost always insufficient.

 

Table 2 gives the recommended partition size in units of MiB and corresponding flash drive size recommendation. If these recommendations are ignored and ESXi crashes then the coredump may be truncated. The footnotes explain the calculation, and note that if using vSAN the values from the right side of the table must be added to those from the left side of the table. To override the default or change coredump partition size after installing, please consult vSphere release documentation or contact VMware.

In general, if the SSD’s host controller interface is supported by a certified IOVP driver, then the SSD drive is supported for ESXi provided that the media meets the endurance requirements above. Therefore, there are no specific vSphere restrictions against SATADOM and M.2 provided, again, that they adhere to the endurance requirements set forth in Table 1 above.

 

For USB storage devices (such as flash drives, SD cards plus readers, and external disks of any kind) the drive vendor must work directly with system manufacturers to ensure that the drives are supported for these systems. USB flash devices and SD cards plus readers are qualified pairwise with USB host controllers and it is possible for a device to fail certification with one host controller but pass with another. VMware strictly recommends that customers who do not have a preinstalled system either obtain a USB flash drive directly from their OEM vendor or purchase a model that has been certified for use with their server.