SD card/USB boot device revised guidance
Article ID: 317631
Updated On:
Products
VMware vSphere ESXi
Issue/Introduction
Background
vSphere customers have encountered many issues related to reliability and performance when SD cards/USB media are used as standalone boot devices. Endurance ratings alone cannot guarantee the viability of SD cards/USB media as a boot option. Mitigations and appropriate messaging through KB articles are already in place to help current customers on vSphere 7.x. An ESXi boot device consists of several partitions. One of the boot partitions, the ESX-OSDATA partition is important for ESX operations. The OSData contains the VMTools and scratch areas that are critical regions.
Please refer to the following blog:
https://core.vmware.com/resource/esxi-system-storage-changes
The following diagram illustrates the transition of boot partition layouts used on the boot devices across the different vSphere releases:
Summary
VMware will continue supporting USB/SD card as a boot device through the vSphere 8.0 product release, including the update releases. Both installs and upgrades will be supported on USB/SD cards. The change from the previous guidance is that SD/USB as a standalone device will now be supported on previously certified server platforms. Customers can still however provision an additional persistent device to store the OSData partition, which VMware recommends. ESX will attempt to relocate the critical regions to such a persistent device. Starting from 7.0U3c, if an USB SD card boot device is detected during installs and upgrades, critical regions of the OSData partition such as VMTools and scratch will automatically be moved out. Moreover with vSphere 8.0, the entire OSData partition can be moved out.
The upgrade or install workflows for vSphere 8.0 will ensure that the OSData partition is relocated away from USB/SD card into a persistent device. There will be an automatic fallback to use a VMFS datastore, or a RAMDisk if such a device is not available. Preferably, the SD cards should be replaced with an SSD or another local persistent device as the standalone boot option.
VMware is working very closely with all the major OEMs to ensure that future generations of server platforms do not support USB/SD card as a boot device and that more reliable mechanisms that conform to the VMware device and endurance requirements below for boot devices are provided on servers.
Caution:
For any new hardware procurement, VMware recommends that customers evaluate available options carefully, and plan to use a persistent device for boot that meets ESX endurance requirements as specified in the Guidelines for vSphere Boot Devices section below. While ordering new hardware for their environment, customers should ensure that SD cards/USB media are not chosen for the boot device. Newer OEM servers/systems that carry SD/USB as a boot device will not certify successfully.
Potential issues seen with SD card/USB as a boot option
vSphere customers have encountered many issues related to device reliability when SD cards are used as the boot device. Endurance ratings can no longer guarantee the viability of SD cards as a boot option.
Some of the reasons why SD cards will be unsustainable with the next major release are:
vSphere customers have encountered many issues related to reliability and performance when SD cards/USB media are used as standalone boot devices. Endurance ratings alone cannot guarantee the viability of SD cards/USB media as a boot option. Mitigations and appropriate messaging through KB articles are already in place to help current customers on vSphere 7.x. An ESXi boot device consists of several partitions. One of the boot partitions, the ESX-OSDATA partition is important for ESX operations. The OSData contains the VMTools and scratch areas that are critical regions.
Please refer to the following blog:
https://core.vmware.com/resource/esxi-system-storage-changes
The following diagram illustrates the transition of boot partition layouts used on the boot devices across the different vSphere releases:
Summary
VMware will continue supporting USB/SD card as a boot device through the vSphere 8.0 product release, including the update releases. Both installs and upgrades will be supported on USB/SD cards. The change from the previous guidance is that SD/USB as a standalone device will now be supported on previously certified server platforms. Customers can still however provision an additional persistent device to store the OSData partition, which VMware recommends. ESX will attempt to relocate the critical regions to such a persistent device. Starting from 7.0U3c, if an USB SD card boot device is detected during installs and upgrades, critical regions of the OSData partition such as VMTools and scratch will automatically be moved out. Moreover with vSphere 8.0, the entire OSData partition can be moved out.
The upgrade or install workflows for vSphere 8.0 will ensure that the OSData partition is relocated away from USB/SD card into a persistent device. There will be an automatic fallback to use a VMFS datastore, or a RAMDisk if such a device is not available. Preferably, the SD cards should be replaced with an SSD or another local persistent device as the standalone boot option.
VMware is working very closely with all the major OEMs to ensure that future generations of server platforms do not support USB/SD card as a boot device and that more reliable mechanisms that conform to the VMware device and endurance requirements below for boot devices are provided on servers.
Caution:
For any new hardware procurement, VMware recommends that customers evaluate available options carefully, and plan to use a persistent device for boot that meets ESX endurance requirements as specified in the Guidelines for vSphere Boot Devices section below. While ordering new hardware for their environment, customers should ensure that SD cards/USB media are not chosen for the boot device. Newer OEM servers/systems that carry SD/USB as a boot device will not certify successfully.
Potential issues seen with SD card/USB as a boot option
vSphere customers have encountered many issues related to device reliability when SD cards are used as the boot device. Endurance ratings can no longer guarantee the viability of SD cards as a boot option.
Some of the reasons why SD cards will be unsustainable with the next major release are:
- Reads/Writes to System Storage continue to grow, to the ESX-OSDATA partition. The OSDATA partition is a critical component that is required to be always available and be persistent across reboots. OSDATA will be used in many ways as services and applications such as vSAN and NSX grow. OSDATA is used to store some of the runtime state, and for probes, backups of items such as configuration, timestamps etc. Access to other areas on OSDATA, such as VMTools and scratch portions will also grow. With each new vSphere release, more features will start relying on the OSDATA partition.
- Performance demands and IO loads cannot be continued to be met by SD cards. For example, the demands on writes to the scratch area have been growing. Another concern is the high frequency of IO with bursts such as logs or traces.
- There is no way to reliably check and monitor for the SD card endurance, as these devices usually do not provide support for diagnostics data. Wear-out issues and remaining endurance or lifetime can go undetected. In addition, not just writes but reads are also known to cause wear issues.
- SD/USB devices are prone to wear over time and are thus subject to failures, they are not designed for enterprise class use-cases.
Resolution
Options for vSphere 8.0
Customers will not see any changes for their upgrade workflows. Customers can continue using their existing boot devices, even if it is a USB/SD card boot device, and OSData will be safely relocated to the best available location.
For upgrades and installs, customers now have an option of selecting a persistent device to store the OSData partition.
One of the following options will be used to store the OSData partition:
Options for 7.x
It is strongly recommended that customers start planning to move away completely from SD card usage by following the options listed above for future releases.
However, it is important to note that we are providing exceptions on SD card usage with 7.x and will continue to provide support.
SD cards, even though not an ideal option, can still be used as a standalone boot option to store all partitions including OSDATA, through all vSphere 7.x releases.
Customers on previous versions (for example, 6.x) of vSphere should feel confident to move to 7.0U2c as several critical issues have been fixed with this release, and some workarounds have been provided as default for the VMTools & Scratch areas.
Note that RAMDisk as storage for OSDATA is also an option that is supported through 7.x and vSphere 8.0 releases.
For fixes for critical storage stack issue available in 7.0 Update 2c
See:
https://kb.vmware.com/s/article/83963
https://docs.vmware.com/en/VMware-vSphere/7.0/rn/vsphere-esxi-70u2c-release-notes.html
(Please look for USB device and Storage stack issues fixed with this release)
Customer actions for 7.x
Important for 7.x: Customers must ensure that the following fixes and workarounds are in place.
Guidelines for vSphere Boot Devices
A SATA/SAS/NVMe SSD that meets VMware specified endurance requirements, or HDD is strongly recommended.
Boot Device:
High-Quality Media: PCIe NVMe, SAS, SATA/SATADOM SSD
Industrial grade flash/SSDs, SLC or pSLC
Boot Device Size:
32GB** minimum (must), 128GB** recommended
Endurance requirements:
128 TBW (over 5 years)
IO Performance
At least 100MB/s
Please refer to the ESXi hardware requirements here: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.esxi.install.doc/GUID-DEB8086A-306B-4239-BF76-E354679202FC.html
FAQ
Based on the boot device used, could an install or upgrade to 7.x or vSphere 8.0 be blocked?
VMware will not block ESXi install or upgrade. It will, however, try to relocate critical regions of the boot device into persistent storage.
What is VMware telling its partners?
VMware is working very closely with its OEM partners. All major OEM server vendors today support options such as an M.2, BOSS, NVMe, SATA with a high-endurance boot device in their current generation of servers. In addition, OEM server vendors have agreed to stop supporting USB/SD card-based boot device with servers having the newer generation of CPU platforms such as Sapphire Rapids and Genoa. Any new server storage for boot devices must comply with vSphere 8.0 endurance requirements. VMware has also requested OEM partners to remove any option to purchase USB/SD as boot device, such as with web-based ordering tools, sales tools, and with channels or resellers.
Is there any certification impact?
Apart from already certified servers, VMware server certification process will block any new server certification for servers that have a USB/SD card as the boot device.
In addition, VMware has stopped certifying flash devices for boot device, and will stop listing them as approved devices starting in vSphere 8.0. Devices already certified for previous vSphere releases will continue to be supported in vSphere 8.0 and future releases until device EOL (End of Life)
(See http://partnerweb.vmware.com/programs/server_docs/Approved%20Flash%20Devices.pdf)
Why should customers move now to a persistent boot device for any new servers?
There are two main reasons to recommend a move now rather than later:
1) To avoid managing 2 separate devices to store boot partitions - SD/USB and another device
2) Future-proofing customer infrastructure
What is the impact of storing OSData on RAMDisk?
With vSphere 8.0, RAMDisk will be used as a last resort option to store OSData. Customers can ensure that if deemed important, traces and logs are configured to be stored separately. Similarly, coredumps can be configured to be stored on alternate locations (see https://kb.vmware.com/s/article/2077516)
Can application data or VMFS partitions be stored on USB/SD card boot devices?
VMware strongly recommends that if USB/SD card is used as the boot device, customers not store any application (VM) data intentionally on boot devices. Some use cases can lead to a lot of Reads/Writes, for example, in some instances, databases are stored on scratch, or new VMFS partitions are created on USB/SD card. Such use of the USB/SD card boot device should be avoided.
How is the VMTools location stored and controlled?
VMTools region is a part of OSData and will be moved to any place where OSData is stored. However, it can be controlled by the ProductLocker advanced setting. At install/upgrade time, it is in the boot device VMFS-L partition (either the LOCKER partition on USB/SDCard, or the OSDATA partition if the OSDATA is installed to non-USB). When the system boots up, the system-storage jumpstart will check if it is in LOCKER on USB/SD device, and if there is a local OSDATA partition, it will be moved there. The customer can change the location anywhere else by setting the ProductLocker value. (Also see https://kb.vmware.com/s/article/2129825)
What type of devices and interfaces is this KB referring to?
We are referring to devices that are unreliable, such as SD cards with unpredictable and unreportable endurance.
We are also referring to unreliable interfaces such as USB.
Please note that when we refer to SAS, SATA (including SATADOM), NVMe, etc., we imply that these devices must be supported in their native formats and not through any conversions (such as NVMe or SAS to USB)
Can OSData be pointed to use a VMFS partition?
VMware is now allowing for such use. On an install/upgrade to vSphere 8.0, OSData can be stored on a VMFS volume with sufficient space (minimum 32GB**)
Can OSData be pointed to use a remote disk?
VMware cautions against configuring and using OSData on a shared remote LUN during install/upgrade. The management of such a use-case is prone to risk as there is no way to control how such LUNs will be coordinated for access especially when multiple hosts can be configured to use the same LUN.
Can the boot LUN be expanded?
Technically, expansion of the LUN can only work if there is no other partition after the VMFS-L OSDATA partition. Usually there is a datastore partition there when upgrading from 6.x to 7.0, and that prevents stretching OSData when you have a larger LUN. If you are upgrading from 6.x to 7.0:
For more FAQ, see ESXi System Storage FAQ
** GB here equals 1024x1024x1024 (2^30=1,073,741,824) bytes as opposed to 10^9 bytes (1,000,000,000)
Customers will not see any changes for their upgrade workflows. Customers can continue using their existing boot devices, even if it is a USB/SD card boot device, and OSData will be safely relocated to the best available location.
For upgrades and installs, customers now have an option of selecting a persistent device to store the OSData partition.
One of the following options will be used to store the OSData partition:
- Local Persistent Disk
-
- If a local persistent device is not already available, add another persistent device. This persistent local boot device can be industrial grade SAS, SATA or PCIe NVMe disk (32GB** minimum, 128GB** recommended)
- SD Card/USB Media + Local Persistent Disk
- SAN/NAS LUN for the entire system storage
- VMFS partitions
- RAMDisk
Options for 7.x
It is strongly recommended that customers start planning to move away completely from SD card usage by following the options listed above for future releases.
However, it is important to note that we are providing exceptions on SD card usage with 7.x and will continue to provide support.
SD cards, even though not an ideal option, can still be used as a standalone boot option to store all partitions including OSDATA, through all vSphere 7.x releases.
Customers on previous versions (for example, 6.x) of vSphere should feel confident to move to 7.0U2c as several critical issues have been fixed with this release, and some workarounds have been provided as default for the VMTools & Scratch areas.
Note that RAMDisk as storage for OSDATA is also an option that is supported through 7.x and vSphere 8.0 releases.
For fixes for critical storage stack issue available in 7.0 Update 2c
See:
https://kb.vmware.com/s/article/83963
https://docs.vmware.com/en/VMware-vSphere/7.0/rn/vsphere-esxi-70u2c-release-notes.html
(Please look for USB device and Storage stack issues fixed with this release)
Customer actions for 7.x
Important for 7.x: Customers must ensure that the following fixes and workarounds are in place.
- Customers must move to 7.0U2C or above. Critical fixes, as outlined in the release notes link above, are available with vSphere 7.0 Update 2c, 7.0 update 3 and above releases
- For customers planning to upgrade to the next release, ESX will automatically move VMTools to RAMDisk. See https://kb.vmware.com/s/article/83376
- For VMware customers on vSphere 7.0 update 2c, there are options to manually configure VMTools to move RAMDisk. See https://kb.vmware.com/s/article/83782
- For customers planning to run 7.X releases, some of the workarounds customers must apply are given below:
- Reduce IOs to SD Card by reducing access to VMTools and scratch areas of OSDAT.
-
For VMTools area of OSDATA, by default, ESX will try to find a local persistent device or a SAN LUN to store VMTools. Customers can also enable options to move VMTools out of SD card into RAMDisk. See https://kb.vmware.com/s/article/83376 and https://kb.vmware.com/s/article/2149257
- For Scratch area, if there is no local HDD, SSD, NVMe device, or M.2 found, ESX will move the scratch area as well to RAMDisk automatically.
- Customers can also configure and move Scratch area to SAN or NFS. See: https://kb.vmware.com/s/article/1033696
- In addition, customers also have options to configure remote syslog to move logs out of OSDATA. See: https://kb.vmware.com/s/article/200332
- Customers can expect to see warnings with vSphere 7.0U3 shown in https://kb.vmware.com/s/article/85615
Guidelines for vSphere Boot Devices
A SATA/SAS/NVMe SSD that meets VMware specified endurance requirements, or HDD is strongly recommended.
Boot Device:
High-Quality Media: PCIe NVMe, SAS, SATA/SATADOM SSD
Industrial grade flash/SSDs, SLC or pSLC
Boot Device Size:
32GB** minimum (must), 128GB** recommended
Endurance requirements:
128 TBW (over 5 years)
IO Performance
At least 100MB/s
Please refer to the ESXi hardware requirements here: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.esxi.install.doc/GUID-DEB8086A-306B-4239-BF76-E354679202FC.html
| Supported=Yes, Not Supported=No | 7.0U2c & 7.0U3 | vSphere 8.0 |
| SD card as boot device | Yes, Not recommended | Yes, Not recommended |
| Use local Persistent disk for ESX-OSDATA | No | Yes |
| Move VMTools area away from SD card/USB (out of ESX-OSDATA) | Yes, Recommended | No (Not required since OSDATA itself will be moved out) |
| Move Scratch area away from SD card/USB (out of ESX-OSDATA) | Yes, Recommended | No (Not required since OSDATA itself will be moved out) |
| Move ESX-OSDATA out of SD card | No | Yes |
| RAMDisk as storage for boot areas like VMTools & scratch | Yes, Not Recommended | Yes, Not Recommended |
| RAMDisk as storage for ESX-OSDATA partition | No | Yes, Not recommended |
| ESX-OSDATA partition can be carved out of existing VMFS datastore (share space) | No | Yes |
FAQ
Based on the boot device used, could an install or upgrade to 7.x or vSphere 8.0 be blocked?
VMware will not block ESXi install or upgrade. It will, however, try to relocate critical regions of the boot device into persistent storage.
What is VMware telling its partners?
VMware is working very closely with its OEM partners. All major OEM server vendors today support options such as an M.2, BOSS, NVMe, SATA with a high-endurance boot device in their current generation of servers. In addition, OEM server vendors have agreed to stop supporting USB/SD card-based boot device with servers having the newer generation of CPU platforms such as Sapphire Rapids and Genoa. Any new server storage for boot devices must comply with vSphere 8.0 endurance requirements. VMware has also requested OEM partners to remove any option to purchase USB/SD as boot device, such as with web-based ordering tools, sales tools, and with channels or resellers.
Is there any certification impact?
Apart from already certified servers, VMware server certification process will block any new server certification for servers that have a USB/SD card as the boot device.
In addition, VMware has stopped certifying flash devices for boot device, and will stop listing them as approved devices starting in vSphere 8.0. Devices already certified for previous vSphere releases will continue to be supported in vSphere 8.0 and future releases until device EOL (End of Life)
(See http://partnerweb.vmware.com/programs/server_docs/Approved%20Flash%20Devices.pdf)
Why should customers move now to a persistent boot device for any new servers?
There are two main reasons to recommend a move now rather than later:
1) To avoid managing 2 separate devices to store boot partitions - SD/USB and another device
2) Future-proofing customer infrastructure
- SD/USB options eventually removed
- New platforms will not be allowed to be certified for USB/SD cards
- Sapphire Rapids/Genoa platforms not certified
- OEMs have been asked to take specific actions to not support USB/SD cards for their future platforms and in their sales process.
What is the impact of storing OSData on RAMDisk?
With vSphere 8.0, RAMDisk will be used as a last resort option to store OSData. Customers can ensure that if deemed important, traces and logs are configured to be stored separately. Similarly, coredumps can be configured to be stored on alternate locations (see https://kb.vmware.com/s/article/2077516)
Can application data or VMFS partitions be stored on USB/SD card boot devices?
VMware strongly recommends that if USB/SD card is used as the boot device, customers not store any application (VM) data intentionally on boot devices. Some use cases can lead to a lot of Reads/Writes, for example, in some instances, databases are stored on scratch, or new VMFS partitions are created on USB/SD card. Such use of the USB/SD card boot device should be avoided.
How is the VMTools location stored and controlled?
VMTools region is a part of OSData and will be moved to any place where OSData is stored. However, it can be controlled by the ProductLocker advanced setting. At install/upgrade time, it is in the boot device VMFS-L partition (either the LOCKER partition on USB/SDCard, or the OSDATA partition if the OSDATA is installed to non-USB). When the system boots up, the system-storage jumpstart will check if it is in LOCKER on USB/SD device, and if there is a local OSDATA partition, it will be moved there. The customer can change the location anywhere else by setting the ProductLocker value. (Also see https://kb.vmware.com/s/article/2129825)
What type of devices and interfaces is this KB referring to?
We are referring to devices that are unreliable, such as SD cards with unpredictable and unreportable endurance.
We are also referring to unreliable interfaces such as USB.
Please note that when we refer to SAS, SATA (including SATADOM), NVMe, etc., we imply that these devices must be supported in their native formats and not through any conversions (such as NVMe or SAS to USB)
Can OSData be pointed to use a VMFS partition?
VMware is now allowing for such use. On an install/upgrade to vSphere 8.0, OSData can be stored on a VMFS volume with sufficient space (minimum 32GB**)
Can OSData be pointed to use a remote disk?
VMware cautions against configuring and using OSData on a shared remote LUN during install/upgrade. The management of such a use-case is prone to risk as there is no way to control how such LUNs will be coordinated for access especially when multiple hosts can be configured to use the same LUN.
Can the boot LUN be expanded?
Technically, expansion of the LUN can only work if there is no other partition after the VMFS-L OSDATA partition. Usually there is a datastore partition there when upgrading from 6.x to 7.0, and that prevents stretching OSData when you have a larger LUN. If you are upgrading from 6.x to 7.0:
- If there is a datastore on the LUN, then no expansion can happen on upgrade
- If you make the LUN larger while in 6.x, followed by upgrading and reboot, then when 7.0 comes up it will repartition the boot device such that system-storage partitions and will occupy the space (up to 128GB**) of the LUN => this is the only case where expansion can happen
For more FAQ, see ESXi System Storage FAQ
** GB here equals 1024x1024x1024 (2^30=1,073,741,824) bytes as opposed to 10^9 bytes (1,000,000,000)
Feedback
Yes
No
Powered by
