Testing CPU Max Turbo Boost Frequency in ESXi

Products

VMware vSphere ESXi

Issue/Introduction

The Intel® Turbo Boost Technology works depending on the workload of the processor; whenever the program requires speeding up the processor speed it will throttle to the maximum allowed.

In some particular scenarios, such as proof-of-concepts setups, end users would like VMware to provide guidance which can be used to verify CPU Max Turbo Frequency in ESXi.

Symptoms:

CPU Max Turbo Boost Frequency keeps showing 125% compared to the physical processor base frequency.
ESXi power management policy is set to "high performance".

Cause

To increase the maximum CPU core frequency, you not only need to rely on the physical servers BIOS parameters, but will also have to adjust the ESXi power management policy to "Balanced" or any other non-"High Performance" policy.

Why do these changes result in an increase in core frequency? This is because the option to run a single CPU core in Turbo Boost depends on multiple factors, such as:

Type of Workload
Number of active cores on the physical CPU
Estimated power and current consumption
processor / core temperature

If the ESXi hosts power management policy is set to "High Performance", it is very likely that all cores on the physical CPU are in a permanent active state, which means high power consumption and therefore higher core temperatures. Turbo Boost will however only be enabled when the CPU has no reason to estimate potential power over-consumption, nor risks to run to hot (which would then result in thermal throttling). Cores on an ESXi running with "High Performance" policy will therefore spend very little to no time at all in Turbo Boost state.

With the ESXi power management policy set to "Balanced":

This policy enables the ESXi to switch cores that are not currently required into run states other than "active", allowing them to cool down and decreasing the potential power consumption and temperature of the processor. The processor gets room to run certain core(s) over their nominal frequency without risking to violate its power or thermal limits. Turbo Boost will be available.

Resolution

Here are the testing steps with the DELL Poweredge R640 server with Intel Xeon Silver 4214R.
Take the following example. The physical CPU clock frequency and CPU core frequency parameters are as below:

Processor Base Frequency 2.40 GHz
Max Turbo Frequency 3.50 GHz

Intel® Xeon® Silver 4214R Processor
https://ark.intel.com/content/www/us/en/ark/products/197100/intel-xeon-silver-4214r-processor-16-5m-cache-2-40-ghz.html

After adjusting the BIOS and ESXis power management mode, testing shows a significant improvement (144%) in the peak of the physical server's core frequency:

Booting the server with Turbo Boost, C-State and C1E enabled in the BIOS, referring to the device manufacturer's recommendations.
The ESXi hosts power management policy is set to "Balanced".
Access the ESXi host via ssh session and execute the following command to cause load for a single core:

# cat /dev/urandom | md5sum

in a second SSH session, open ESXTOP
press "p" to change to the processor power management tab
add the %A/MPERF field to the view (press "f", then "f" again", then [return]
observe the values in this column. 100 means that the core is running on nominal frequency, higher values means it runs with higher frequency.

Lab tests have shown values around 140 for this CPU, with maximums of 144.7. That means the core was running with 144.7% of its nominal frequency. 2.40 GHz * 144.7 % is 3.47 GHz.
The maximum Turbo Boost for this CPU model is 3.50 GHz, Turbo Boost was indeed triggered.