Knowledge Base

The VMware Knowledge Base provides support solutions, error messages and troubleshooting guides

Search the VMware Knowledge Base (KB)

View by Article ID

HaltingIdleMsecPenalty Parameter: Guidance for Modifying vSphere's Fairness/Throughput Balance (1020233)

Details

ESX provides fine-grained CPU scheduling that, among other things, enforces resource settings like CPU reservations, limits, and shares. Based on those settings, the scheduler determines if a VM has fallen behind in the amount of CPU time it should have received. Should the VM indeed be behind, attempts are made to help it catch up, usually by scheduling it more frequently.

If the processor has hyper-threading, however, scheduling more frequently does not guarantee that the VM will catch up. This is because the amount of CPU resources received by the vCPU is affected by the activity on the other logical processor on the same physical core. To guarantee the vCPU that is behind can catch up, ESX will sometimes not schedule VMs on the other logical processor, effectively leaving it idle.

In the case of the Intel Xeon 5500 series (and other modern hyper-threaded) processors, not scheduling on a logical processor may have a measurable impact on the overall throughput of the system. As a result, in systems that:

have more than 50% CPU utilization, and
are very close to exactly committed (number of vCPUs = number of pCPUs +/- 25%), and
have particular kinds of bursty CPU usage patterns,

we have observed a throughput drop of up to 15% when this fairness algorithm takes effect.

Note: This issue does not affect ESXi 5.0.

Solution

A parameter called HaltingIdleMsecPenalty determines when this fairness algorithm should come into effect. When, across all vCPUs of the VM, the total amount of time by which a VM has fallen behind exceeds the value of this parameter (in milliseconds), the scheduler will attempt to catch up the VM using the method described above. Note that setting the parameter too high may decrease the accuracy of resource settings, especially CPU reservations and shares. Also note that in an overwhelming majority of circumstances, setting the parameter larger than the default has no performance impact.

Caution: HaltingIdleMsecPenalty is an advanced parameter and should only be changed under guidance from VMware Support or after thorough testing in the user’s environment.

To set this parameter, use the following command on the ESX service console:

esxcfg-advcfg --set <desired value, e.g. 100> /Cpu/HaltingIdleMsecPenalty

For ESXi, use the following command on a Windows or Linux machine with vSphere CLI installed:

vicfg-advcfg --set <desired value, e.g. 100> /Cpu/HaltingIdleMsecPenalty

As of patch-set ESX400-201003001 to ESX 4.0 (available now on the VMware support website) and all versions of ESX going forward, a much larger maximum value of this parameter is allowed (up to 10,000 instead of the current 100). Larger values may make sense in environments where achieving maximal aggregated throughput is more important than fair sharing of CPU resources. Internal testing shows that values greater than 2,000 show little gain in total throughput.

Starting with ESX 4.0 U2, the meaning of this parameter changes slightly. Before it is used, it is multiplied by the number of virtual CPUs. After this operation, the value is capped by the new parameter HaltingIdleMsecPenaltyMax (maximum allowed value is 80,000). Both parameters may be set on the command line as shown above or changed in the vSphere Client (Host > Advanced Settings > CPU). The fairness check can be disabled entirely by setting HaltingIdleMsecPenalty to 0.

Keywords

HT, HTT, hyperthreading, hyper-threading, HIMP

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

1 2 3 4 5
11 Ratings

1 2 3 4 5
11 Ratings

Actions

Bookmark Document

KB:

Did this article help you?
This article resolved my issue. This article did not resolve my issue. This article helped but additional information was required to resolve my issue.
What can we do to improve this information? (4000 or fewer characters)