HaltingIdleMsecPenalty Parameter: Guidance for Modifying vSphere's Fairness/Throughput Balance (1020233)
ESX provides fine-grained CPU scheduling that, among other things, enforces resource settings like CPU reservations, limits, and shares. Based on those settings, the scheduler determines if a VM has fallen behind in the amount of CPU time it should have received. Should the VM indeed be behind, attempts are made to help it catch up, usually by scheduling it more frequently.
If the processor has hyper-threading, however, scheduling more frequently does not guarantee that the VM will catch up. This is because the amount of CPU resources received by the vCPU is affected by the activity on the other logical processor on the same physical core. To guarantee the vCPU that is behind can catch up, ESX will sometimes not schedule VMs on the other logical processor, effectively leaving it idle.
In the case of the Intel Xeon 5500 series (and other modern hyper-threaded) processors, not scheduling on a logical processor may have a measurable impact on the overall throughput of the system. As a result, in systems that:
- have more than 50% CPU utilization, and
- are very close to exactly committed (number of vCPUs = number of pCPUs +/- 25%), and
- have particular kinds of bursty CPU usage patterns,
A parameter called HaltingIdleMsecPenalty determines when this fairness algorithm should come into effect. When, across all vCPUs of the VM, the total amount of time by which a VM has fallen behind exceeds the value of this parameter (in milliseconds), the scheduler will attempt to catch up the VM using the method described above. Note that setting the parameter too high may decrease the accuracy of resource settings, especially CPU reservations and shares. Also note that in an overwhelming majority of circumstances, setting the parameter larger than the default has no performance impact.
Caution: HaltingIdleMsecPenalty is an advanced parameter and should only be changed under guidance from VMware Support or after thorough testing in the user’s environment.
To set this parameter, use the following command on the ESX service console:
esxcfg-advcfg --set <desired value, e.g. 100> /Cpu/HaltingIdleMsecPenalty
For ESXi, use the following command on a Windows or Linux machine with vSphere CLI installed:
vicfg-advcfg --set <desired value, e.g. 100> /Cpu/HaltingIdleMsecPenalty