Impact of virtual machine memory and CPU resource limits

Products

VMware vSphere ESXi

Issue/Introduction

This article provides information about the impact of virtual machine resource limits on CPU and memory. A virtual machine experiences poor performance even though the ESX host is not fully utilized. Ballooning and vmkernel swapping occur

Environment

VMware ESX Server 3.5.x
VMware ESXi 4.1.x Installable
VMware ESX 4.1.x
VMware ESXi 3.5.x Installable
VMware ESX Server 3.0.x
VMware vSphere ESXi 5.0
VMware ESXi 4.1.x Embedded
VMware ESXi 3.5.x Embedded
VMware ESXi 4.0.x Embedded
VMware ESXi 4.0.x Installable
VMware vSphere ESXi 5.5
VMware ESX 4.0.x
VMware vSphere ESXi 5.1

Resolution

Resource limits on a virtual machine for both CPU and memory form an artificial boundary for the virtual machine.

Memory Limits

When a memory limit is set less than the virtual machine's provisioned memory, it is considered as the upper boundary for the amount of physical memory that can be directly assigned to this particular virtual machine. The guest operating system is not aware of this limit, and it optimizes memory management options to the assigned memory size.

When the limit is reached or exceeded, the guest operating system can still request new pages, but due to the limit the VMkernel does not allow the guest to directly consume more physical memory and treats the virtual machine as if the resource is under contention. Memory reclamation techniques are used to enable the virtual machine to consume requested memory.

Depending on the number of pages requested by the virtual machine, the VMkernel might resort to VMkernel swap to fulfil the request, in the worst case scenario. The VMkernel first tries to reclaim memory by inflating the Balloon Driver to let the guest memory manager decide what to page out. In ESX 4.1, the VMkernel also tries to compress memory pages before swapping them out.

You can verify the impact of a memory limit by running esxtop and looking at MCTLSZd and MCTLTGT, SWCUR and SWTGT, and CACHEUSD.

For more information on using esxtop, see Interpreting esxtop Statistics.

CPU Limits

When a CPU Limit is set on a virtual machine resource setting, the virtual machine is deliberately held from being scheduled to a PCPU when it has used up its allocated CPU resource. This happens regardless of the CPU utilization. If the limit is set to 500MHz, the virtual machine is rescheduled from the PCPU and has to wait before it is allowed to be scheduled again. As such, the virtual machine might experience performance degradation.

Note: For an SMP virtual machine, the sum of all vCPUs cannot exceed the specified limit. For example, 4 vCPU virtual machine with a limit of 1200MHz and equal load among vCPUs would result in a max of 300MHz per vCPU.

You can verify the impact of a CPU limit by running esxtop and looking at %RDY and %MLMTD.

For more information on using esxtop, see Interpreting esxtop Statistics.

Additional Information

You can also use shares to restrict resource usage of certain virtual machines only when the host becomes overcommitted.

You can use a limit to simulate resource over commitment in advance or to dynamically constrain the resource usage of a virtual machine (for example, in case a rogue virtual machine's excessive resource utilization negatively affects the host and the other virtual machines).

For more information on how memory management works:

In ESX 4.1, see the Understanding memory Resource Management in VMware ESX 4.1.
In ESX 5.5, see the vSphere Resource Management 5.5 Guide.
In ESX 6.0, see the vSphere Resource Management 6.0 Guide

For related information, see:

Time falls behind in a virtual machine when the memory of the virtual machine is paged from disk by the VMkernel
Virtual machines configured with CPU limits running on ESX/ESXi 4.x hosts experience performance degradation
仮想マシンメモリと CPU リソースの制限の影響
虚拟机内存和 CPU 资源限制的影响