Memory usage alarm triggers for certain types of Virtual Machines in ESXi

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

Virtual machine memory usage alarm is displayed for the virtual machine.
The Virtual Machine's Memory Usage/Active performance metric is continually reported as 100%.
On vSphere 6.0, the Virtual Machine is configured with Latency Sensitivity set to High.
On vSphere 6.5 and later the Virtual Machine is configured with at least one of these options:

PCI passthrough devices
Fault Tolerance (FT) enabled
Latency Sensitivity set to High
vGPU enabled
Manually configured "sched.mem.prealloc.pinnedMainMem = True"

On vSphere 6.7 and later the Virtual Machine is configured with either of the above or:
1GB Pages

Note: If Latency Sensitivity was set to High while the Virtual Machine was powered on, the pre-allocation will only apply at the next power cycle.

Environment

VMware vSphere ESXi 7.0.0
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.0
VMware vSphere ESXi 7.0.1

Cause

The behavior of showing 100% "Memory Active", depending on the ESXi version, is by design for Virtual Machines in the specified configurations.

ESXi's active memory metric, despite being called "Memory Utilization" or "Memory Usage" in different parts of the UI, is in no way related to the in-guest memory metrics. It doesn't show how much guest OS memory is available nor how much guest memory is in an "active" working set or "resident". It is only used for making memory reclamation decisions in addition to other resource controls like shares, limits and reservation

"Memory Active" is a heuristic utilizing a weighted moving average based on reads and writes to a small, moving subset of pages of the Virtual Machine's memory. The sampling of this subset of 100 random pages over one minute also incurs a minimal overhead and preallocating memory is designed to remove most possibilities of jitter. The Virtual Machine configurations mentioned in the Symptoms section force a full memory reservation and are not subjected to reclamation techniques, hence memory sampling is disabled and active memory will default to a display value of 100%.

Resolution

This is a display artifact only and has no negative performance impact on the virtual machine or the host it is running on. To limit false positives, the memory activity for VMs with preallocated memory has been reduced to 75% in ESXi 7.0 U2.

Workaround:

Disable the Virtual machine memory usage alarm to avoid false positives.

To disable the alarm definition using the vSphere Web Client:

Select the vCenter object in the navigation pane of the vSphere Web Client.
Click the Monitor tab > Issues sub-tab.
Select Alarm Definitions and search for Virtual machine memory usage.
Highlight the alarm and click Edit.
Deselect Enable this alarm.
Click Finish.

To disable the alarm definition using the HTML5 vSphere Client:

Select the vCenter object in the navigation pane of the vSphere Client
Click Configure > Alarm Definitions
In the Alarm Name column header, click the filter icon and search for Virtual machine memory usage
Highlight the alarm
Click Disable

Note: These steps will disable the alarm for all virtual machines. If you require the memory usage alarm to be functional for other than the VMs affected by the behavior described in this article, you will need to create a custom alarm with the same settings as the default one, on an object these VMs are nested under (such as a VM folder for example).

Additional Information

VM memory usage heuristic over-reporting on ESXi 6.5

简体中文：对 ESXi 6.x 中某些类型的虚拟机触发内存使用情况警报