"Appliance is running low on memory. Add more memory to the machine", Memory exhaustion on vCenter Server Appliance 7.x
search cancel

"Appliance is running low on memory. Add more memory to the machine", Memory exhaustion on vCenter Server Appliance 7.x

book

Article ID: 318571

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:
Post upgrading vCenter Server to 7.x or with a fresh deployment of VC 7.x, you may experience the below Symptoms :
  •  VAMI of vCenter Server Appliance will show Overall Health & Memory in Alert Status with the message: Appliance is running low on memory. Add more memory to the machine.
KB_83824.jpg
  • High (more than 1GB) percpu allocator memory usage ("Percpu:" line in /proc/meminfo)
CapturePR1.PNG
  • High number (100+) of VMware Pod service restarts, you may verify it using the below command on VCSA SSH session
journalctl -b 0 | grep "Started VMware Pod" | wc -l

KB_83824_6.jpg


Environment

VMware vCenter Server 7.0.x
VMware vCenter Server 7.0.2

Cause

High percpu memory usage is caused by huge number of dying memory cgroups as result of frequent "VMware Pod" service restart. The concern of dying memory cgroups was discussed on Linux Plumbers 2020. The percpu memory overhead is especially high on VM with CPU Hot Add enabled.

Resolution

This issue is resolved in vCenter Server 7.0 U2c. For more information, see VMware vCenter Server 7.0 Update 2c Release Notes

Workaround:
To workaround this issue :
  1. Stop and disable vmware-pod service so the percpu memory usage won't grow further.
    • Login to VCSA using SSH (putty)
    • Execute the below commands to stop and disable "vmware-pod" service (Note: It is safe to disable vmware-pod service in on-prem environments)
      • systemctl stop vmware-pod
      • systemctl disable vmware-pod
      • KB_83824_5.jpg
  2. [Immediate action] - no reboot required: drop caches to free dirty pages which will release hanging memory cgroups in dying state. Most of percpu memory will be freed.
    • Login to VCSA using SSH (putty)
    • Execute below command to drop caches
      • echo 3 > /proc/sys/vm/drop_caches
      • KB_83824_4.jpg
  3. [Recommended] Disable kernel memory accounting for cgroups. Reboot required.
    • Login to VCSA using SSH (putty)
    • Edit file /boot/photon.cfg using VI editor
      • KB_83824_2.jpg
    • Add cgroup.memory=nokmem parameter to the end of  photon_cmdline=
      • KB_83824_3.jpg
    • Save the file using 'Esc + wq!' and exit VI editor
    • Reboot VCSA
  4. [Optional] Disable CPU Hot Add of VCSA VM if not needed. VM power off required.
    • Shutdown the vCenter Server Appliance Virtual Machine
    • Right-click a virtual machine in the inventory and select Edit Settings.
    • On the Virtual Hardware tab, expand CPU, and uncheck Enable CPU Hot Add.
    • Click OK
    • Power ON the VC VM


Powered by Wolken Software