This article addresses the issue of running out of heap space on an ESXi/ESX host when a large quantity of virtual disk space is active.

Symptoms:

• An ESXi/ESX 3.5/4.0 host has more than 4 terabytes of virtual disks (.vmdk files) open.
  
• After virtual machines are failed over by vSphere HA from one host to another due to a host failover, the virtual machines fail to power on with the error:
  
  vSphere HA unsuccessfully failed over this virtual machine. vSphere HA will retry if the maximum number of attempts has not been exceeded. Reason: Cannot allocate memory.
  
  
• In the /var/log/messages or /var/log/vmkernel.log file, you see entries similar to:
  
  vmkernel: cpu2:1410)WARNING: Heap: 1370: Heap_Align(vmfs3, 4096/4096 bytes, 4 align) failed. caller: 0x8fdbd0
vmkernel: cpu2:1410)WARNING: Heap: 1266: Heap vmfs3: Maximum allowed growth (24) too small for size (8192)
cpu4:1959755)WARNING:Heap: 2525: Heap vmfs3 already at its maximum size. Cannot expand.
cpu4:1959755)WARNING: Heap: 2900: Heap_Align(vmfs3, 2099200/2099200 bytes, 8 align) failed. caller: 0x418009533c50
cpu7:5134)Config: 346: "SIOControlFlag2" = 0, Old Value: 1, (Status: 0x0)

  cpu29:8350374)WARNING: J3: 1666: Failed to reserve space for journal on 4f8444cf-ae524ea8-a556-5cf3fcdad836 : Out of memory
  cpu29:8350374)WARNING: Fil3: 2492: Failed to reserve volume f530 28 1 4f8444cf ae524ea8 f35ca556 36d8dafc 0 0 0 0 0 0 0
  
  
• Adding a VMDK to a virtual machine running on an ESXi/ESX host where VMFS-3 heap is maxed out fails.
• When attempting to migrate a virtual machine to another host, you see the error:
  
  Failed to initialize migration at source
  
  
• When you try to manually power on a migrated virtual machine, you see the error:
  
  The VM failed to resume on the destination during early power on.
Reason: 0 (Cannot allocate memory).
Cannot open the disk 'vmdk_location' or one of the snapshot disks it depends on.
  
  
• A virtual machine fails to power on and you see this error in the vSphere Client:
  
  An unexpected error was received from the ESX host while powering on VM vm-xxx. Reason: (Cannot allocate memory)
  
  
• A similar error appears if you try to migrate or Storage vMotion a virtual machine to a destination ESXi/ESX host on which VMFS-3 heap is maxed out.
• Cloning a virtual machine using the vmkfstools -i command fails and you see the error:
  
  Clone: 43% done. Failed to clone disk: Cannot allocate memory (786441)
  
  
• Cold storage migration fails with the error:
  
  Could not complete network copy for file
  
  
• In the /var/log/vmfs/volumes/DatastoreName/VirtualMachineName/vmware.log file, you see messages similar to:
  
  vmx| FileIOErrno2Result: Unexpected errno=12, Cannot allocate memory
vmx| AIOGNRC: Failed to open '/vmfs/volumes/xxxx-flat.vmdk' : Cannot allocate memory (c00000002) (0x2013).
vmx| DISKLIB-VMFS : "/vmfs/volumes/xxxx-flat.vmdk" : failed to open (Cannot allocate memory): AIOMgr_Open failed. Type 3
vmx| DISKLIB-LINK : "/vmfs/volumes/xxxx.vmdk" : failed to open (Cannot allocate memory).
vmx| DISKLIB-CHAIN : "/vmfs/volumes/xxxx.vmdk" : failed to open (Cannot allocate memory).
vmx| DISKLIB-LIB : Failed to open '/vmfs/volumes/xxxx.vmdk' with flags 0xa Cannot allocate memory (786441).
vmx| DISK: Cannot open disk "/vmfs/volumes/xxxx.vmdk": Cannot allocate memory (786441).
vmx| Msg_Post: Error
vmx| [msg.disk.noBackEnd] Cannot open the disk '/vmfs/volumes/xxxx.vmdk' or one of the snapshot disks it depends on.
vmx| [msg.disk.configureDiskError] Reason: Cannot allocate memory.


• You see virtual machines run more slowly than usual or become unresponsive.
• The vmkernel stops logging in the appropriate /var/log file of the ESX/ESXi host.

VMware vSphere ESXi 7.0.0
VMware vSphere ESXi 6.0
VMware vSphere ESXi 5.5
VMware vSphere ESXi 5.0
VMware vSphere ESXi 5.1
VMware ESX Server 3.5.x
VMware ESX 4.0.x
VMware ESX 4.1.x
VMware ESXi 4.0.x Embedded
VMware ESXi 3.5.x Installable
VMware vSphere ESXi 6.7
VMware ESXi 3.5.x Embedded
VMware ESX Server 3.0.x
VMware vSphere ESXi 6.5
VMware ESXi 4.0.x Installable
VMware ESXi 4.1.x Installable
VMware ESXi 4.1.x Embedded

1. Log into vCenter Server or the ESXi/ESX host using the vSphere Client or VMware Infrastructure (VI) Client. If connecting to vCenter Server, select the ESXi/ESX host from the inventory.
2. Under the Configuration tab, click Advanced Settings.
3. Click VMFS3.
4. Update the field in VMFS3.MaxHeapSizeMB.
5. Reboot the ESXi/ESX host for the changes to take affect.
   

1. Log into vCenter Server or the ESXi/ESX host using the vSphere Client or VMware Infrastructure (VI) Client. If connecting to vCenter Server, select the ESXi/ESX host from the inventory.
2. Under the Configuration tab, click Advanced Settings.
3. Select VMFS3.
4. Update the field in VMFS3.MinHeapSizeMB.
5. Reboot the ESXi/ESX host for the changes to take affect.

Command Line Usage

The esxcfg-advcfg command can be used to set/get the maximum heap size for all VMFS file systems.

• To check the exiting value, run the command:

  esxcfg-advcfg -g /VMFS3/MaxHeapSizeMB
  
  Sample Output:
  
  Value of MaxHeapSizeMB is 256
  
  
  
• To change the heap size, run the command:
  
  esxcfg-advcfg -s <heap_value> /VMFS3/MaxHeapSizeMB
  For example:
  
  esxcfg-advcfg -s 640 /VMFS3/MaxHeapSizeMB

Default, Min, and Max Table

To be alerted when this document is updated, click the Subscribe to Article link in the Actions box

Correlating build numbers and versions of VMware products
Understanding the VMFS heap eviction process in ESXi 5.5 and later
ESXi 5.5 における VMFS ヒープの扱いについて
まとめて 4 TB または 20 TB の仮想ディスク ストレージを使用する仮想マシンをホストしている場合に ESXi/ESX ホストが VMFS ヒープの警告を報告する
在托管集中使用 4TB 或 20TB 的虚拟磁盘存储的虚拟机时，ESXi/ESX 主机会报告 VMFS 堆警告

Impact/Risks: