Knowledge Base

The VMware Knowledge Base provides support solutions, error messages and troubleshooting guides
 
Search the VMware Knowledge Base (KB)   View by Article ID
 

ESXi/ESX host reports VMFS heap warnings when hosting virtual machines that collectively use 4 TB or 20 TB of virtual disk storage (1004424)

Symptoms

  • An ESXi/ESX 3.5/4.0 host has more than 4 terabytes (TB) of virtual disks (.vmdk files) open.
  • After virtual machines are failed over by vSphere HA from one host to another due to a host failover, the virtual machines fail to power on with the error:

    vSphere HA unsuccessfully failed over this virtual machine. vSphere HA will retry if the maximum number of attempts has not been exceeded. Reason: Cannot allocate memory.

  • You see warnings in /var/log/messages or /var/log/vmkernel.log similar to:

    vmkernel: cpu2:1410)WARNING: Heap: 1370: Heap_Align(vmfs3, 4096/4096 bytes, 4 align) failed. caller: 0x8fdbd0
    vmkernel: cpu2:1410)WARNING: Heap: 1266: Heap vmfs3: Maximum allowed growth (24) too small for size (8192)
    cpu4:1959755)WARNING:Heap: 2525: Heap vmfs3 already at its maximum size. Cannot expand.
    cpu4:1959755)WARNING: Heap: 2900: Heap_Align(vmfs3, 2099200/2099200 bytes, 8 align) failed. caller: 0x418009533c50
    cpu7:5134)Config: 346: "SIOControlFlag2" = 0, Old Value: 1, (Status: 0x0)

  • Adding a VMDK to a virtual machine running on an ESXi/ESX host where VMFS-3 heap is maxed out fails.

  • When attempting to migrate a virtual machine to another host, you may see the error:

    Failed to initialize migration at source

  • When you try to manually power on a migrated virtual machine, you may see the error:

    The VM failed to resume on the destination during early power on.
    Reason: 0 (Cannot allocate memory).
    Cannot open the disk '<<Location of the .vmdk>>' or one of the snapshot disks it depends on.

  • A virtual machine fails to power on and you see an error in the vSphere Client:

    An unexpected error was received from the ESX host while powering on VM vm-xxx. Reason: (Cannot allocate memory)

  • A similar error may appear if you try to migrate or Storage vMotion a virtual machine to a destination ESXi/ESX host on which VMFS-3 heap is maxed out.

  • Cloning a virtual machine using the vmkfstools -i command fails and you see the error:

    Clone: 43% done. Failed to clone disk: Cannot allocate memory (786441)

  • In the /var/log/vmfs/volumes/DatastoreName/VirtualMachineName/vmware.log file, you may see error messages similar to:

    2012-05-02T23:24:07.900Z| vmx| FileIOErrno2Result: Unexpected errno=12, Cannot allocate memory
    2012-05-02T23:24:07.900Z| vmx| AIOGNRC: Failed to open '/vmfs/volumes/xxxx-flat.vmdk' : Cannot allocate memory (c00000002) (0x2013).
    2012-05-02T23:24:07.900Z| vmx| DISKLIB-VMFS : "/vmfs/volumes/xxxx-flat.vmdk" : failed to open (Cannot allocate memory): AIOMgr_Open failed. Type 3
    2012-05-02T23:24:07.900Z| vmx| DISKLIB-LINK : "/vmfs/volumes/xxxx.vmdk" : failed to open (Cannot allocate memory).
    2012-05-02T23:24:07.900Z| vmx| DISKLIB-CHAIN : "/vmfs/volumes/xxxx.vmdk" : failed to open (Cannot allocate memory).
    2012-05-02T23:24:07.900Z| vmx| DISKLIB-LIB : Failed to open '/vmfs/volumes/xxxx.vmdk' with flags 0xa Cannot allocate memory (786441).
    2012-05-02T23:24:07.900Z| vmx| DISK: Cannot open disk "/vmfs/volumes/xxxx.vmdk": Cannot allocate memory (786441).
    2012-05-02T23:24:07.900Z| vmx| Msg_Post: Error
    2012-05-02T23:24:07.900Z| vmx| [msg.disk.noBackEnd] Cannot open the disk '/vmfs/volumes/xxxx.vmdk' or one of the snapshot disks it depends on.
    2012-05-02T23:24:07.900Z| vmx| [msg.disk.configureDiskError] Reason: Cannot allocate memory.

Purpose

This article addresses the issue of running out of heap space on an ESXi/ESX host when a large quantity of virtual disk space is active.

Resolution

The default heap size in ESXi/ESX 3.5/4.0 for VMFS-3 is set to 16 MB. This allows for a maximum of 4 TB of open virtual disk capacity on a single ESXi/ESX host. The default heap size has been increased in ESXi/ESX 4.1 and ESXi 5.x to 80 MB, which allows for 8 TB of open virtual disk capacity on a single ESXi/ESX host.

The default heap size has been further increased in ESXi 5.0 Patch Release ESXi500-201303001 to 640 MB, which allows for 60 TB of open virtual disk capacity on a single ESXi/ESX host.

Note: Although the examples here reference VMFS-3, this issue can also affect VMFS-5 datastores. The VMFS3.MaxHeapSizeMB variable also applies to VMFS-5 volumes.

To adjust the maximum values of VMFS heap:
  1. Log into vCenter Server or the ESXi/ESX host using the vSphere Client or VMware Infrastructure (VI) Client. If connecting to vCenter Server, select the ESXi/ESX host from the inventory.
  2. Click the Configuration tab.
  3. Click Advanced Settings.
  4. Click VMFS3.
  5. Update the field in VMFS3.MaxHeapSizeMB.

    Notes:
    • In ESXi/ESX 4.x and earlier releases, the maximum heap size is 128 MB. This allows a maximum of 32 TB of open storage per ESXi/ESX host.
    • In ESXi 5.x, the maximum heap size is 256 MB. This allows a maximum of 25 TB of open storage per ESXi host.
    • In ESXi 5.0 Patch Release ESXi500-201303001 and ESXi 5.1 Update 1, the maximum heap size was increased to 640 MB. This allows a maximum of 60 TB of open storage per ESXi host. For more information, see the ESXi 5.1 U1 Release Notes.
    • In ESXi 5.1 Patch 1 (build 914609), the maximum heap size is 256 MB.


  6. Reboot the ESXi/ESX host for the changes to take affect.
This table outlines the minimum and maximum heap sizes that can be set for the various builds of ESXi/ESX:

Version/build Minimum heap value Maximum heap value Maximum open VMDK storage per host
ESXi/ESX 4.x N/A 128 MB 32 TB
ESXi 5.0 Build 914586 and earlier N/A 256 MB 25 TB
ESXi 5.0 Build 1024429 and later 256 MB 640 MB 60 TB
ESXi 5.1 Build 914609 and earlier N/A 256 MB 25 TB
ESXi 5.1 Build 1065491 and later 256 MB 640 MB 60 TB

ESXi 5.0 Patch Release ESXi500-201303001 and ESXi 5.1 Update 1 also provide another option to reserve up to 256 MB of VMFS heap up front for the boot process. This is done by modifying the VMFS3.MinHeapSizeMB option.

Note: Although VMFS3.MinHeapSizeMB can only be set to 256 MB, the heap can still grow to the maximum size of 640 MB in these releases.

To adjust the minimum values of VMFS heap:
  1. Log into vCenter Server or the ESXi/ESX host using the vSphere Client or VMware Infrastructure (VI) Client. If connecting to vCenter Server, select the ESXi/ESX host from the inventory.
  2. Click the Configuration tab.
  3. Click Advanced Settings.
  4. Select VMFS3.
  5. Update the field in VMFS3.MinHeapSizeMB.
  6. Reboot the ESXi/ESX host for the changes to take affect.
Notes:
  • The net effect of this change is that the ESXi/ESX kernel will require a small amount of additional memory (such as the 256 MB we use to get a maximum of 25 TB in the example above) for the larger heap, but it will allow virtual machines with more than 4 TB (ESXi/ESX 4.x) or 8 TB (ESXi 5.0) of virtual disk to be addressed.
  • Open VMDK storage refers to any open virtual disks, including thin disks.

Additional Information

This issue applies only to VMFS volumes and does not impact RDMs (Raw Device Mapping) or VMDK files on NFS datastores.

This Article Replaces

2020208

Update History

11/28/2011 - Updated the Additional Information section 03/23/2012 - Added additional symptoms 07/26/2012 - Added cloning virtual machine symptom 08/28/2012 - Corrected heap size for ESX 4.1 and ESXi 5.x 08/29/2012 - Added symptoms when using vSphere HA 09/27/2012 - Added that this issue applies only to VMFS volumes and not to RDMs 10/25/2012 - Removed lines from symptoms that did not apply 04/19/2013 - Added additional symptom

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 40 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.
What can we do to improve this information? (4000 or fewer characters)
  • 40 Ratings
Actions
KB: