Knowledge Base

The VMware Knowledge Base provides support solutions, error messages and troubleshooting guides
 
Search the VMware Knowledge Base (KB)   View by Article ID
 

"Heap globalCartel-1 already at its maximum size” error in ESXi host (2085618)

Symptoms

  • Cannot perform a vMotion to and from an ESXi host
  • vSphere Client reports the error:

    Heap globalCartel-1 already at its maximum size of 312160. Cannot expand.
    Could not start VMX: msg.vmk.status.VMK_NO_MEMORY

     
    Note: For additional symptoms and log entries, see the Additional Information section. 

Purpose

You must upgrade your host to ESXi 5.5 Update 3 or upgrade AMS to version 10.0.1 or later to resolve this issue.

Cause

This issue occurs with ESXi hosts running on HP hardware with these versions of AMS:
  • hp-ams 500.9.6.0-12.434156
  • hp-ams-550.9.6.0-12.1198610
  • hp-ams 500.10.0.0-18.434156
  • hp-ams-550.10.0.0-18.1198610

Resolution

This is a known issue affecting ESXi 5.x.

To resolve this issue, use one of these options:
To work around this issue if you do not want to upgrade, remove the AMS package on all hosts.

Note: In some cases, commands running on the ESXi host fails with cant't fork. In this case, the virtual machines running on the ESXi host needs to be powered off and the ESXi host rebooted.

To verify the installed versions of AMS, run this command:

esxcli software vib list | grep ams


To remove the package on all hosts running on these AMS versions:
  1. Log in to the host using SSH. 
  2. Run this command to stop the HP service.

    /etc/init.d/hp-ams.sh stop

    Note: The changes do not persist across reboot.
  3. Run this command to remove the VIB:

    esxcli software vib remove -n hp-ams 
  4. Restart the host.
Disclaimer: VMware is not responsible for the reliability of any data, opinions, advice, or statements made on third-party websites. Inclusion of such links does not imply that VMware endorses, recommends, or accepts any responsibility for the content of such sites. 

Additional Information

You experience these additional symptoms:
  • ESXi host becomes unresponsive
  • When attempting to enable services or perform a vMotion, the ESXi host fails with one of these errors:
    • A general system error occurred: Command /bin/sh failed
    • A general system error occured: Launch Failure

  • When connecting to a virtual machine console, you see one of these errors:
    • Unable to contact the MKS: Could not connect to pipe\\.\pipe\vmware-authdpipe
    • Unable to connect to the MKS: connection terminated by server.

  • When powering on a virtual machine, you see the error:

    VMK_NO_MEMORY

  • When logging in to the ESXi shell, you see entry:

    can't fork

  • When pressing  Alt+F12 at the DCUI, you see the error:

    WARNING: Heap: 2677: Heap globalCartel-1 already at its maximum size. Cannot expand.
  • In the /var/log/vmkwarning file, you see entries similar to:

    <YYYY-MM-DD>T<TIME> cpu18:4287108)WARNING: Heap: 3058: Heap_Align(globalCartel-1, 136/136 bytes, 8 align) failed. caller: 0x41802a2ca2fd
    <YYYY-MM-DD>T<TIME>cpu8:4287111)WARNING: Heap: 2677: Heap globalCartel-1 already at its maximum size. Cannot expand.
    <YYYY-MM-DD>T<TIME>cpu8:4287111)WARNING: Heap: 3058: Heap_Align(globalCartel-1, 136/136 bytes, 8 align) failed. caller: 0x41802a2ca2fd
    <YYYY-MM-DD>T<TIME>cpu1:4287132)WARNING: Heap: 2677: Heap globalCartel-1 already at its maximum size. Cannot expand.
    <YYYY-MM-DD>T<TIME>cpu1:4287132)WARNING: Heap: 3058: Heap_Align(globalCartel-1, 136/136 bytes, 8 align) failed. caller: 0x41802a2ca2fd
    <YYYY-MM-DD>T<TIME>cpu24:4287142)WARNING: Heap: 2677: Heap globalCartel-1 already at its maximum size. Cannot expand.
    <YYYY-MM-DD>T<TIME>cpu24:4287142)WARNING: Heap: 3058: Heap_Align(globalCartel-1, 136/136 bytes, 8 align) failed. caller: 0x41802a2ca2fd
    <YYYY-MM-DD>T<TIME>cpu5:4025029)WARNING: Heap: 2677: Heap globalCartel-1 already at its maximum size. Cannot expand.


    Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
  • Running commands in the ESXi console fail with the error:

    -sh: can't set tty process group (Operation not permitted)

Tags

vMotion fails, host not responding

Keywords

vMotion fails, host not responding

See Also

This Article Replaces

2099351

Update History

09/11/2014 - Added VMware ESXi 5.5.x 09/24/2014 - Added VMware ESXi 5.0.x

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 89 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.

What can we do to improve this information? (4000 or fewer characters)




Please enter the Captcha code before clicking Submit.
  • 89 Ratings
Actions
KB: