Search the VMware Knowledge Base (KB)
View by Article ID

EAM fails with OutOfMemoryErrors (2135378)

  • 2 Ratings

Symptoms

During host preparation in VMware NSX for vSphere 6.x, you experience these symptoms:
  • The vSphere ESX Agent Manager (EAM) threads get stuck or EAM service may not start.
  • In the C:\ProgramData\VMware\VMware VirtualCenter\Logs\eam.log file on the vCenter Server, you see entries similar to:

    2015-04-23T06:31:38.998Z | DEBUG | agent-4 | AgentImpl.java | 2251 | Agent AgentImpl(ID:'Agent:e83ee5ce-b8f8-4d3b-b106-1900e08d6d20:3c36c535-2eb8-434e-b7c4-e07173a69e11') (VcHostSystem(ID: host-359)) scheduled a VIB task and is waiting for it to complete.
    2015-04-23T06:31:40.379Z | DEBUG | compute-res-0 | ExecutorImpl.java | 57 | JOB SUBMITTED: [#1420322380] ClusterVibConditionalJob(ManagedObjectReference: type = ClusterComputeResource, value = domain-c343, serverGuid = 885ab205-24f0-4c05-8539-7e59caf9a679), trace: .:scheduleNewVibTask:.
    2015-04-23T06:31:40.379Z | DEBUG | compute-res-0 | AuditedJob.java | 70 | JOB STARTED: [#1420322380] ClusterVibConditionalJob(ManagedObjectReference: type = ClusterComputeResource, value = domain-c343, serverGuid = 885ab205-24f0-4c05-8539-7e59caf9a679), trace: .:scheduleNewVibTask:.

    INFO   | jvm 1    | 2015/04/23 06:32:20 | java.lang.OutOfMemoryError: Java heap space
    INFO   | jvm 1    | 2015/04/23 06:32:20 | Dumping heap to /var/log/vmware/eam/java_pid24722.hprof ...
    INFO   | jvm 1    | 2015/04/23 06:32:22 | Heap dump file created [138746180 bytes in 2.438 secs]
    INFO   | jvm 1    | 2015/04/23 06:32:22 | Exception in thread "host-359-0" java.lang.OutOfMemoryError: Java heap space
    INFO   | jvm 1    | 2015/04/23 06:32:22 |     at sun.nio.cs.StreamEncoder.write(Unknown Source)
    INFO   | jvm 1    | 2015/04/23 06:32:22 |     at java.io.OutputStreamWriter.write(Unknown Source)
    INFO   | jvm 1    | 2015/04/23 06:32:22 |     at java.io.Writer.write(Unknown Source)

    INFO   | jvm 1    | 2015/04/23 06:32:23 |     at
    com.vmware.eam.vc.VcListener$4.call(VcListener.java:338)
    INFO   | jvm 1    | 2015/04/23 06:32:23 |     at com.vmware.eam.vc.VcListener$4.call(VcListener.java:335)
    INFO   | jvm 1    | 2015/04/23 06:32:23 |     at com.vmware.eam.lock.RaceProtection.exec(RaceProtection.java:68)
    INFO   | jvm 1    | 2015/04/23 06:32:23 |     at com.vmware.eam.lock.RaceProtection.exec(RaceProtection.java:51)
    INFO   | jvm 1    | 2015/04/23 06:32:23 |     at com.vmware.eam.vc.VcListener.unregisterObserver(VcListener.java:335)

    2015-04-23T06:35:43.033Z | DEBUG | agent-3 | AgentImpl.java | 2251 | Agent AgentImpl(ID:'Agent:86cab9ca-79da-412c-a8a3-b3f178121da5:3c36c535-2eb8-434e-b7c4-e07173a69e11') (VcHostSystem(ID: host-363)) scheduled a VIB task and is waiting for it to complete.


    For more information on vCenter Server log location, see Location of vCenter Server log files (1021804).

    Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Cause

This issue occurs due to the bug in EAM where it tries to poll the same updates again from VM exhausting the memory of the EAM.

Resolution

This is a known issue affecting vCenter Server 6.0.x.
 
This issue is resolved in vCenter Server 6.0 Update 3, Available at VMware Downloads

To work around the issue, increase the EAM heap size to the desired level using cloudvm-ram-size utility.

To increase the heap size:
  1. Stop the VMware VirtualCenter Server service that also stops the vCenter Inventory service. For more information, see Stopping, starting, or restarting VMware vCenter Server services (1003895).

    Note: To stop the EAM service on the vCenter Server, run this command:

    service vmware-eam stop

  2. Open command line utility and navigate to the folder C:\Program Files\VMware\vCenter Server\visl-integration\usr\sbin.

  3. Run the command:

    cloudvm-ram-size -C NewMemorysizeinMB EsxAgentManager

    Incase of vCenter Appliance:

    cloudvm-ram-size -C NewMemorysizeinMB vmware-eam

  4. Restart the VMware VirtualCenter Server service.

    To restart the eam service on vCenter Server, run this command:

    service vmware-eam restart
To confirm that heap size is increased, run the commands:
  1. cloudvm-ram-size -J vmware-eam -O /etc/vmware/java/vmware-eam_jvm.conf

    This output shows the allocated heap size for EAM.

  2. Echo $?

    0 indicates success, any non-zero value indicates failure.

  3. If there is a failure, check the file permissions of vmware-eam_jvm.conf and ensure read and write privileges on the file.

    ls -ltr /etc/vmware/java/vmware-eam_jvm.conf
    -rw------- 1 eam users 77 Mar 10 06:56 /etc/vmware/java/vmware-eam_jvm.conf

See Also

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 2 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.

What can we do to improve this information? (4000 or fewer characters)




Please enter the Captcha code before clicking Submit.
  • 2 Ratings
Actions
KB: