EAM fails with OutOfMemoryErrors
search cancel

EAM fails with OutOfMemoryErrors

book

Article ID: 325734

calendar_today

Updated On:

Products

VMware NSX Networking VMware vCenter Server

Issue/Introduction

Symptoms:
During host preparation in VMware NSX for vSphere 6.x, you experience these symptoms:
  • The vSphere ESX Agent Manager (EAM) threads get stuck or EAM service may not start.
  • In the C:\ProgramData\VMware\VMware VirtualCenter\Logs\eam.log file on the vCenter Server, you see entries similar to:

    2015-04-23T06:31:38.998Z | DEBUG | agent-4 | AgentImpl.java | 2251 | Agent AgentImpl(ID:'Agent:e83ee5ce-b8f8-4d3b-b106-1900e08d6d20:3c36c535-2eb8-434e-b7c4-e07173a69e11') (VcHostSystem(ID: host-359)) scheduled a VIB task and is waiting for it to complete.
    2015-04-23T06:31:40.379Z | DEBUG | compute-res-0 | ExecutorImpl.java | 57 | JOB SUBMITTED: [#1420322380] ClusterVibConditionalJob(ManagedObjectReference: type = ClusterComputeResource, value = domain-c343, serverGuid = 885ab205-24f0-4c05-8539-7e59caf9a679), trace: .:scheduleNewVibTask:.
    2015-04-23T06:31:40.379Z | DEBUG | compute-res-0 | AuditedJob.java | 70 | JOB STARTED: [#1420322380] ClusterVibConditionalJob(ManagedObjectReference: type = ClusterComputeResource, value = domain-c343, serverGuid = 885ab205-24f0-4c05-8539-7e59caf9a679), trace: .:scheduleNewVibTask:.

    INFO | jvm 1 | 2015/04/23 06:32:20 | java.lang.OutOfMemoryError: Java heap space
    INFO | jvm 1 | 2015/04/23 06:32:20 | Dumping heap to /var/log/vmware/eam/java_pid24722.hprof ...
    INFO | jvm 1 | 2015/04/23 06:32:22 | Heap dump file created [138746180 bytes in 2.438 secs]
    INFO | jvm 1 | 2015/04/23 06:32:22 | Exception in thread "host-359-0" java.lang.OutOfMemoryError: Java heap space
    INFO | jvm 1 | 2015/04/23 06:32:22 | at sun.nio.cs.StreamEncoder.write(Unknown Source)
    INFO | jvm 1 | 2015/04/23 06:32:22 | at java.io.OutputStreamWriter.write(Unknown Source)
    INFO | jvm 1 | 2015/04/23 06:32:22 | at java.io.Writer.write(Unknown Source)

    INFO | jvm 1 | 2015/04/23 06:32:23 | at     com.vmware.eam.vc.VcListener$4.call(VcListener.java:338)
    INFO | jvm 1 | 2015/04/23 06:32:23 | at com.vmware.eam.vc.VcListener$4.call(VcListener.java:335)
    INFO | jvm 1 | 2015/04/23 06:32:23 | at com.vmware.eam.lock.RaceProtection.exec(RaceProtection.java:68)
    INFO | jvm 1 | 2015/04/23 06:32:23 | at com.vmware.eam.lock.RaceProtection.exec(RaceProtection.java:51)
    INFO | jvm 1 | 2015/04/23 06:32:23 | at com.vmware.eam.vc.VcListener.unregisterObserver(VcListener.java:335)

    2015-04-23T06:35:43.033Z | DEBUG | agent-3 | AgentImpl.java | 2251 | Agent AgentImpl(ID:'Agent:86cab9ca-79da-412c-a8a3-b3f178121da5:3c36c535-2eb8-434e-b7c4-e07173a69e11') (VcHostSystem(ID: host-363)) scheduled a VIB task and is waiting for it to complete.

    For more information on vCenter Server log location, see Location of vCenter Server log files (1021804).

    Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.


Environment

VMware vCenter Server 6.0.x
VMware NSX for vSphere 6.0.x
VMware vCenter Server Appliance 6.0.x
VMware NSX for vSphere 6.1.x
VMware NSX for vSphere 6.2.x

Cause

This issue occurs due to the bug in EAM where it tries to poll the same updates again from VM exhausting the memory of the EAM.

Resolution

This issue is resolved in:
To work around the issue, increase the EAM heap size to the desired level using cloudvm-ram-size utility.
 
To increase the heap size:
  1. Stop the VMware VirtualCenter Server service that also stops the vCenter Inventory service. For more information, see Stopping, starting, or restarting VMware vCenter Server services (1003895).

    Note: To stop the EAM service on the vCenter Server, run this command:

    service vmware-eam stop
     
  2. Open command line utility and navigate to the folder C:\Program Files\VMware\vCenter Server\visl-integration\usr\sbin.
     
  3. Run the command:

    cloudvm-ram-size -C NewMemorysizeinMB EsxAgentManager

    In case of vCenter Appliance:

    cloudvm-ram-size -C NewMemorysizeinMB vmware-eam

    In case of Windows vCenter Server:

    cloudvm-ram-size -C NewMemorysizeinMB vspherewebclientsv
     
  4. Restart the VMware VirtualCenter Server service.

    To restart the eam service on vCenter Server, run this command:

    service vmware-eam restart
To confirm that heap size is increased, run the commands:
  1. cloudvm-ram-size -J vmware-eam -O /etc/vmware/java/vmware-eam_jvm.conf

    This output shows the allocated heap size for EAM.
     
  2. Echo $?

    0 indicates success, any non-zero value indicates failure.
     
  3. If there is a failure, check the file permissions of vmware-eam_jvm.conf and ensure read and write privileges on the file.

    ls -ltr /etc/vmware/java/vmware-eam_jvm.conf
    -rw------- 1 eam users 77 Mar 10 06:56 /etc/vmware/java/vmware-eam_jvm.conf


Additional Information

Powered by Wolken Software