Troubleshooting JVM maximum memory heap problems in vCenter Server and vCenter Server Appliance 5.x
search cancel

Troubleshooting JVM maximum memory heap problems in vCenter Server and vCenter Server Appliance 5.x

book

Article ID: 344696

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:
  • An offline migration of a virtual machine, vMotion, or Storage vMotion fails with the error:

    • pbm.fault.PBMFault.summary
    • A general system error occured: Invalid response code 503: service unavailable

  • In the vpxd.log file, you see entries similar to:

    • YYYY-MM-DD TIME [7F9E4821F700 error 'VmProv' opID=E679B8A1-00009372-4e-93] [CallbackManager] Got exception while invoking precheck on PbmCallBack: Invalid response code: 503 Service Unavailable

      --> backtrace[00] rip 00007f9e54bdafa4 Vmacore::System::Stacktrace::CaptureWork(unsigned int)
      mem> YYYY-MM-DD TIME [7F9E4821F700 info 'Default' opID=E679B8A1-00009372-4e-93] [VpxLRO] -- ERROR task-internal-15041259 -- -- VmprovWorkflow: vmodl.fault.SystemError:
      mem> --> Result:
      mem> --> (vmodl.fault.SystemError) {
      mem> --> dynamicType = <unset>,
      mem> --> faultCause = (vmodl.MethodFault) null,
      mem> --> reason = "Invalid response code: 503 Service Unavailable",
      mem> --> msg = "",
      mem> --> }
      mem> --> Args:

    • <YYYY-MM-DD>T<time> [11208 error 'VmProv' opID=62adac7c-56] [CallbackManager] Got exception while invoking precheck on PbmCallBack: pbm.fault.PBMFault
      <YYYY-MM-DD>T<time> [11208 error 'commonvpxLro' opID=62adac7c-56] [VpxLRO] Unexpected Exception: pbm.fault.PBMFault
      <YYYY-MM-DD>T<time> [11208 info 'Default' opID=62adac7c-56] [VpxLRO] -- ERROR task-internal-180505 -- -- VmprovWorkflow: pbm.fault.PBMFault:
      --> (pbm.fault.PBMFault) {
      <YYYY-MM-DD>T<time> [11208 error 'Default' opID=62adac7c] (Log recursion level 2) pbm.fault.PBMFault
      <YYYY-MM-DD>T<time>[11208 error 'commonvpxLro' opID=62adac7c-56] [VpxLRO] Unexpected Exception: pbm.fault.PBMFault
      <YYYY-MM-DD>T<time> [11208 info 'Default' opID=62adac7c] [VpxLRO] -- ERROR task-13745 -- vm-133 -- vim.VirtualMachine.clone: pbm.fault.PBMFault:
      --> (pbm.fault.PBMFault)
      </time></time></time></time></time></time>

      Note: For more information on the location of vCenter Server and vCenter Server Appliance log file locations, see Location of vCenter Server log files (1021804).
  • In a Windows vCenter Server 5.5 the C:/ProgramData/VMware/Infrastructure/Profile-Driven Storage/Logs/\java_pid####.hprof consumes a large amount of disk space for the Profile-Driven Storage Service.
  • In the wrapper.log file for the Inventory Service or Profile-Driven Storage Service, you see entries similar to:

    INFO | jvm 3 | 2015/02/11 11:37:59 | java.lang.OutOfMemoryError: Java heap space
    INFO | jvm 3 | 2015/02/11 11:37:59 | Dumping heap to C:/ProgramData/VMware/Infrastructure/Profile-Driven Storage/Logs/\java_pid####.hprof ...


    Or

    INFO | jvm 4 | 2015/02/11 13:36:00 | java.lang.OutOfMemoryError: GC overhead limit exceeded
    INFO | jvm 4 | 2015/02/11 13:36:00 | Dumping heap to C:/ProgramData/VMware/Infrastructure/Profile-Driven Storage/Logs/\java_pid####.hprof ...


Note: This Knowledge Base article is applicable only if the wrapper.log file includes the java.lang.OutOfMemoryError error.

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment


Environment

VMware vCenter Server Appliance 5.1.x
VMware vCenter Server 5.1.x
VMware vCenter Server 5.0.x
VMware vCenter Server Appliance 5.0.x
VMware vCenter Server Appliance 5.5.x
VMware vCenter Server 5.5.x

Cause

This issue is occurs if the allocated JVM maximum memory heap is too low for the VMware Inventory service, VMware Profile-Driven Storage service, or VMware VirtualCenter Web Management service.

Several core vCenter Server services use individual configuration files (wrapper.conf) to determine the JVM maximum heap size that each service can use. The VMware Inventory service, VMware Profile-Driven Storage service, and VMware VirtualCenter Web Management service rely on the JVM maximum memory heap configured in their individual wrapper.conf file and play a critical role in the functionality of vCenter Server.

Note: If vCenter Server 5.5 has recently been updated beyond 5.5 U2, see Upgrading VMware vCenter Server 5.5 to 5.5 U2 or later reduces the JVM maximum memory heap size (2114669).

Resolution

This is a known issue affecting vCenter Server 5.x.

To resolve this issue, manually set the inventory size.

To change the JVM maximum memory heap size for a Windows installed vCenter Server 5.x, see, see Configuring Tomcat server settings in VMware vCenter Server 5.1 and 5.5 (2021302).

To manually set the inventory size in vCenter Server Appliance 5.x:
  1. Log in to vCenter Server Appliance administration interface (VAMI) URL:

    https://vCenter_Server_FQDN\IP:5480

  2. Click the Services tab.
  3. Change the Inventory Size to Medium or Large, depending on your environment.

    Note: The system memory may need to be increased when changing this setting. For more information on environment sizing, see the Configure Inventory Size for the VMware vCenter Server Appliance section of the Configuring Hosts and vCenter Server guide.

  4. Restart the VMware Profile-Driven Storage Service and VMware Inventory Service. For more information, see Stopping, starting, or restarting vCenter Server Appliance services (2054085).


Additional Information


Configuring Tomcat server settings in VMware vCenter Server 5.1 and 5.5
How to stop, start, or restart vCenter Server Appliance 5.x services
解决 vCenter Server 和 vCenter Server Appliance 5.x 中的 JVM 最大内存堆问题
vCenter Server と vCenter Server Appliance 5.x における JVM の最大メモリ ヒープの問題のトラブルシューティング