VMware vCenter Server resource usage is 100% and becomes unresponsive
search cancel

VMware vCenter Server resource usage is 100% and becomes unresponsive

book

Article ID: 344108

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction


Symptoms:
  • The vpxd.exe process consumes 100% of the CPU resources.
  • The vpxd.exe process consumes large amount of the physical memory.
  • You cannot log into VMware vCenter Server.
  • In the performance chart for the cluster view, you see the message:

    The data source timed out

  • You see the error:

    The request failed because the remote server took too long to respond

  • In the C:\ProgramData\VMware\VMware VirtualCenter\Logs\vpxd.log file, you see calls to the vCenter Server PerformanceManager API multiple times per second that appear similar to:

    [2012-07-17 09:47:24.861 13880 info 'App' opID=51d096b0] [VpxLRO] -- BEGIN task-internal-19566612 --
    -- vim.PerformanceManager.queryStats -- A5113EB8-C947-4D5D-A477-B9CFB084AD19(374B3D36-B796-45FB-
    27-5A6464630DDD)


Environment

VMware vCenter Server 4.1.x
VMware vSphere ESXi 5.5
VMware vCenter Server 5.0.x
VMware vCenter Server 5.5.x
VMware vCenter Server 4.0.x
VMware vCenter Server 5.1.x

Cause

Excessive calls to the PerformanceManager API cause vpxd.exe to consume a large amount of resources. These calls are extremely resource intensive as they generate statistics on the fly, and are highly computational.

Some third-party monitoring products generate excessive calls to this API and drive up the resource load to unsustainable levels.

Resolution

The session ID observed in the vpxd.log file (in these examples, A5113EB8-C947-4D5D-A477-B9CFB084AD19) can be used to search the vpxd-profiler.log file to determine the offending user or service account name, as well as the originating IP address.

To resolve this issue:
  1. Search for the session ID in the vpxd-profiler.log file. This reveals an additional information about the origin of the session causing the resource load on the PerformanceManager interface.

    For example:

    B9CFB084AD19'/Username='DOMAIN\TivoliVI'/ClientIP='10.0.0.1'/PropertyCollector/NullCollectorCount/total 0
    /SessionStats/SessionPool/Session/Id='A5113EB8-C947-4D5D-A477-B9CFB084AD19'/Username='DOMAIN\TivoliVI'/ClientIP='10.0.0.1'/PropertyCollector/QueuedOpsCount/total 0
    /SessionStats/SessionPool/Session/Id='A5113EB8-C947-4D5D-A477-B9CFB084AD19'/Username='DOMAIN\TivoliVI'/ClientIP='10.0.0.1'/PropertyCollector/ReadLocked/total 0
    /SessionStats/SessionPool/Session/Id='A5113EB8-C947-4D5D-A477-B9CFB084AD19'/Username='DOMAIN\TivoliVI'/ClientIP='10.0.0.1'/PropertyCollector/TotalObjectCount/total 3239
    /SessionStats/SessionPool/Session/Id='A5113EB8-C947-4D5D-A477-B9CFB084AD19'/Username='DOMAIN\TivoliVI'/ClientIP='10.0.0.1'/PropertyCollector/TriggeredFiltersCount/total 1
    /SessionStats/SessionPool/Session/Id='A5113EB8-C947-4D5D-A477-B9CFB084AD19'/Username='DOMAIN\TivoliVI'/ClientIP='10.0.0.1'/PropertyCollector/TriggeredProcessGUReqs/total 0
    /SessionStats/SessionPool/Session/Id='A5113EB8-C947-4D5D-A477-B9CFB084AD19'/Username='DOMAIN\TivoliVI'/ClientIP='10.0.0.1'/SoapSession/Id='11808511'/Count/total 1


    In this example, the account used to call the PerformanceManager API was DOMAIN\TivoliVI, and it originated from a workstation with an IP of 10.0.0.1.

  2. In the vpx-profiler.log file, ensure that the CPU usage tracking counter is running at 100%. Locate lines which repeat approximately every 5 minutes which are similar to:

    /ProcessStats/ProcessorCpuUsage/mean 100


    • In vCenter 5.x, this value is expressed as a percentage multiplied by 100. Therefore 100% is represented by a value of 10000.
    • In vCenter 4.1 and earlier, this value is expressed as a percentage only, therefore 100% is represented by a value of 100.

  3. After you determine the user or service account and the origin of the connection, disable the account or service to prevent it from connecting to vCenter Server.

  4. After disabling the account or service, verify that resource usage returns to acceptable levels.

Note: If the issue is found to be with a third-party product, you must contact the vendor for additional support.


Additional Information

To be alerted when this document is updated, click the Subscribe to Article link in the Actions box..
Windows Server のリソースモニターを使用して vpxd.exe のリソース使用状況を判断する方法
VMware vCenter Server のリソース使用率が 100% に達し、応答しなくなる
VMware vCenter Server 资源使用率为 100% 并且无响应