Search the VMware Knowledge Base (KB)
View by Article ID

Consumption of VMware Extended Guest Statistics on specific ESXi 5.5 releases or ESXi 6.0 may lead to virtual machine hanging issues (2137310)

  • 10 Ratings
Language Editions

Symptoms

When running applications, for example the SAP Host Agent 7.21 PL5, which make use of the latest ESXi Extended Guest Statistics within a virtual machine on an ESXi 5.5 Update 3 or ESXi 6.0 host, you may experience these symptoms in either Windows or Linux guest operating systems:
  • The virtual machine becomes hung irregularly.
  • In the esxtop  performance monitoring utility, you observe that the memory usage statistics of the virtual machine world (OVHD) are steadily increasing.
  • In the the virtual machine log file vmware.log, you observe:

    YYYY-MM-DDT<Time>Z| vcpu-1| I120: VERIFY bora/lib/misc/strutil.c:1079
    YYYY-MM-DDT<Time>Z| vcpu-1| W110: A core file is available in "/vmfs/volumes/<VMFS Datastore>/<Virtual Machine Name>/vmx-zdump.001"
    YYYY-MM-DDT<Time>Z| vcpu-1| W110: Writing monitor corefile "/vmfs/volumes/<VMFS Datastore>/<Virtual Machine Name>/vmmcores.gz"
    YYYY-MM-DDT<Time>Z| vcpu-1| W110: CoreDump error line 2160, error Cannot allocate memory


    For more information about the location of virtual machine log files, see  Locating virtual machine log files on an ESXi/ESX host (1007805)

  • In the /var/log/vmkernel.log  file, you see entries similar to: 

    YYYY-MM-DDT<Time>Z cpu21:101694)UserDump: 1907: Dumping cartel 100618 (from world 101694) to file /vmfs/volumes/<VMFS Datastore>/<Virtual Machine Name>/vmx-zdump.001 ...
    YYYY-MM-DDT<Time>Z cpu20:101694)UserDump: 2027: Userworld coredump complete.
    YYYY-MM-DDT<Time>Z cpu48:33427)WARNING: LinuxThread: 340: Error cloning thread: -28 (bad0081)
    YYYY-MM-DDT<Time>Z cpu28:100618)WARNING: World: vm 100618: 3974: VMMWorld group leader = 100619, members = 2
     

                Cause

                Recent enhancements in the Extended Guest Statistics collection allows the vendors to fetch metrics from within a guest operating system of a virtual machine. However, an issue has been found in the statistics collection in ESXi which leads to a memory leak and, in consequence, the virtual machine fails due to memory exhaustion.

                These collected statistics are, for example, heavily used in the context of SAP’s Host Agent (SAP Host Agent 7.21 PL5 and higher) and VMware vSphere (ESXi 5.5 Update 3 or ESXi 6.0).

                Resolution

                VMware understands the critical impact of this issue on its customers. The root cause has been identified.
                 
                This is a known issue affecting VMware vSphere (ESXi 5.5 Update 3, ESXi minor releases after Update 3b, ESXi 6.0, ESXi 6.0u1, or ESXi 6.0u1a).

                This issue is resolved in:
                If you are unable to upgrade and you are running one of the affected versions of VMware vSphere ESXi listed above and the SAP Host Agent (SAP Host Agent 7.21 PL5 or later), VMware strongly recommends you open a support request to engage direct assistance by a VMware Technical Support Engineer to review the issue and implement the most appropriate resolution option. Mention this document to the responding Technical Support Engineer. For details on opening a support request, see How to file a Support Request in My VMware (2006985).
                 
                To avoid experiencing this issue when using a third-party monitoring and statistics solution, postpone upgrading to vSphere 5.5 Update 3 or vSphere 6.0.x until an upgrade for your version of ESXi is available which fixes the issue. If you are already running ESXi 5.5 U3 or vSphere 6.0.x, your environment may not be affected if your statistics solution does not use VMware Extended Guest Statistics. Review the Administration guide and release notes of your third party statistics solution to see if the release you plan to install utilizes VMware Extended Guest Statistics. If yes, postpone the installation or upgrade of the solution or implement the workaround option until an upgrade for your version of ESXi is available which fixes the issue.
                 
                To work around this issue, you may perform a vMotion migration of the virtual machine between ESXi hosts to temporarily clear the memory leak. VMware recommends that this migration should happen at least once each week. If your environment doesn't already migrate the affected virtual machines often enough, you can automate the migrations using a Schedule Task operation.
                 
                A scheduled task can be used to periodically and automatically relocate the affected virtual machines. As stated, VMware recommends scheduling these tasks on a weekly basis. However, depending on the activity and load of the virtual machine(s), the scheduled task may need to be more frequent. For more information on creating a scheduled task for vMotion migration, see the Schedule Tasks section of the vSphere 6.0 vCenter Server and Host Management Guide.

                This article will be updated as soon as the regular patch release is available. You may subscribe to this article for RSS updates using the Subscribe to Document link in the Actions box.

                Additional Information

                For information in regards to the affected combination with SAP Host Agent 7.21 PL5, see the SAP Notes Related to VMware sub-article Note 2237937. This link requires a SAP account.

                Note: The preceding links were correct as of November 04, 2016. If you find a link is broken, provide feedback and a VMware employee will update the link.

                For more information on using esxtop to monitor performance issues, see Troubleshooting ESX/ESXi virtual machine performance issues (2001003).

                To be alerted when this article is updated, click Subscribe to Document in the Actions box.

                See Also

                Request a Product Feature

                To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

                Feedback

                • 10 Ratings

                Did this article help you?
                This article resolved my issue.
                This article did not resolve my issue.
                This article helped but additional information was required to resolve my issue.

                What can we do to improve this information? (4000 or fewer characters)




                Please enter the Captcha code before clicking Submit.
                • 10 Ratings
                Actions
                KB: