Search the VMware Knowledge Base (KB)
View by Article ID
Consumption of VMware Extended Guest Statistics on specific ESXi 5.5 releases or ESXi 6.0 may lead to virtual machine hanging issues (2137310)
- The virtual machine becomes hung irregularly.
- In the esxtop performance monitoring utility, you observe that the memory usage statistics of the virtual machine world (OVHD) are steadily increasing.
- In the the virtual machine log file vmware.log, you observe:
YYYY-MM-DDT<Time>Z| vcpu-1| I120: VERIFY bora/lib/misc/strutil.c:1079
YYYY-MM-DDT<Time>Z| vcpu-1| W110: A core file is available in "/vmfs/volumes/<VMFS Datastore>/<Virtual Machine Name>/vmx-zdump.001"
YYYY-MM-DDT<Time>Z| vcpu-1| W110: Writing monitor corefile "/vmfs/volumes/<VMFS Datastore>/<Virtual Machine Name>/vmmcores.gz"
YYYY-MM-DDT<Time>Z| vcpu-1| W110: CoreDump error line 2160, error Cannot allocate memory
For more information about the location of virtual machine log files, see Locating virtual machine log files on an ESXi/ESX host (1007805)
- In the /var/log/vmkernel.log file, you see entries similar to:
YYYY-MM-DDT<Time>Z cpu21:101694)UserDump: 1907: Dumping cartel 100618 (from world 101694) to file /vmfs/volumes/<VMFS Datastore>/<Virtual Machine Name>/vmx-zdump.001 ...
YYYY-MM-DDT<Time>Z cpu20:101694)UserDump: 2027: Userworld coredump complete.
YYYY-MM-DDT<Time>Z cpu48:33427)WARNING: LinuxThread: 340: Error cloning thread: -28 (bad0081)
YYYY-MM-DDT<Time>Z cpu28:100618)WARNING: World: vm 100618: 3974: VMMWorld group leader = 100619, members = 2
Recent enhancements in the Extended Guest Statistics collection allows the vendors to fetch metrics from within a guest operating system of a virtual machine. However, an issue has been found in the statistics collection in ESXi which leads to a memory leak and, in consequence, the virtual machine fails due to memory exhaustion.
These collected statistics are, for example, heavily used in the context of SAP’s Host Agent (SAP Host Agent 7.21 PL5 and higher) and VMware vSphere (ESXi 5.5 Update 3 or ESXi 6.0).
VMware understands the critical impact of this issue on its customers. The root cause has been identified.
This is a known issue affecting VMware vSphere (ESXi 5.5 Update 3, ESXi minor releases after Update 3b, ESXi 6.0, ESXi 6.0u1, or ESXi 6.0u1a).
This issue is resolved in:
- ESXi 5.5 Update 3b, available at VMware Downloads. For more information, see the VMware ESXi 5.5 Update 3b Release Notes, and the Patch Release notice, VMware ESXi 5.5, Patch Release ESXi550-201512001 (2135410).
Important: Due to a change merge process error, subsequent ESXi 5.5 minor releases do not contain this fix. This is fixed in VMware ESXi 5.5, Patch Release ESXi550-201612001 (2147788) - Build 4722766.
For more information about downloading patches, see How to download patches in MyVMware (1021623).
The affected releases are:
- VMware ESXi 5.5, Patch Release ESXi550-201601501 (2141164) - Build 3343343 - Release Date: January 4, 2016
- VMware ESXi 5.5, Patch Release ESXi550-201602001 (2144353) - Build 3568722 - Release Date: February 21, 2016
- VMware ESXi 5.5, Patch Release ESXi550-201608001 (2144359) - Build 4179633 - Release Date: August 04, 2016
- VMware ESXi 5.5, Patch Release ESXi550-201609001 (2146717) - Build 4345813 - Release Date: September 15 , 2016
- ESXi 6.0 Update 1b and later, available at VMware Downloads. For more information, see the VMware ESXi 6.0 Update 1b Release Notes.
To avoid experiencing this issue when using a third-party monitoring and statistics solution, postpone upgrading to vSphere 5.5 Update 3 or vSphere 6.0.x until an upgrade for your version of ESXi is available which fixes the issue. If you are already running ESXi 5.5 U3 or vSphere 6.0.x, your environment may not be affected if your statistics solution does not use VMware Extended Guest Statistics. Review the Administration guide and release notes of your third party statistics solution to see if the release you plan to install utilizes VMware Extended Guest Statistics. If yes, postpone the installation or upgrade of the solution or implement the workaround option until an upgrade for your version of ESXi is available which fixes the issue.
To work around this issue, you may perform a vMotion migration of the virtual machine between ESXi hosts to temporarily clear the memory leak. VMware recommends that this migration should happen at least once each week. If your environment doesn't already migrate the affected virtual machines often enough, you can automate the migrations using a Schedule Task operation.
A scheduled task can be used to periodically and automatically relocate the affected virtual machines. As stated, VMware recommends scheduling these tasks on a weekly basis. However, depending on the activity and load of the virtual machine(s), the scheduled task may need to be more frequent. For more information on creating a scheduled task for vMotion migration, see the Schedule Tasks section of the vSphere 6.0 vCenter Server and Host Management Guide.
This article will be updated as soon as the regular patch release is available. You may subscribe to this article for RSS updates using the Subscribe to Document link in the Actions box.
Note: The preceding links were correct as of November 04, 2016. If you find a link is broken, provide feedback and a VMware employee will update the link.
For more information on using esxtop to monitor performance issues, see Troubleshooting ESX/ESXi virtual machine performance issues (2001003).
To be alerted when this article is updated, click Subscribe to Document in the Actions box.
Request a Product Feature
To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.