Understanding VMM fault and VMM64 fault virtual machine monitor and executable failures

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

Services running on the guest operating system of a virtual machine do not respond
The guest operating system does not respond to ICMP (ping) network traffic
The virtual machine is powered off
In the virtual machine.log file, you see entry similar to:

<YYYY>T<time> vcpu-0| MONITOR PANIC: vcpu-0:VMM64 fault 14: src=MONITOR rip=0xfffffffffc000000 regs=0xfffffffffc008e80</time>

Or

<YYYY>T<time></time>: vmx| MONITOR PANIC: VMM fault: regs=0x2f9c, exc=14, eip=0xb01dc

Environment

VMware ESXi 3.5.x Embedded
VMware ESX 4.1.x
VMware ESXi 4.1.x Installable
VMware ESXi 4.1.x Embedded
VMware ESX Server 3.5.x
VMware ESX Server 3.0.x
VMware ESXi 3.5.x Installable
VMware ESXi 4.0.x Installable
VMware ESXi 4.0.x Embedded
VMware ESX 4.0.x

Resolution

Overview

A virtual machine monitor (VMM), a process that runs in the VMkernel, virtualizes the guest operating system instructions and manages memory. The VMM passes storage and network I/O requests to the VMkernel and passes all other requests to the VMX process. There is a VMM for each virtual CPU assigned to a virtual machine.

A VMM fault or VMM64 fault event in the virtual machine logs indicates that a virtual machine monitor responsible for running the virtual machine experienced a failure in the form of an exception. An Exception Handler is a computer hardware mechanism designed to handle some condition that changes the normal flow of execution (for example, Division by Zero or Page Fault. There is no trace from handlers, so you need logging to determine if the handler faulted (or single step debugging).

32bit VMM fault

The following is an example of the events on a 32-bit virtual machine:

<YYYY>T<time></time>: vmx| MONITOR PANIC: VMM fault: regs=0x2f9c, exc=14, eip=0xb01dc
<YYYY>T<time></time>: vmx| Core dump with build build-14062
<YYYY>T<time></time>: vmx| Writing monitor corefile "/usr/local/vm01/runtime/vm01/vmware-core0.gz"

The following describes important parts of the events:

VMM fault indicates that the monitor is 32-bit and that the failure that occurred was an exception.
exc=14 indicates that exception that was experienced was an exception 14 or a page fault.
Writing monitor corefile indicates where the the core dump of the VMM was stored. This dump file can be used to analyze the circumstances around the failure.
build-14062 represents the build number and eip=0xb01dc points to the section of code that was running at the time of the failure. If these two values are the same across multiple failures, it may indicate that the same instruction is running at the time of the failures.

64bit VMM Fault

The following is an example of the events on a 64-bit virtual machine:

<YYYY>T<time></time>: vcpu-1| MONITOR PANIC: vcpu-0:VMM64 fault 14: src=MONITOR rip=0xfffffffffc2e99d3 regs=0xfffffffffc008e98
<YYYY>T<time></time>: vcpu-0| Exiting vcpu-0
<YYYY>T<time></time>: vcpu-1| Core dump with build build-169571
<YYYY>T<time></time>: vcpu-1| Writing monitor corefile "/vmfs/volumes/48e9a96d-d9c74134-a5b8-001d0964e2fc/vm01/vmware-core0.gz"

The following describes important parts of the events:

VMM64 fault 14 indicates that the monitor is 64-bit and that the exception is an exception 14 or a page fault.
Writing monitor corefile indicates where the the core dump of the VMM was stored. This dump file can be used to analyze the circumstances around the failure.
build-169571 represents the build number and rip=0xfffffffffc2e99d3 points to the section of code that was running at the time of the failure. If these two values are the same across multiple failures, it may indicate that the same instruction is running at the time of the failures.

Exceptions

The following tables provides information about the exceptions that may be reported by the virtual machine monitor:

Type	Example error	Description
Exception 13 (General Protection Fault)	MONITOR PANIC: VMM fault: regs=0x2f9c, exc=13, eip=0xb01dc	Occurs when protection mechanisms within the processor architecture are violated. under any of the following circumstances: The page being requested does not belong to the program requesting it (and not mapped in program memory). The program does not have rights to perform a read, write, or execute operation to a page in memory (or the page was not designated to be used as such).
Exception 14 (Page Fault)	MONITOR PANIC: vcpu-0:VMM64 fault 14: src=MONITOR rip=0xfffffffffc2e99d3 regs=0xfffffffffc008e98\	Occurs when a program attempts to access a page mapped in the virtual address space, but it has not been successfully loaded into memory.
Exception 6 (Invalid Opcode)	MONITOR PANIC: vcpu-0:VMM64 fault 6: src=MONITOR rip=0xfffffffffc2d4f23 regs=0xfffffffffc008c40\	Occurs when the program sends an instruction to the CPU that either the CPU does not understand or supports, or it considers the instruction illegal.

Exceptions can be caused by either hardware or software issues. The Additional Information section of this article includes a list of knowledge base articles that include these exceptions as symptoms. If your symptoms do not exactly may any of the articles listed, collect the diagnostic information from the ESX host that is hosting your virtual machine and submit a support request. For more information, see Collecting Diagnostic Information for VMware Products (1008524) and How to Submit a Support Request.

Additional Information

If you experience an VMM fault or VMM64 fault event, see these articles for more information:

Note: If the error has not been documented in the Knowledge Base, collect diagnostic information from the ESX host that is hosting the virtual machine and submit a support request. For more information, see Collecting Diagnostic Information for VMware Products (1008524) and How to Submit a Support Request.