Symptoms:
- Virtual Machine tools status changes automatically from running to not running.
- Quiesced snapshots of Windows 2008R2/2012/2012R2 VMs with VMware Tools 10.1.0 or 10.1.5 are sporadically failing due to tools service not responding.
- Debug logging is been enabled for VMware Tools within Guest OS.
- In the vmsvc.log you will see entries similar to:
[ warning] [vmsvc] ----Win32 exception detected, exceptionCode 0xc0000005 (access violation)----
[ warning] [vmsvc] ExceptionAddress 0x7fef1be4cef eflags 0x00010202
[ warning] [vmsvc] rwFlags 0 badAddr 0x28
[ warning] [vmsvc] rax 0x1 rbx 0 rcx 0x4d9eb3c9e5d60000
[ warning] [vmsvc] rdx 0x1 rsi 0 rdi 0x980540
[ warning] [vmsvc] r8 0x980570 r9 0x3 r10 0x2
[ warning] [vmsvc] r11 0x693c5e000a001d r12 0x127f790 r13 0
[ warning] [vmsvc] r14 0 r15 0
[ warning] [vmsvc] rip 0x7fef1be4cef rsp 0x127f600 rbp 0x18f810
[ warning] [vmsvc] LastBranchToRip 0 LastBranchFromRip 0
[ warning] [vmsvc] LastExceptionToRip 0 LastExceptionFromRip 0
[ warning] [vmsvc] The following data was delivered with the exception:[ warning] [vmsvc] -- 0
[ warning] [vmsvc] -- 0x28
[ info] [vmsvc] CoreDump: Minidump file C:\Windows\TEMP\vmware-SYSTEM\vmtoolsd.dmp exists. Rotating ...
[ warning] [vmsvc] CoreDump: Writing minidump to C:\Windows\TEMP\vmware-SYSTEM\vmtoolsd.dmp
[ info] [vmsvc] CoreDump: including module base 0x0x13ff70000 size 0x0x00014000
- In the vmware.log, you see entries similar to:
vmx| I120: SnapshotVMX_TakeSnapshot start: '<backup name>-148911122268886c5d233d46c6169ae283395bd7bdf6b6af5e', deviceState=0, lazy=0, logging=0, quiesced=1, forceNative=0, tryNative=1, sibling=0 saveAllocMaps=0 cb=1D2B96E0, cbData=32574D80
vmx| I120: GuestRpcSendTimedOut: message to toolbox timed out.
vmx| I120: Tools: Tools heartbeat timeout.
vmx| I120: GuestRpcSendTimedOut: message to toolbox timed out.
vmx| I120: Msg_Post: Warning
vmx| I120: [msg.snapshot.quiesce.rpc_timeout] A timeout occurred while communicating with VMware Tools in the virtual machine.
| vmx| I120: ----------------------------------------
vmx| I120: ToolsBackup: changing quiesce state: IDLE -> DONE
vmx| I120: SnapshotVMXTakeSnapshotComplete: Done with snapshot 'Avamar-148911122268886c5d233d46c6169ae283395bd7bdf6b6af5e': 0
vmx| I120: SnapshotVMXTakeSnapshotComplete: Snapshot 0 failed: Failed to quiesce the virtual machine (40)
vmx| I120: GuestRpcSendTimedOut: message to toolbox timed out.
vmx| I120: Tools: Tools heartbeat timeout.
vmx| I120: GuestRpcSendTimedOut: message to toolbox timed out.
vmx| I120: Msg_Post: Warning
vmx| I120: [msg.snapshot.quiesce.rpc_timeout] A timeout occurred while communicating with VMware Tools in the virtual machine.
| vmx| I120: ----------------------------------------
vmx| I120: ToolsBackup: changing quiesce state: IDLE -> DONE
vmx| I120: SnapshotVMXTakeSnapshotComplete: Done with snapshot 'Avamar-148911122268886c5d233d46c6169ae283395bd7bdf6b6af5e': 0
vmx| I120: SnapshotVMXTakeSnapshotComplete: Snapshot 0 failed: Failed to quiesce the virtual machine (40)