VMware VirtualCenter Server service fails due to a quiesced snapshot operation on the vCenter Server database virtual machine

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

vCenter Server database is located on a local/remote Database Server running in a virtual machine (Linux/Windows).
While taking a quiesced snapshot or while deleting the snapshot of the database virtual machine (for example, during a backup operation) vCenter Server loses connectivity to the database and fails.
Windows Event Viewer logs contain errors similar to:

An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) "ODBC error: (23000) - [Oracle][ODBC][Ora]ORA-00001: unique constraint (VMWARE_VC.PK_VPX_TASK) violated" is returned when executing SQL statement "INSERT INTO VPX_TASK (TASK_ID, NAME, DESCRIPTIONID, ENTITY_ID, ENTITY_TYPE, ENTITY_NAME, LOCKED_DATA, COMPLETE_STATE, CANCELLED, CANCELLABLE, ERROR_DATA, RESULT_DATA, REASON_DATA, QUEUE_TIME, EVENT_CHAIN_ID, USERNAME, VM_ID, HOST_ID, COMPUTERESOURCE_ID, DATACENTER_ID) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"
If the vpxd.log is set to verbose logging, the log contains entries similar to:

[YYYY-MM-DD 20:07:24.513 04560 error 'App'] [Vdb::Connection::Commit] Connection commit failed: -1. Start retrying 20 more times.
[YYYY-MM-DD 20:07:24.513 04560 verbose 'App'] [Vdb::Connection::Mend] Mending the connection
[YYYY-MM-DD 20:07:27.700 04560 verbose 'App'] [Vdb::Connection::Replay] Playing back INSERT INTO VPX_TASK (TASK_ID, NAME, DESCRIPTIONID, ENTITY_ID, ENTITY_TYP
E, ENTITY_NAME, LOCKED_DATA, COMPLETE_STATE, CANCELLED, CANCELLABLE, ERROR_DATA, RESULT_DATA, REASON_DATA, QUEUE_TIME, EVENT_CHAIN_ID, USERNAME, VM_ID, HOST_ID, COMPUTERESOURCE_ID, DATACENTER_ID) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
[YYYY-MM-DD 20:07:28.669 04560 error 'App'] [Vdb::IsRecoverableErrorCode] Unable to recover from 23000:1
[YYYY-MM-DD 20:07:28.669 04560 error 'App'] [VdbStatement] SQL execution failed: INSERT INTO VPX_TASK (TASK_ID, NAME, DESCRIPTIONID, ENTITY_ID,
ENTITY_TYPE, ENTITY_NAME, LOCKED_DATA, COMPLETE_STATE, CANCELLED, CANCELLABLE, ERROR_DATA, RESULT_DATA, REASON_DATA, QUEUE_TIME, EVENT_CHAIN_ID, USERNAME, VM_ID, HOST_ID, COMPUTERESOURCE_ID, DATACENTER_ID) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
[2011-06-23 20:07:28.669 04560 error 'App'] [VdbStatement] Diagnostic data from driver is 23000:1:1:[Oracle][ODBC][Ora]ORA-00001: unique constraint (VMWARE_VC.PK_VPX_TASK) violated

Note: For more information, see Enabling trivia logging in vCenter Server 4.0.x/4.1.x and 5.0.x/5.1.x/5.5.x (1001584).

Environment

VMware vCenter Server 5.0.x
VMware vCenter Server 5.5.x
VMware vCenter Server 5.1.x
VMware vCenter Server 4.1.x

Cause

This issue occurs if the vCenter Server database virtual machine is in a quiesced state during a snapshot operation.

The error Connection commit failed: -1 indicates that the vCenter Server is unaware whether the database commit operation was successful or failed.

The vCenter database layer (Vdb) replays the failed SQL statement requests to continue the vCenter operation. During the replay process, if it turns out that the previously failed SQL statement has been committed to the database, and if there is a unique constraint definition on the specific table, the ODBC driver reports the unique constraint violated error to the VMware VirtualCenter Server service and the service shuts down to prevent corruption of the vCenter Server database.

Resolution

Currently, VMware does not support quiesced snapshots of virtual machines running the vCenter Server database.

To work around this issue, use one of these options:

If quiesced snapshots are created by backup software to back up the virtual machine data, either use:
- A backup solution that provides application-level quiescing.
- Backup Agents in the guest operating system.
  
  Note: Any backup agent that quiesces the file system causes the issue described in this article.
If snapshots are created manually during virtual machine maintenance (for example, guest os patching, configuration changes), deselect the Quiesce guest file system option while taking the virtual machine snapshot.

Additional Information

Quiescing indicates pausing or altering the state of running processes on a computer, particularly those that might modify information stored on disk during a backup, to guarantee a consistent and usable backup.

To be alerted when this document is updated, click the Subscribe to Article link in the Actions box

Enabling trivia logging in VMware vCenter Server
Cannot create a quiesced snapshot because the snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine
vCenter Server データベース仮想マシンでの静止スナップショット操作が原因で VMware VirtualCenter Server サービスが失敗する
A virtual machine can freeze under load when you take quiesced snapshots or use custom quiescing scripts

Impact/Risks:
Disabling quiescing can result in a crash-consistent file system state.