Symptoms:
When performing a quiesced snapshot of a virtual machine, the backup fails with these errors in the /var/log/vmware/hostd.log file on the ESX host:

• Protocol error from VMX
• Quiesced snapshot via backup agent failed vmodl.fault.SystemError
• An error occurred while creating quiesced snapshot via backup agent an error occurred creating quiesced snapshot via backup agent

VMware Data Recovery 1.x
VMware ESX 4.0.x
VMware Consolidated Backup 1.5.x
VMware ESXi 4.0.x Installable
VMware ESXi 4.1.x Embedded
VMware vCenter Server 5.0.x
VMware Consolidated Backup 1.1.x
VMware vCenter Server 4.1.x
VMware ESXi 4.0.x Embedded
VMware vCenter Server 4.0.x
VMware vSphere ESXi 5.0
VMware VirtualCenter 2.5.x
VMware ESXi 4.1.x Installable
VMware ESXi 3.5.x Installable
VMware ESXi 3.5.x Embedded
VMware ESX Server 3.5.x
VMware ESX 4.1.x

To perform a quiesced snapshot without issues:

1. Verify that no third-party or partner backup software on the virtual machine affects or interferes with the functionality of Volume Shadow Copy Service. For more information, see Cannot create quiesced snapshots if backup or system recovery applications are installed (1009558).
2. Restart VMware Tools within the guest operating system.
3. Attempt the backup operation.

If the issue persists:

• If you are using VMware Consolidated Backup (VCB), disable quiescing from the VCB Proxy. This makes all backups crash-consistent.
• Use pre-freeze/post-thaw scripts to gracefully stop services which may adversely affect quiescing capabilities. For more information, see Running custom quiescing scripts inside Windows virtual machine with ESX 3.5 Update 2 and later (1006671).
• Disable the virtual machine's sync driver. This prevents quiescing of the virtual machine resulting in crash-consistent backups. For more information, see A virtual machine can freeze under load when you take quiesced snapshots or use custom quiescing scripts (5962168).
• If the time on the ESX/ESXi host and the guest virtual machine is not synchronized, disable VMware Tools time synchronization on the guest virtual machine. For more information, see Disabling Time Synchronization (1189).

Running custom quiescing scripts inside Windows virtual machine with ESX 3.5 Update 2 and later
Cannot create quiesced snapshots if backup or system recovery applications are installed
Disabling Time Synchronization
La copia de seguridad de máquinas virtuales falla con el error: Protocol error from VMX
备份虚拟机失败并出现错误：VMX 中的协议错误
「Protocol error from VMX」というエラーが表示されて仮想マシンのバックアップに失敗する
A virtual machine can freeze under load when you take quiesced snapshots or use custom quiescing scripts