Unable to take quiesced snapshot of Windows Server virtual machine with gpt disks without MSR partition

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

Creating a quiesced snapshot (or running a backup job with quiescing) fails with the following errors on Server 2012/2016 VM's:
On the vCenter, you see the error:

An error occurred while quiescing the virtual machine. See the virtual machine's event log for details.

The Vmware.log reports error:

2019-07-25T15:40:40.519Z| vmx| I125: SNAPSHOT: SnapshotPrepareTakeDoneCB: Prepare phase complete (The operation completed successfully).
2019-07-25T15:40:40.709Z| vcpu-1| I125: ToolsBackup: changing quiesce state: IDLE -> STARTED
2019-07-25T15:41:16.428Z| vmx| I125: Msg_Post: Warning
2019-07-25T15:41:16.428Z| vmx| I125: [msg.snapshot.quiesce.vmerr] The guest OS has reported an error during quiescing.
2019-07-25T15:41:16.428Z| vmx| I125+ The error code was: 5
2019-07-25T15:41:16.428Z| vmx| I125+ The error message was: 'VssSyncStart' operation failed: IDispatch error #8454 (0x80042306)
2019-07-25T15:41:16.428Z| vmx| I125: ----------------------------------------
2019-07-25T15:41:16.430Z| vmx| I125: ToolsBackup: changing quiesce state: STARTED -> ERROR_WAIT
2019-07-25T15:41:18.438Z| vmx| I125: ToolsBackup: changing quiesce state: ERROR_WAIT -> IDLE
2019-07-25T15:41:18.438Z| vmx| I125: ToolsBackup: changing quiesce state: IDLE -> DONE
2019-07-25T15:41:18.438Z| vmx| I125: SnapshotVMXTakeSnapshotComplete: Done with snapshot 'VM Snapshot 7%252f25%252f2019 9:40 AM UTC-06:00': 0
2019-07-25T15:41:18.438Z| vmx| I125: SnapshotVMXTakeSnapshotComplete: Snapshot 0 failed: Failed to quiesce the virtual machine (29).
2019-07-25T15:41:18.438Z| vmx| I125: VigorTransport_ServerSendResponse opID=TakeSnapshotFormMediator-add-4334090-ngc:70324732-f-e6-5e48 seq=55743: Completed Snapshot request.
2019-07-25T15:41:46.438Z| vmx| I125: Guest: *** WARNING: GuestInfo collection interval longer than expected; actual=73 sec, expected=30 sec. ***

Windows Event ID 8193 reports:

Volume Shadow Copy Service error: Unexpected error calling routine IOCTL_DISK_SET_SNAPSHOT_INFO(\\.\PHYSICALDRIVE1) fails with winerror 1168. hr = 0x80070490, Element not found.
Operation:
   Processing EndPrepareSnapshots
   Executing Asynchronous Operation
Context:
   Provider Name: VMware Snapshot Provider
   Provider Version: 1.0.0
   Provider ID: {564d7761-7265-2056-5353-2050726f7669}
   Current State: DoSnapshotSet

Volume Shadow Copy Service error: Unexpected error calling routine IOCTL_DISK_SET_SNAPSHOT_INFO(\\.\PHYSICALDRIVE1) fails with winerror 1168. hr = 0x80070490, Element not found.
Operation:
   Processing AbortSnapshots
   Executing Asynchronous Operation
Context:
   Device: \\.\PHYSICALDRIVE1
   Provider Name: VMware Snapshot Provider
   Provider Version: 1.0.0
   Provider ID: {564d7761-7265-2056-5353-2050726f7669}
   Current State: DoSnapshotSet

Environment

VMware vSphere ESXi 6.5
VMware vSphere ESXi 7.0.0

Cause

This issue occurs because the VM has a GPT disk without an MSR (Microsoft Reserved) partition. VMware Snapshot Provider works under Microsoft VSS framework. The Microsoft VSS framework will send the IOCTL IOCTL_DISK_SET_SNAPSHOT_INFO to the disks. If the disk is a GPT disk without a MSR partition, the IOCTL would fail and whole VSS process fails. Microsoft requires every GPT disk to have an MSR:

https://docs.microsoft.com/en-us/previous-versions/windows/hardware/design/dn640535(v=vs.85)#gpt_faq_what_is_msr

Resolution

To resolve this issue, use one of these options:

Option 1:

Add a new GPT disk with an MSR, and migrate the data to the new disk. Then, remove the old GPT disk. Microsoft support may need to verify the exact procedure, depending on what is stored on the disk.

Option 2:

If there is no space available to create a new MSR, shrink the volume by the default MSR size (32MB or 128MB). Microsoft support may need to verify the exact procedure for this.

Workaround:
To work around this issue, use one of these options:

Note: These options will prohibit application consistent backups, if there are no available 3rd party VSS Snapshot providers.

Option 1:

Do not use the "VMWare Snapshot Provider" service by disabling it, or uninstalling that component from VMware Tools.

Option 2:

Do not quiesce the guest OS when taking snapshots (or backups), or disable application level quiescing. For more information, see Disabling VSS quiesced application based snapshots in virtual machine.