vSphere Client shows repeated power on tasks for vSphere Cluster Services (vCLS) VMs, all of which are failing with an error message:

Cluster Agent VM cannot be powered on due to insufficient resource

When looking at the detail for the failed tasks, there are other errors, such as:

The target host does not support the virtual machine's current hardware requirements.

or:

Feature'MWAIT'was absent, but must be present.

such as in the screenshot below:

the log of vCenter Server service, vpxd.log, contains the following error:

xxxxxxxxx  info vpxd[50390] [Originator@6876 sub=VmCheck opID=28aabd9e-01-02] CompatCheck results: (vim.vm.check.Result) [
-->    (vim.vm.check.Result) {
-->       vm = 'vim.VirtualMachine:e0803d6b-748b-41fb-8337-9dae8ac6e7c8:vm-5002',
-->       host = 'vim.HostSystem:e0803d6b-748b-41fb-8337-9dae8ac6e7c8:host-1095',
-->       error = (vmodl.MethodFault) [
-->          (vim.fault.FeatureRequirementsNotMet) {
-->             faultMessage = (vmodl.LocalizableMessage) [
-->                (vmodl.LocalizableMessage) {
-->                   key = "com.vmware.vim.vmfeature.cpuid.mwait",
-->                },
-->                (vmodl.LocalizableMessage) {
-->                   key = "com.vmware.vim.vpxd.vmcheck.featureRequirementsNotMet.useClusterOrPerVmEvc",
-->                }
-->             ],
-->             featureRequirement = (vim.vm.FeatureRequirement) [
-->                (vim.vm.FeatureRequirement) {
-->                   key = "cpuid.mwait",
-->                   featureName = "cpuid.mwait",
-->                   value = "Num:Match:1"
-->                }
-->             ],
-->             host = 'vim.HostSystem:e0803d6b-748b-41fb-8337-9dae8ac6e7c8:host-1095',
-->             msg = "",
-->          }
-->       ],
-->    }
--> ]

and in the virtual machine log (vmware.log) of the vCLS VMs the following entry can be found:

xxxxxxxxxxx| vmx| I125+ Power on failure messages: Feature 'cpuid.mwait' was absent, but must be present.

vCLS VMs are by default deployed with a "per VM EVC" mode that expects the CPU to provide the flag cpuid.mwait. For this, Monitor/MWAIT needs to be enabled in the BIOS of the physical server ESXi is running on.

To solve this problem, please enable Monitor/MWAIT in the BIOS of the ESXi.
Alternatively, please apply the steps outlined in the Workaround section of this article.

To workaround this issue, you can temporarily disable per VM EVC for the VCLS VMs.

Note: please be aware that the following workaround is only possible, if the version of the hosts where the vCLS VM are located on is at least ESXi 6.7. The workaround requires an upgrade of the virtual hardware version of the vCLS VMs to 14, which is only compatible with ESXi 6.7 an later versions.

To apply the workaround, take the following steps:

1. Locate the 1st powered off vCLS VM in vCenter and open the Configure tab. Notice that "VMware EVC" is not offered as an option.
2. Identify the ESXi host where the 1st vCLS resides. 
3. Open the ESXi Host Client for this ESXi and login as root .
4. Right click the vCLS VM within the host, and select "Upgrade VM Compatibility". Select "ESXi 6.7 virtual machine" to change the virtual hardware version to 14, and click on [Upgrade].
5. Confirm the VM Compatibility Upgrade (click on [YES]).
6. Return to the vCLS VM in vCenter and click one the Configure tab for the VM. Notice that "VMware EVC" is now an option. Edit EVC. Refresh vCenter if "Edit" is greyed out.
7. Disable EVC for the vCLS vm (this is temporary as EVC will actually then re-enable as Intel "Cascade Lake" Generation...for the purposes of satisfying the MWAIT error, this is an acceptable workaround).
8. Another vCLS will power on the cluster note this.

In certain cases, the issue could also be solved by enabling vCLS retreat mode (see https://kb.vmware.com/s/article/80472) to remove the existing vCLS VMs and disabling it again to deploy new ones.

HA and DRS cannot be enabled if the vCLS VMs cannot power on.