VMware ESX 4.0.x
VMware ESX Server 3.5.x
VMware vSphere ESXi 6.0
VMware ESXi 3.5.x Embedded
VMware vSphere ESXi 7.0.x
VMware vCenter Server 5.5.x
VMware vSphere ESXi 7.0.0
VMware ESX 4.1.x
VMware VirtualCenter 2.5.x
VMware vSphere ESXi 6.7
VMware vSphere ESXi 8.0.0
VMware ESXi 4.0.x Embedded
VMware vCenter Server 4.0.x
VMware vSphere ESXi 8.0.1
VMware vCenter Server 5.1.x
VMware ESXi 4.0.x Installable
VMware vSphere ESXi 5.5
VMware vCenter Server 5.0.x
VMware vSphere ESXi 5.1
VMware vSphere ESXi 5.0
VMware vCenter Server 4.1.x
VMware ESXi 3.5.x Installable
VMware ESXi 4.1.x Installable
VMware ESXi 4.1.x Embedded
VMware vCenter Server 7.0.x

What is EVC?

EVC is short for Enhanced vMotion Compatibility. EVC allows you to migrate virtual machines between different generations of CPUs.

What is the benefit of EVC?

Because EVC allows you to migrate virtual machines between different generations of CPUs, with EVC you can mix older and newer server generations in the same cluster and be able to migrate virtual machines with vMotion between these hosts. This makes adding new hardware into your existing infrastructure easier and helps extend the value of your existing hosts. With EVC, full cluster upgrades can be achieved with no virtual machine downtime whatsoever. As you add new hosts to the cluster, you can migrate your virtual machines to the new hosts and retire the older hosts.

How do I use EVC?

EVC is enabled for a cluster in the VirtualCenter or vCenter Server inventory. After it is enabled, EVC ensures that migration with vMotion is possible between any hosts in the cluster. Only hosts that preserve this property can be added to the cluster.

How does it work?

After EVC is enabled, all hosts in the cluster are configured to present the CPU features of a user-selected processor type to all virtual machines running in the cluster. This ensures CPU compatibility for vMotion even though the underlying hardware might be different from host to host. Identical CPU features are exposed to virtual machines regardless of which host they are running on, so that the virtual machines can migrate between any hosts in cluster.

Can migrations with vMotion still fail for virtual machines within an EVC cluster?

Yes. More factors than CPU compatibility are used to determine vMotion compatibility. For example, if the virtual machine does not reside on storage shared by the source and destination hosts, then migration with vMotion cannot occur between these hosts.

Does EVC allow AMD and Intel CPUs to be vMotion compatible?

No. An EVC-enabled cluster only allows CPUs from a single vendor in the cluster. VirtualCenter and vCenter Server do not allow you to add a host from a different CPU vendor into an EVC-enabled cluster.

What EVC modes are available?

vCenter 5.0 supports these EVC modes:

AMD Opteron Generation 1 (Rev. E)
AMD Opteron Generation 2 (Rev. F)
AMD Opteron Generation 3 (Greyhound)
AMD Opteron Generation 3 (no 3Dnow!) (Greyhound)
AMD Opteron Generation 4 (Bulldozer)
Intel "Merom" Generation (Intel Xeon Core 2)
Intel "Penryn" Generation (Intel Xeon 45nm Core2)
Intel "Nehalem" Generation (Intel Xeon Core i7)
Intel "Westmere" Generation (Intel Xeon 32nm Core i7)
Intel "Sandy Bridge" Generation

vCenter 5.1 supports these EVC modes:

AMD Opteron Generation 1 (Rev. E)
AMD Opteron Generation 2 (Rev. F)
AMD Opteron Generation 3 (Greyhound)
AMD Opteron Generation 3 (no 3Dnow!) (Greyhound)
AMD Opteron Generation 4 (Bulldozer)
AMD Opteron "Piledriver" Generation
Intel "Merom" Generation (Intel Xeon Core 2)
Intel "Penryn" Generation (Intel Xeon 45nm Core2)
Intel "Nehalem" Generation (Intel Xeon Core i7)
Intel "Westmere" Generation (Intel Xeon 32nm Core i7)
Intel "Sandy Bridge" Generation
Intel "Ivy Bridge" Generation

vCenter 5.5 supports these EVC modes:

AMD Opteron Generation 1 (Rev. E)
AMD Opteron Generation 2 (Rev. F)
AMD Opteron Generation 3 (Greyhound)
AMD Opteron Generation 3 (no 3Dnow!) (Greyhound)
AMD Opteron Generation 4 (Bulldozer)
AMD Opteron "Piledriver" Generation
Intel "Merom" Generation (Intel Xeon Core 2)
Intel "Penryn" Generation (Intel Xeon 45nm Core2)
Intel "Nehalem" Generation (Intel Xeon Core i7)
Intel "Westmere" Generation (Intel Xeon 32nm Core i7)
Intel "Sandy Bridge" Generation
Intel "Ivy Bridge" Generation

vCenter 6.0 supports these EVC modes:

AMD Opteron Generation 1 (Rev. E)
AMD Opteron Generation 2 (Rev. F)
AMD Opteron Generation 3 (Greyhound)
AMD Opteron Generation 3 (no 3Dnow!) (Greyhound)
AMD Opteron Generation 4 (Bulldozer)
AMD Opteron "Piledriver" Generation
Intel "Merom" Generation (Intel Xeon Core 2)
Intel "Penryn" Generation (Intel Xeon 45nm Core2)
Intel "Nehalem" Generation (Intel Xeon Core i7)
Intel "Westmere" Generation (Intel Xeon 32nm Core i7)
Intel "Sandy Bridge" Generation
Intel "Ivy Bridge" Generation
Intel "Haswell" Generation

vCenter 6.5 supports these EVC modes:

AMD Opteron Generation 1 (Rev. E)
AMD Opteron Generation 2 (Rev. F)
AMD Opteron Generation 3 (Greyhound)
AMD Opteron Generation 3 (no 3Dnow!) (Greyhound)
AMD Opteron Generation 4 (Bulldozer)
AMD Opteron "Piledriver" Generation
AMD Opteron "Steamroller" Generation
Intel "Merom" Generation (Intel Xeon Core 2)
Intel "Penryn" Generation (Intel Xeon 45nm Core2)
Intel "Nehalem" Generation (Intel Xeon Core i7)
Intel "Westmere" Generation (Intel Xeon 32nm Core i7)
Intel "Sandy Bridge" Generation
Intel "Ivy Bridge" Generation
Intel "Haswell" Generation
Intel "Broadwell" Generation

vCenter 6.7 supports these EVC modes:

AMD Opteron Generation 1 (Rev. E)
AMD Opteron Generation 2 (Rev. F)
AMD Opteron Generation 3 (Greyhound)
AMD Opteron Generation 3 (no 3Dnow!) (Greyhound)
AMD Opteron Generation 4 (Bulldozer)
AMD Opteron "Piledriver" Generation
AMD Opteron "Steamroller" Generation
AMD "Zen" Generation
Intel "Merom" Generation (Intel Xeon Core 2)
Intel "Penryn" Generation (Intel Xeon 45nm Core2)
Intel "Nehalem" Generation (Intel Xeon Core i7)
Intel "Westmere" Generation (Intel Xeon 32nm Core i7)
Intel "Sandy Bridge" Generation
Intel "Ivy Bridge" Generation
Intel "Haswell" Generation
Intel "Broadwell" Generation
Intel "Skylake" Generation

vCenter 7.0 supports these EVC modes:

AMD Opteron Generation 1 (Rev. E)
AMD Opteron Generation 2 (Rev. F)
AMD Opteron Generation 3 (Greyhound)
AMD Opteron Generation 3 (no 3Dnow!) (Greyhound)
AMD Opteron Generation 4 (Bulldozer)
AMD Opteron "Piledriver" Generation
AMD Opteron "Steamroller" Generation
AMD "Zen" Generation
AMD "Zen 2" Generation
Intel "Merom" Generation (Intel Xeon Core 2)
Intel "Penryn" Generation (Intel Xeon 45nm Core2)
Intel "Nehalem" Generation (Intel Xeon Core i7)
Intel "Westmere" Generation (Intel Xeon 32nm Core i7)
Intel "Sandy Bridge" Generation
Intel "Ivy Bridge" Generation
Intel "Haswell" Generation
Intel "Broadwell" Generation
Intel "Skylake" Generation
Intel "Cascade Lake" Generation

vCenter 7.0 U2 supports these EVC modes:

AMD Opteron Generation 1 (Rev. E)
AMD Opteron Generation 2 (Rev. F)
AMD Opteron Generation 3 (Greyhound)
AMD Opteron Generation 3 (no 3Dnow!) (Greyhound)
AMD Opteron Generation 4 (Bulldozer)
AMD Opteron "Piledriver" Generation
AMD Opteron "Steamroller" Generation
AMD "Zen" Generation
AMD "Zen 2" Generation
Intel "Merom" Generation (Intel Xeon Core 2)
Intel "Penryn" Generation (Intel Xeon 45nm Core2)
Intel "Nehalem" Generation (Intel Xeon Core i7)
Intel "Westmere" Generation (Intel Xeon 32nm Core i7)
Intel "Sandy Bridge" Generation
Intel "Ivy Bridge" Generation
Intel "Haswell" Generation
Intel "Broadwell" Generation
Intel "Skylake" Generation
Intel "Cascade Lake" Generation
Intel "Ice Lake" Generation

vCenter 8.0 supports these EVC modes:

AMD Opteron Generation 1 (Rev. E)
AMD Opteron Generation 2 (Rev. F)
AMD Opteron Generation 3 (Greyhound)
AMD Opteron Generation 3 (no 3Dnow!) (Greyhound)
AMD Opteron Generation 4 (Bulldozer)
AMD Opteron "Piledriver" Generation
AMD Opteron "Steamroller" Generation
AMD "Zen" Generation
AMD "Zen 2" Generation
AMD "Zen 3" Generation
Intel "Merom" Generation (Intel Xeon Core 2)
Intel "Penryn" Generation (Intel Xeon 45nm Core2)
Intel "Nehalem" Generation (Intel Xeon Core i7)
Intel "Westmere" Generation (Intel Xeon 32nm Core i7)
Intel "Sandy Bridge" Generation
Intel "Ivy Bridge" Generation
Intel "Haswell" Generation
Intel "Broadwell" Generation
Intel "Skylake" Generation
Intel "Cascade Lake" Generation
Intel "Ice Lake" Generation

vCenter 8.0 U1/U2 supports these EVC modes:

AMD Opteron Generation 1 (Rev. E)
AMD Opteron Generation 2 (Rev. F)
AMD Opteron Generation 3 (Greyhound)
AMD Opteron Generation 3 (no 3Dnow!) (Greyhound)
AMD Opteron Generation 4 (Bulldozer)
AMD Opteron "Piledriver" Generation
AMD Opteron "Steamroller" Generation
AMD "Zen" Generation
AMD "Zen 2" Generation
AMD "Zen 3" Generation
AMD "Zen 4" Generation
Intel "Merom" Generation (Intel Xeon Core 2)
Intel "Penryn" Generation (Intel Xeon 45nm Core2)
Intel "Nehalem" Generation (Intel Xeon Core i7)
Intel "Westmere" Generation (Intel Xeon 32nm Core i7)
Intel "Sandy Bridge" Generation
Intel "Ivy Bridge" Generation
Intel "Haswell" Generation
Intel "Broadwell" Generation
Intel "Skylake" Generation
Intel "Cascade Lake" Generation
Intel "Ice Lake" Generation
Intel "Sapphire Rapids" Generation

Which CPUs are compatible with each EVC mode?

To determine the EVC modes compatible with your CPU, search the VMware Compatibility Guide. Search for the server model or CPU family, and click the entry in the CPU Series column to display the compatible EVC modes.

What are the differences between the modes?

Each mode corresponds very closely to the features available in processors with the same name. Newer processors and their corresponding modes include additional features such as new instructions. For example, the Intel Xeon Core i7 processor added the SSE4.2 instruction set. These instructions are not available in earlier processors or their corresponding EVC modes.

AMD EVC modes expose these features:

EVC Mode	Available Features
AMD Opteron Generation 1	All features of AMD Opteron Rev. E CPUs
AMD Opteron Generation 2	All features of AMD Opteron Generation and additional CPU features including CMPXCHG16B and RDTSCP
AMD Opteron Generation 3	All features of AMD Opteron Generation 2 and additional CPU features including SSE4A, MisAlignSSE, POPCOUNT, ABM (LZCNT)
AMD Opteron Generation 3 (no 3Dnow!)	Applies baseline feature set of AMD Opteron Generation 3 (Greyhound) processors, with 3DNow! support removed, to all hosts in the cluster. This mode allows you to prepare clusters containing AMD hosts to accept AMD processors without 3DNow! support.
AMD Opteron Generation 4	Applies baseline feature set of AMD Opteron Generation 4 (Bulldozer) processors to all hosts in the cluster. This EVC mode exposes additional CPU features including SSSE3, SSE4.1, AES, PCLMULQDQ, XSAVE, AVX, XOP and FMA4.
AMD Opteron "Piledriver" Generation	Applies baseline feature set of AMD Opteron "Piledriver" Generation processors to all hosts in the cluster. This EVC mode exposes additional CPU features including FMA3 (three operand FMA, aka Intel's FMA), BMI1 and TBM.
AMD Opteron "Steamroller" Generation	Applies baseline feature set of AMD Opteron "Steamroller" Generation processors to all hosts in the cluster. This EVC mode exposes additional CPU features including XSAVEOPT and FSGSBASE.
AMD "Zen" Generation	Applies baseline feature set of AMD "Zen" Generation processors to all hosts in the cluster. This EVC mode exposes additional CPU features including XSAVEOPT, RDFSBASE, RDGSBASE, WRFSBASE, WRGSBAS and FSGSBASE.
AMD "Zen 2" Generation	Applies baseline feature set of AMD "Zen 2" Generation processors to all hosts in the cluster. This EVC mode exposes additional CPU features including CLWB, UMIP, RDPID, XGETBV with ECX = 1, WBNOINVD, and GMET.
AMD "Zen 3" Generation	Applies the baseline feature set of AMD "Zen 3" Generation processors to all hosts in the cluster. This EVC mode exposes additional CPU features including always serializing LFENCE, INVPCID, PSFD, SSBD, PCID, PKU, VAES, VPCLMULQDQ, and shadow stacks.
AMD "Zen 4" Generation	Applies the baseline feature set of AMD "Zen 4" Generation processors to all hosts in the cluster. Compared to the AMD "Zen 3" Generation EVC mode, this EVC mode exposes additional CPU features including Fast Short CMPSB and STOSB, Automatic IBRS, AVX512BF16, AVX512BITALG, AVX512BW, AVX512CD, AVX512DQ, AVX512F, AVX512IFMA, AVX512VBMI, AVX512VBMI2, AVX512VL, AVX512VNNI, AVX512VPOPCNTDQ, GFNI, IBRS, and Upper Address Ignore.

Intel EVC modes expose these features:

EVC Mode	Available Features
Intel "Merom" Generation (Intel Xeon Core2)	All features of Intel Core2 CPUs
Intel "Penryn" Generation (Intel Xeon 45nm Core2)	All features of Intel Core2 CPUs and additional CPU features including SSE4.1
Intel "Nehalem" Generation (Intel Xeon Core i7)	All features of Intel Core2 CPUs and additional CPU features including SSE4.2 and POPCOUNT
Intel "Westmere" Generation (Intel Xeon 32nm Core i7)	Applies baseline feature set of Intel Xeon 32nm Corei7 (Westmere) processors to all hosts in the cluster. Compared to the Intel Xeon Corei7 mode, this EVC mode exposes additional CPU features including AES and PCLMULQDQ. Note: Intel i3/i5 Xeon Clarkdale Series processors that do not support AESNI and PCLMULQDQ cannot be admitted to EVC modes higher than the Intel Xeon Corei7 mode. Note: Intel Atom™ C2300-C2700 processors support the Intel "Westmere" Gen. EVC baseline although their architecture is different from the architecture of the Intel "Westmere" Generation processors.
Intel "Sandy Bridge" Generation	Applies baseline feature set of Intel “Sandy Bridge” Generation processors to all hosts in the cluster. This EVC mode exposes additional CPU features including AVX, XSAVE and ULE. Note: Some Sandy Bridge microarchitecture processors do not provide the full Sandy Bridge feature set. Such processors do not support EVC mode; they will only be admitted to the Intel Nehalem Generation mode or below.
Intel "Ivy Bridge" Generation	Applies baseline feature set of Intel “Ivy Bridge” Generation processors to all hosts in the cluster. This EVC mode exposes additional CPU features including ENFSTRING, F16C, FSGSBASE, SMEP and CPUID Faulting.
Intel "Haswell" Generation	Applies the baseline feature set of Intel "Haswell" Generation processors to all hosts in the cluster. This EVC mode exposes additional CPU features including Advanced Vector Extensions 2, fused multiply-adds, Transactional Synchronization Extensions, and new bit manipulation instructions.
Intel "Broadwell" Generation	Applies the baseline feature set of Intel "Broadwell" Generation processors to all hosts in the cluster. This EVC mode exposes additional CPU features including ADCX/ADOX, RDSEED, SMAP (CLAC/STAC), PREFETCHW,     RTM (Restricted Transactional Memory), and HLE (Hardware Lock Elision).
Intel "Skylake" Generation	Applies the baseline feature set of Intel "Skylake" Generation processors to all hosts in the cluster. This EVC mode exposes additional CPU features including Advanced Vector Extensions 512, Persistent Memory Support Instructions, Protection Key Rights, Save Processor Extended States with Compaction, and Save Processor Extended States Supervisor
Intel "Cascade Lake" Generation	Applies the baseline feature set of Intel "Cascade Lake" Generation processors to all hosts in the cluster. This EVC mode exposes additional CPU features including VNNI and XGETBV with ECX = 1.
Intel "Ice Lake" Generation	Applies the baseline feature set of Intel "Ice Lake" Generation processors to all hosts in the cluster. This EVC mode exposes additional CPU features including SHA extensions, Vectorized AES, User Mode Instruction Prevention, Read Processor ID, Fast Short REP MOV, WBNOINVD, Galois Field New Instructions, and AVX512 Integer Fused Multiply Add, Vectorized Bit Manipulation, and Bit Algorithms Instructions
Intel "Sapphire Rapids" Generation	Applies the baseline feature set of Intel® "Sapphire Rapids" Generation processors (including Intel® "Emerald Rapids" Generation processors) to all hosts in the cluster. This EVC mode exposes additional CPU features, including Control-Flow Enforcement Technology, Advanced Matrix Extensions, Supervisor Protection Keys, AVX-VNNI, AVX512 FP16, AVX512 BF16, CLDEMOTE, SERIALIZE, WBNOINVD, and MOVDIRI instructions.

 

How can I find the EVC mode supported by a host?

In the vSphere Client connected to vCenter Server 4.0, the host Summary tab indicates whether EVC is enabled, and displays the current EVC mode for the host. Click the blue icon next to the EVC mode to display a list of all the supported EVC modes for the host.

If I have an EVC-enabled cluster set to a particular EVC mode, and I have a host outside the cluster that matches the EVC mode, can I migrate a virtual machine with vMotion between that host and the cluster?

Yes, but not always.  In some cases, the EVC mode does not include all of the features of a stand alone host.  When a stand alone host has more features than its corresponding EVC mode, then a VM that was powered-on on the stand alone host may not be able to be vmotioned to the EVC cluster because the cluster may be missing some features.  The missing features will be reported by the UI or in the vpxd log file.
If the VM was powered-on in the EVC cluster, then it can be vmotioned to the stand alone host, and back.
An EVC mode for a CPU generation may not have all of features of the hardware for various reasons, such as having to pick a common set of features from a set of server SKUs for a CPU generation.

How do I handle Intel Xeon E3 and E5 Processors or Intel Xeon X-series, E-series, L-series Processors that have different revision or series numbers (e.g. v2, v3, v4 or x5680, x5687, x5690)

When enabling EVC for a cluster with Intel processors of the Xeon E3 or Xeon E5 family that have different revision numbers (v2, v3, v4), an EVC baseline is required. This is due to new instruction sets being available in the different revisions. Although the processors will be the same EVC baseline, this is required for the processors to all present the same instruction sets to the virtual machines.

As with Xeon process with different revisions, when enabling EVC for a cluster with Intel processors of the same Xeon family (e.g. E56xx, X56xx, L56xx), an EVC baseline is required. This is due to new instruction sets being available in the different revisions. Although the processors will be the same EVC baseline, this is required for the processors to all present the same instruction sets to the virtual machines.

What is the difference between EVC and the old CPUID masking feature (accessed from the Virtual Machine Settings dialog box, Options tab, CPUID mask option)?

The older masking feature involved applying manual masks to individual virtual machines. EVC takes effect on a whole cluster and all virtual machines in the cluster. More accurately, EVC affects the hosts themselves, making all the hosts in the cluster appear to be the same type of CPU hardware, even if they are different.

Why is the . vmx config file for the virtual machine updated with CPUID bits?

The .vmx file contains information about the features that the virtual machine is using. VirtualCenter and vCenter Server need this information to provide accurate vMotion compatibility checks. These values are unrelated to the old CPUID mask values that also appear in the .vmx file.

What happens when a host is removed from an EVC-enabled cluster?

When a host leaves an EVC-enabled cluster, it reverts to its normal behavior. New virtual machines started on that host can access all the features of the CPU, and are not limited by the EVC mode that was in effect while the host was in the EVC cluster. Note that virtual machines that were once able to migrate to the host might no longer be permitted to do so.

If I raise the EVC mode, will virtual machines be able to access the new features available in that mode?

Not until you power off and then power on the virtual machines. A reboot of the guest operating system is not sufficient. A virtual machine determines which features are available to it at power on, and cannot access any new features that are added until it is powered off.

What is an ill-behaved application, and why does it affect EVC?

An ill-behaved application is one that does not use CPU-vendor-recommended methods of detecting features supported on a CPU. The recommended method is to run the CPUID instruction and look for the correct feature bits for the capabilities the application is expected to use. Unsupported methods used by ill-behaved applications include try-catch-fail or inferring the features present from the CPU version information. When unsupported methods are used, an application might detect features on a host in an EVC cluster that are being masked from the virtual machines. The CPUID-masking MSRs provided by CPU vendors do not disable the actual features. Therefore, an application can still use masked features. If a virtual machine running such an application is then migrated with vMotion to a host that does not physically support those features, the application might fail. VMware is not aware of any commercially-available ill-behaved applications. For more information, see Detecting and Using CPU Features in Applications (1005763).

Can I create an EVC-enabled cluster if some of the hosts I want to add do not have AMD-V Extended Migration or Intel VT FlexMigration?

Yes. EVC-enabled clusters can contain hosts that do not feature AMD-V Extended Migration or Intel VT FlexMigration technology. For example, Intel Xeon Core2 processors do not have Intel VT FlexMigration technology, but can be added to clusters configured with an Intel Xeon Core2 EVC mode.

Can I add an ESXi/ESX 3.5 Update 1 or earlier host to an EVC-enabled cluster?

No. EVC is supported only on ESXi/ESX 3.5 Update 2 and later. vCenter Server does not permit the addition of hosts that do not provide support for EVC into an EVC-enabled cluster.

If I add newer hardware into an EVC-enabled cluster with a lower EVC mode, do I lose performance?

All CPU features provided by your host hardware are available to the hypervisor. Optimizations for CPU virtualization such as AMD-V and Intel VT-x or facilities for MMU virtualization such as AMD RVI or Intel EPT support are still used by the hypervisor. Only those CPU instructions that are unique to the new CPU are hidden from virtual machines when the host joins the EVC-enabled cluster. Typically this includes new SIMD instructions, such as the latest SSE additions. It is possible, but unlikely, that an application running in a virtual machine would benefit from these features, and that the application performance would be lower as the result of using an EVC mode that does not include the features. Check with the application vendor to determine which CPU features are used by the application.

How do I know when I can raise the EVC mode for a cluster?

     

What is the impact on vSphere Cluster Features when I disable the EVC mode?

Additional Information

• VMware Skyline Health Diagnostics for vSphere - FAQ
• The target host does not support the virtual machine's current hardware requirements
• https://blogs.vmware.com/vsphere/2019/06/enhanced-vmotion-compatibility-evc-explained.html​
• How to enable EVC in vCenter Server 6.5/6.7/7.0 if VC VM is part of the same cluster
• Skylake EVC virtual machines fail to power on
• vCenter Server 7.0 services fails to start in an EVC enabled cluster
• Virtual machine migration fails due to EVC mode mismatch

For translated versions of this article, see:

• Português: Compatibilidade do EVC e da CPU - Perguntas freqüentes (2015997)
• 日本語: EVC と CPU の互換性の FAQ (2011037)
• 简体中文: EVC 和 CPU 兼容性常见问题解答 (2081853)

Skylake EVC virtual machines fail to power on