The VMware Knowledge Base provides support solutions, error messages and troubleshooting guides
Search the VMware Knowledge Base (KB)
View by Article ID
NUMA imbalance after vMotion of virtual machines (2000740)
Symptoms
After performing vMotion operations on virtual machines, NUMA imbalance occurs on servers with a large number of cores per processor
All virtual machines appear to be running on a single NUMA node
Slow performance of virtual machines on a lightly-loaded ESX/ESXi host
esxtop reports high CPU RDY values for virtual machines
Cause
To avoid performance problems, the ESX/ESXi NUMA scheduler is coded to initially place virtual machines on a node where the most memory is available. If the nodes have very little memory, the scheduler uses a round robin algorithm to place virtual machines onto NUMA nodes. This typically only occurs when powering on a virtual machine.
With vMotion, the copy of the memory happens before the worlds are completely initiated. A single node is chosen as the home node to preserve memory locality and virtual machines are all placed on the single node. Performance issues can occur if there is a light workload running on the ESX/ESXi host. Under light workloads, NUMA node migrations do not occur as there is not enough load to warrant a migration.
Note: In some instances, this issue may still be encountered in ESX/ESXi 4.1 Update 2. VMware is aware of this issue.
To work around this issue when you do not want to upgrade to ESX/ESXi 4.1 Update 2, disable NUMA on the system by enabling Node Interleaving from the BIOS of the server. For more information on disabling node interleaving, see Performance Best Practices for VMware vSphere 4.1.
Request a Product Feature
To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.
