Knowledge Base

The VMware Knowledge Base provides support solutions, error messages and troubleshooting guides
 
Search the VMware Knowledge Base (KB)   View by Article ID
 

How to Run VMware Hosted Products on Systems on Which TSCs Are Not in Sync (2039)

Details

How can I work around problems on multiprocessor systems on which the timestamp counters do not stay in sync, such as IBM x-Series systems and some 64-bit AMD systems?

Solution

You must perform two actions.

  1. Disable a feature in some versions of VMware products that attempts to resynchronize the TSCs whenever a virtual machine is started. See the section Avoiding Forced TSC Resynchronization, below.
  2. Assign each virtual machine to a subset of processors whose TSCs are synchronized with each other. See the section Assigning a Virtual Machine to Processors with Synchronized TSCs, below.

Avoiding Forced TSC Resynchronization

On a Windows host operating system, you may encounter a problem with unwanted resynchronization of timestamp counters (TSCs) when a virtual machine starts. The workaround is to add the following line to your global configuration file:

host.TSC.noForceSync = TRUE

The global configuration file is normally found at:

  • C:\Documents and Settings\All Users\Application Data\VMware\VMware Workstation\config.ini for VMware Workstation
  • C:\Documents and Settings\All Users\Application Data\VMware\VMware GSX Server\config.ini for GSX Server
If this file does not exist, see http://kb.vmware.com/kb/1754.
 
Note (8/15/07): "Avoiding Forced TSC Resynchronization" is no longer necessary for current versions of VMware products, because the default value for that option is now TRUE. But it doesn't hurt if you have added the option explicitly.

Assigning a Virtual Machine to Processors with Synchronized TSCs

When a system has processors that have timestamp counters which are not all synchronized, the host operating system may move a virtual machine between two processors on which the timestamp counters are out of sync. This can cause the virtual machine clock to perform unpredictably. The clock may run too quickly or too slowly, or may even stop at times.

On an affected IBM x-Series system or its derivatives, each NUMA node (or CEC, in IBM terminology) has processors whose TSCs are synchronized with each other, but the TSCs of different NUMA nodes are not synchronized. So this issue can be solved by assigning each virtual machine to run only on the processors of a single NUMA node.

On affected 64-bit AMD systems, depending on which power management features are in use, every processor core's TSC may be out of sync with the others. (In other cases, the two cores in each dual-core processor may remain in sync.) If disabling power management does not solve this issue for you, it is safest to assign each virtual machine to only one processor core.

The details of how to assign a virtual machine to a subset of processors depend upon whether you are running GSX Server 3.2 or Workstation 5.5, or an earlier version of GSX Server or Workstation.

For Virtual Machines Running on a GSX Server 3.2, VMware Server 1.0, or Workstation 5.5 Host (or later versions)

These VMware products assign each virtual machine to a single NUMA node on an x440-class server running one of the following host operating systems:

  • Windows Server 2003
  • Any Linux 2.6.x kernel
  • Linux 2.4.21 or later kernel

When you power on a virtual machine, the VMware software by default assigns it to a NUMA node at random. You can configure a virtual machine to run on a specific NUMA node if you prefer.

To assign a virtual machine to a specific NUMA node, complete the following steps.

  1. Make sure the virtual machine is powered off.
  2. In a text editor, open the virtual machine's configuration file (.vmx file).
  3. Look for a line that starts with processors.NUMAnode =. If the line does not exist, add it.
  4. Change or set the value after the equal sign ( = ) to the number of the desired NUMA node. Put the value in quotation marks. For example, to assign the virtual machine to NUMA node 2, add the following line to its configuration file:
    processors.NUMAnode = "2"

To return this virtual machine to the default behavior, in which the VMware software assigns the virtual machine to a NUMA node at random, complete the following steps.

  1. Make sure the virtual machine is powered off.
  2. In a text editor, open the virtual machine's configuration file (.vmx file).
  3. Delete the line that starts with processors.NUMAnode =.
  4. Also delete any lines that start with processor<n>.use (where <n> is any number). These lines may be present if you previously applied the older workaround from GSX Server 3.1 and earlier or Workstation 5.0 and earlier, as described below.

In general, do not use the processor<n>.use option described below together with the processors.NUMAnode option. If both options are present in the configuration file, any processor<n>.use options are ignored.

If you are using GSX Server, VMware Server, or Workstation on a 64-bit AMD multiprocessor system, the VMware product does not assign each virtual machine to a subset of processor cores by default. If you need this assignment to be done on your 64-bit AMD system, choose a specific processor core to which to assign each virtual machine, using the processor<n>.use options as described in the next section.

For Virtual Machines Running on a Workstation 5.0 or Earlier Host, or on a GSX Server 3.1 or Earlier Host

To work around this problem for systems running on Workstation 5.0 or earlier, or GSX Server 3.1 or earlier, choose one NUMA node for each virtual machine and associate the virtual machine with the processors in that NUMA node. You can associate different virtual machines with different NUMA nodes; just make sure you do not allow any single virtual machine to run on multiple NUMA nodes. To associate a virtual machine with the processors in one NUMA node, complete the following steps.

  1. Make sure the virtual machine is powered off.
  2. In a text editor, open the virtual machine's configuration file (.vmx file).
  3. Add the following line for each processor with which you do not want to associate the virtual machine (where <n> is the number of the processor on the host):
    processor<n>.use = FALSE

For example, you have an eight-processor host with processors 0 through 3 on NUMA node 0 and processors 4 through 7 on NUMA node 1, and there are two virtual machines on the host. To associate the first virtual machine with NUMA node 0, add the following lines to that virtual machine's configuration file:

processor4.use = FALSE
processor5.use = FALSE
processor6.use = FALSE
processor7.use = FALSE

To associate the second virtual machine with NUMA node 1, add the following lines to that virtual machine's configuration file:

processor0.use = FALSE
processor1.use = FALSE
processor2.use = FALSE
processor3.use = FALSE

If your host has four processors, they may either be located all in one NUMA node or be split between two NUMA nodes. If all the processors are located on the same NUMA node, this problem does not occur. If the processors are split between two NUMA nodes, add the following lines to the virtual machine's configuration file to associate it with NUMA node 0:

processor2.use = FALSE
processor3.use = FALSE

Then add the following lines to the virtual machine's configuration file to associate it with NUMA node 1:

processor0.use = FALSE
processor1.use = FALSE

Caution: On a Linux host, GSX Server 3.1 (and earlier) and Workstation 5.0 (and earlier) do not honor the processor<n>.use option. You should not run GSX Server 3.1 and earlier, or Workstation 5.0 and earlier, at all on a machine that uses Linux as the host operating system and that has multiple NUMA nodes on which the TSCs are not synchronized. You need to upgrade to GSX Server 3.2 or Workstation 5.5.

Caution: The above examples assume that the GSX Server or Workstation host does not have hyperthreading enabled for its processors. For information about how hyperthreading affects which processors a virtual machine uses, read the next section.

How Hyperthreading Affects the Way in Which a Virtual Machine Is Associated with a Processor in a NUMA Node

When you enable hyperthreading on a host, the processor<n>.use option associates the virtual machine with CPU <n>, which is now a logical processor.

Continuing with the example above, if you enable hyperthreading on an eight-processor host with two NUMA nodes, and you want to associate a virtual machine with NUMA node 0, add the following lines to that virtual machine's configuration file:

processor4.use = FALSE
processor5.use = FALSE
processor6.use = FALSE
processor7.use = FALSE
processor12.use = FALSE
processor13.use = FALSE
processor14.use = FALSE
processor15.use = FALSE

When hyperthreading is enabled, an eight-processor Windows host has sixteen logical processors, numbered as follows:

  • Physical CPU 0: logical CPU 0, 8
  • Physical CPU 1: logical CPU 1, 9
  • Physical CPU 2: logical CPU 2, 10
  • Physical CPU 3: logical CPU 3, 11
  • Physical CPU 4: logical CPU 4, 12
  • Physical CPU 5: logical CPU 5, 13
  • Physical CPU 6: logical CPU 6, 14
  • Physical CPU 7: logical CPU 7, 15

Each NUMA node includes the following logical processors:

  • NUMA node 0 includes logical CPUs 0, 1, 2, 3, 8, 9, 10, 11
  • NUMA node 1 includes logical CPUs 4, 5, 6, 7, 12, 13, 14, 15

In enabling or disabling hyperthreading, use caution when associating virtual machines with processors. When you enable hyperthreading on the host, you should modify each virtual machine's configuration file to account for all the logical processors on the host.

However, disabling hyperthreading does not require you to modify the virtual machines' configuration files, as long as you do not make hardware changes to the host (such as adding or removing NUMA nodes or physical processors, or moving processors between NUMA nodes). GSX Server and Workstation ignore any processor<n>.use option where <n> is greater than the highest numbered processor available to the host operating system. Thus, with hyperthreading disabled, the options that you added for the first hyperthread on each physical processor (CPUs 0 through 7 above) now apply to the physical processor itself, while those you added for the second hyperthread (CPUs 8 through 15) are now ignored.

Caution: You must consider the change to the CPU numbering scheme when you add or remove NUMA nodes or physical processors to or from the host. With hyperthreading enabled, the number for each second logical processor changes when you add or remove a NUMA node or physical processor. Adding or removing a NUMA node or physical processor to or from a host requires you to re-associate virtual machines with the correct processors on each NUMA node.

Note: Knowledge base articles 2039, 2040, and 2041 replace knowledge base article 1236.

Keywords

2039; WS500; WS550; WS400; WS300; GSX200; GSX250; GSX300; urlz; dual core; dualcore

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 5 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.
What can we do to improve this information? (4000 or fewer characters)
  • 5 Ratings
Actions
KB: