VMware
 

Knowledge Base

Search the Knowledge Base:
Products:
Search In:
 

Time falls behind in a virtual machine when the guest operating system writes to previously unwritten regions of its virtual disk

Symptoms

  • Time in virtual machine falls behind
  • Virtual machine becomes sluggish when time falls behind
  • The vCPU has a high %WAIT in esxtop
  • The hypervisor cannot deliver requested rate of timer interrupts

Purpose

The problem is that for some types of VMDKs, the first write to a previously unwritten disk block by the guest requires a synchronous write to zero the block. While this is happening, the guest is not able to run, so time in the virtual machine falls behind. This stops being an issue when the guest operating system reduces the frequency of writes to previously unwritten regions.

Resolution

Background

The default VMDK type for ESX is zeroedthick. Refer to the vmkfstools section in the Server Configuration Guide for your version of ESX/ESXi, for an explanation of various disk types on ESX. Disk space for this VMDK type is allocated at VMDK creation, but the disk blocks are zeroed only on demand as and when the guest operating system writes to a previously unwritten region of disk. Zeroing the disk blocks is a synchronous operation which causes the vCPU thread to block and prevents guest timer interrupts from being delivered. This results in time in the virtual machine falling behind.
 
This is an issue for zeroedthick, and thin VMDK types. It is not an issue for RDMs or eagerzeroedthick VMDKs.

Solution

There are two solutions:
  1. Use a tickless guest operating system kernel. This prevents the issue from being a timekeeping issue, but does not resolve the virtual machine being starved for resources.

    Note: Currently, SLES 10 SP2 running on ESX 3.5 is the only guest for which this works.

  2. Make sure all of the disk blocks are zeroed before the virtual machine goes into production.

    There are four ways to ensure this. Perform one of the following, based on your environment and comfort level:

    1. Create the VMDK using vmkfstools explicitly specifying the eagerzeroedthick format:

      vmkfstools -c 12G -d eagerzeroedthick -a lsilogic foo.vmdk

      This create a 12GB disk named foo.vmdk. The important part is -d eagerzeroedthick.

    2. After creating the virtual machine, run:

      vmkfstools --writezeros foo.vmdk

      This writes zeros to every block in the VMDK. Because of this, it must be done before the operating system is installed.

      WARNING: Any data previously stored in foo.vmdk is destroyed.

    3. Clone an existing (non-eager zeroed) VMDK to an eagerzeroedthick disk:

      vmkfstools -i originaldisk.vmdk newdisk.vmdk

      This produces a new VMDK that has the same contents as the original VMDK but with all unmodified blocks zeroed.

    4. Write to all of the disk blocks from within the virtual machine. Unfortunately, this is not easy, because a write is required. A read is not sufficient to cause the disk block to be written with zeros.

Additional Information

This problem is a specific cause of the more general problem of timer interrupt delivery falling behind in a virtual machine. This is discussed in detail in Time falls behind in a virtual machine when the memory of the virtual machine is paged from disk by the VMkernel (1005861).

Feedback

Rating: 1 - Lowest 2 3 4 5 - Highest (0 Ratings)   

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.
What can we do to improve this information? (2000 or fewer characters)
Submit
Rating: 1 - Lowest 2 3 4 5 - Highest (0 Ratings)   
Actions
  • KB Article: 1008284
  • Updated: Aug 14, 2009
  • Products:
    VMware ESX
    VMware ESXi
  • Product Versions:
    VMware ESX 3.0.x
    VMware ESX 3.5.x
    VMware ESXi 3.5.x Embedded
    VMware ESXi 3.5.x Installable