Search the VMware Knowledge Base (KB)
Products:
View by Article ID

Troubleshooting NTP on ESX and ESXi 4.x / 5.x / 6.x (1005092)

  • 32 Ratings

Purpose

This article provides troubleshooting steps for identifying and isolating problems with NTP time synchronization on ESX and ESXi hosts.

Resolution

Use these methods to troubleshoot NTP on ESXi/ESX hosts:

Validate connectivity using ping

Validate network connectivity between the ESXi/ESX host and the NTP server using the ping command. For more information, see Testing network connectivity with the ping command (1003486).

Query ntpd service using ntpq

Use the NTP Query utility program ntpq to remotely query the ESXi/ESX host's ntpd service.

The ntpq utility is commonly installed on Linux clients and is also available in the ESX service console and the vSphere Management Assistant. For more information on the installation and use of the ntpq utility program on a given Linux distribution, see your Linux distribution's documentation.

For an ESXi 5.x host, the ntpq utility is included by default and does not need to be installed. It can be run locally from the ESXi 5.x host.

The ntpq utility is not available on ESXi 3.x/4.x. To query an ESXi host's NTP service ntpd, install ntpq on a remote Linux client and query the ESXi host's ntpd service from the Linux client.

Notes:
  • ESXi/ESX 4.0 ships with restricted remote query permissions by default. To enable remote query of the NTP daemon, see Enabling remote query of NTP service on ESXi/ESX 4.0 (1036357).
  • There are known NTP server synchronization issues with certain versions of ESXi 5.x. These issues have been resolved in ESXi 5.x patch releases. Ensure your host is patched to the correct version. For full details, see ESXi host does not synchronize time with NTP server (2075424).
  • For ESXi 5.5 and 5.1, ntpq is already installed and a Linux client is not required. Enter these commands in an SSH shell or local console session.
  • Ensure that the Linux client has the NTP client tools installed. For more information, see your Linux distribution documentation or deploy an instance of the vSphere Management Assistant.
To use the NTP Query utility ntpq to remotely query the ESX host's NTP service (ntpd) and determine whether it is successfully synchronizing with the upstream NTP server:
  1. When using a Linux client, open a console session on the client where ntpq is installed.
  2. Run this command:

    When using an SSH shell or local console session on ESXi 5.5 and 5.1:

    "watch ntpq -p localhost_or_127.0.0.1"

    When using a Linux client for ESXi/ESX 4.x:

    watch "ntpq -p ESX_host_IP_or_domain_name"

  3. Monitor the output for 30 seconds and press Ctrl+C on your keyboard to stop the watch command.

    From the watch command, you see output similar to:

    Every 2 seconds: ntpq -p ESX_host_IP_or_domain_name

    Note: In ESXi 5.5 and 5.1, the output you see either localhost or loopback (127.0.0.1).

    remote        refid    st  t  when poll reach delay  offset  jitter
    ==============================================================================
    *10.11.12.130  1.0.0.0  1  u   46   64   377   43.76  5.58   40000


    Note: If you receive the message No association ID's returned, the ESXi/ESX host cannot reach the configured NTP server. If you receive the message ***Request timed out, the ntpq command did not receive a response from the ESXi/ESX host's NTP daemon. Skip to the Capture network traffic section below.

    The fields returned by ntpq have these meanings:

    remote Hostname or IP address of the configured upstream NTP server.
    refid Identification of the time stream to which the NTP server is synchronized. If you receive a refid of ".INIT.", the ESXi/ESX host has not received a response from the configured NTP server.
    st Stratum is a value representing the hierarchy of the upstream NTP servers. Higher values indicate NTP servers further away from the root time source. Values are relative, and can be set manually by an NTP server.
    t Type of packet exchange used for NTP communication. Usually "u" for unicast UDP.
    when Quantity of seconds which have elapsed since the last attempted poll of the configured upstream NTP server.
    poll Interval in seconds which the ESXi/ESX host polls the configured NTP server.
    reach An 8-bit shift register in octal (base 8), with each bit representing success (1) or failure (0) in contacting the configured NTP server. A value of 377 is 11111111 (base 2), which indicates that every query was successful during the last 8 poll intervals.
    delay Round trip delay (in milliseconds) for communication between the configured NTP server and the ESXi/ESX host.
    offset The offset (in milliseconds) between the time on the configured NTP server and the ESXi/ESX host. A value closer to 0 is ideal.
    jitter The observed timing jitter or variation between clock pulses of time with the configured NTP server. A value closer to 0 is ideal.
For more information, see the NTP.org Troubleshooting documentation and the NTP Query Program documentation.

The preceding links were correct as of October 3, 2013. If you find a link is broken, provide feedback and a VMware employee will update the link.

Review ntpd log entries

Review the ntpd log entries in the /var/log/messages log file to determine whether the NTP daemon has synchronized with the remote NTP server.
  1. Open a console to the ESXi/ESX host. For more information, see Unable to connect to an ESX host using Secure Shell (SSH) (1003807) or Using Tech Support Mode in ESXi 4.1 and ESXi 5.0 (1017910).

  2. View the /var/log/messages log file by running the command:

    Note: In ESXi 5.0, the NTP service logs to the /var/log/syslog.log file.

    less /var/log/messages
Notes:
  • Messages similar to this indicate that ntpd is successfully connecting to the remote NTP server:

    ntpd[263140]: synchronized to <ntp.server.ip.address>, stratum <X>

  • Messages similar to this indicate that the time offset between the ESXi/ESX host and the remote NTP server is too high to be corrected automatically. Set the ESXi/ESX host time manually and start the NTP daemon again.

    ntpd[263140]: time correction of <NNNN> seconds exceeds sanity limit (1000); set clock manually to the correct UTC time.

    In addition, a message similar to this may appear in the hostd.log file on ESXi 5.x:

    [info 'ha-eventmgr'] Event 91 : NTP daemon stopped. Time correction 1206 > 1000 seconds. Manually set the time and restart ntpd.

  • Messages similar to this indicate that the time error between the ESXi/ESX host and the remote NTP server exceeded the step threshold. The NTP daemon has corrected the ESXi/ESX host time to match the NTP server in a large jump. No action is needed.

    ntpd[263140]: time reset "+/- <xxx.yyyyyy s>"

  • Messages similar to this indicate that the time error between the ESXi/ESX host and the remote NTP server exceeded the step threshold. The NTP daemon will slowly correct the ESXi/ESX host time to match the NTP server. No action is needed.

    ntpd[263140]: time slew "+/- <xxx.yyyyyy s>"

Capture network traffic

Capture network traffic flowing between the ESXi/ESX host and the NTP server to determine whether packets are being sent and received.

For ESXi:
  1. Open a console to the ESXi host. For more information, see Using Tech Support Mode in ESXi 4.1 and ESXi 5.x (1017910).

  2. Obtain a list of available VMkernel network interfaces using this command:

    esxcfg-vmknic -l

  3. Capture NTP network traffic on port 123 flowing to and from the NTP server using this command:

    tcpdump-uw -c 5 -n -i network_interface host ntp_server_ip_address and port 123

    Example: When using a VMkernel interface vmk0 and an NTP server at 10.11.12.13:

    tcpdump-uw -c 5 -n -i vmk0 host 10.11.12.13 and port 123

  4. Monitor the output for 30 seconds. Messages similar to this indicate NTP synchronization:

    21:04:45.446566 172.16.24.16.ntp > 192.168.38.127.ntp: v4 client strat 2 poll 10 prec -16 (DF) [tos 0x10]

  5. Press Ctrl+C on your keyboard to stop tcpdump-uw.

For ESX:
  1. Open a console to the ESX host. For more information, see Unable to connect to an ESX host using Secure Shell (SSH) (1003807).

  2. Obtain a list of available Service Console network interfaces using this command:

    esxcfg-vswif -l

  3. Capture NTP network traffic on port 123 flowing to and from the NTP server using this command:

    tcpdump -c 5 -n -i network_interface host ntp_server_ip_address and port 123

    Example: When using the Service Console interface vswif0 and an NTP server at 10.11.12.13:

    tcpdump -c 5 -n -i vswif0 host 10.11.12.13 and port 123

  4. Monitor the output for 30 seconds. Messages similar to this indicate NTP synchronization:

    21:04:45.446566 172.16.24.16.ntp > 192.168.38.127.ntp: v4 client strat 2 poll 10 prec -16 (DF) [tos 0x10]

    Note: If there are no responses to NTP traffic returning to the ESX host, there may be an external firewall blocking UDP traffic on port 123.

  5. Press Ctrl+C on your keyboard to stop tcpdump.

    Create a log file of the delay occurring from NTP. This allows you to review the offset generated at the end of the day.

Review logs and network traffic

When successful network communication is established between the ESXi/ESX host and the NTP server, review the logs and network traffic to ensure that NTP synchronization is occurring and that the discrepancy is being reduced.

Note: It may take from 1 to 15 minutes to get the time synchronized after the packets are sent and received correctly by the ESXi/ESX host.

Additional Information

By default, ESXi/ESX uses NTPv4 but some NTP sources use NTPv3. The version mismatch leads to a synchronization failure. To resolve this, you must update the /etc/ntp.conf file to include the version you wish to use.

To update the /etc/ntp.conf file:
  1. Back up the /etc/ntp.conf file. Do not skip this step.
  2. Open the /etc/ntp.conf file in a text editor. For more information, see Editing configuration files in VMware ESXi and ESX (1017022).

  3. Add a line for the NTPv3 server:

    server x.x.x.x version 3

    For example, after making the modification, the contents of the ntp.conf file is similar to:

    restrict 127.0.0.1
    restrict default kod nomodify notrap
    driftfile /etc/ntp.drift
    server 192.168.0.10 version 3


  4. Save and close the file.
  5. Restart the NTP services for the change to take effect.

    For ESXi:
    # /etc/init.d/ntpd restart

    For ESX:
    # service ntpd restart

Note: To review the delay of the ntpq offset at end of day, create a folder named /var/log/ntp with the command:
mkdir /var/log/ntp
Append these 4 lines to the ntp.conf file:
statistics loopstats
statsdir /var/log/ntp/
filegen peerstats file peers type day link enable
filegen loopstats file loops type day link enable
The logs are now created in the new ntp directory.

Note: VMware recommends that you only configure one time service (netlogond or ntp). However, if you require NTP in conjunction with Active Directory (AD), configure the AD server to use a reliable time source and configure the NTP server for the ESXi/ESX host to use the AD server or the same NTP server that AD is using.

For more information on NTP and NetLogond, see the VMware documentation.

For related information, see Installing and Configuring NTP on an ESX host (1339) and Synchronizing ESXi/ESX time with a Microsoft Domain Controller (1035833).

See Also

Update History

03/27/2012 - Added note about time synchronization 07/04/2012 - Added information on the ntpq utility

Request a Product Feature

To request a new product feature or to provide feedback on a VMware product, please visit the Request a Product Feature page.

Feedback

  • 32 Ratings

Did this article help you?
This article resolved my issue.
This article did not resolve my issue.
This article helped but additional information was required to resolve my issue.

What can we do to improve this information? (4000 or fewer characters)




Please enter the Captcha code before clicking Submit.
  • 32 Ratings
Actions
KB: