NTP Debugging Techniques

This page discusses ntpd program monitoring and debugging techniques using the ntpq - standard NTP query program, either on the local server or from a remote machine. The ntpq program implements the management functions specified in the NTP specification RFC 5905. In addition, the program can be used to send remote configuration commands to the server.

The ntpd daemon can operate in two modes, depending on the presence of the -n command-line option. Without the option the daemon detaches from the controlling terminal and proceeds autonomously. With one or more -d options the daemon generates special trace output useful for debugging. In general, interpretation of this output requires reference to the sources. However, a single -d does produce only mildly cryptic output and can be very useful in finding problems with configuration and network troubles.

Any Firewall needs to allow NTP traffic. The server side of ntpd listens on UDP port 123. The client side also sends from port 123 but not all implementations do that and ntpq sends from system assigned ports. If you are running an NTS server, you also need to allow TCP port 4460.

Other problems are apparent in the system log, which ordinarily shows the startup banner, some cryptic initialization data and the computed precision value. Event messages at startup and during regular operation are sent to the optional protostats monitor file, as described on the Event Messages and Status Words page. These and other error messages are sent to the system log, as described on the ntpd System Log Messages page. In real emergencies the daemon will send a terminal error message to the system log and then cease operation.

The next most common problem is incorrect DNS names. Check that each DNS name used in the configuration file exists and that the address responds to the Unix ping command. The Unix traceroute utility can be used to verify a partial or complete path exists. Most problems reported to the NTP newsgroup are not NTP problems, but problems with the network or firewall configuration.

If you use GPS, and your time is off by 19 years, you may have been bitten by the GPS 1024 week number rollover bug - WNRO. Please see Rollover issues in time sources

Unless using the iburst option, the client normally takes a few minutes to synchronize to a server. If the client time at startup happens to be more than 1000 s distant from NTP time, the daemon exits with a message to the system log directing the operator to manually set the time within 1000 s and restart. If the time is less than 1000 s but more than 128 s distant, a step correction occurs and the daemon restarts automatically.

When started for the first time and a frequency file - usually ntp.drift - is not present, the daemon enters a special mode in order to calibrate the frequency. This takes 900 ms during which the time is not disciplined. When calibration is complete, the daemon creates the frequency file and enters normal mode to amortize whatever residual offset remains.

The ntpq commands pe, as and rv are normally sufficient to verify correct operation and assess nominal performance. The pe command displays a list showing the DNS name or IP address for each association along with selected status and statistics variables. The first character in each line is the tally code, which shows which associations are candidates to set the system clock and of these which one is the system peer. The encoding is shown in the select field of the peer status word.

The as command displays a list of associations and association identifiers. Note the condition column, which reflects the tally code. The rv command displays the system variables billboard, including the system status word. The rv assocID command, where assocID is the association ID, displays the peer variables billboard, including the peer status word. Note that, except for explicit calendar dates, times are in milliseconds and frequencies are in parts-per-million (ppm).

A detailed explanation of the system, peer and clock variables in the billboards is beyond the scope of this page; however, a comprehensive explanation for each one is in the NTPv4 protocol specification. The following observations will be useful in debugging and monitoring.

Sometimes the time distribution of errors can be revealing. It’s a good idea to look occasionally at the plots produced by ntpviz.

The frequency tolerance of computer clock oscillators varies widely, sometimes above 500 ppm. While the daemon can handle frequency errors up to 500 ppm, or 43 seconds per day, values much above 100 ppm reduce the headroom, especially at the lowest poll intervals. To determine the particular oscillator frequency, start ntpd using the noselect option with the server configuration command.

Record the time of day and offset displayed by the ntpq peer command. Wait for an hour or so and record the time of day and offset. Calculate the frequency as the offset difference divided by the time difference. If the frequency offset is much above 100 ppm, the ntpfrob(8) program might be useful to adjust the kernel clock frequency below that value. For systems that do not support this program, this might be one using a command in the system startup file.

Provisions are included in ntpd for access controls which deflect unwanted traffic from selected hosts or networks. The controls described on the Access Control Options include detailed packet filter operations based on source address and address mask. Normally, filtered packets are dropped without notice other than to increment tally counters. However, the server can be configured to send a "kiss-o'-death" (KoD) packet to the client either when explicitly configured or when cryptographic authentication fails for some reason. The client association is permanently disabled, the access denied bit (BOGON4) is set in the flash variable and a message is sent to the system log.

The access control provisions include a limit on the packet rate from a host or network. If an incoming packet exceeds the limit, it is dropped and a KoD sent to the source. If this occurs after the client association has synchronized, the association is not disabled, but a message is sent to the system log. See the Access Control Options page for further information.

In some reported scenarios an access line may show low to moderate network delays during some period of the day and moderate to high delays during other periods. Often the delay on one direction of transmission dominates, which can result in large time offset errors, sometimes in the range up to a few seconds. It is not usually convenient to run ntpd throughout the day in such scenarios, since this could result in several time steps, especially if the condition persists for greater than the stepout threshold.

Specific provisions have been built into ntpd to cope with these problems. The scheme is called "huff-'n-puff and is described on the Miscellaneous Options page. An alternative approach in such scenarios is first to calibrate the local clock frequency error by running ntpd in continuous mode during the quiet interval and let it write the frequency to the ntp.drift file. Then, run ntpd -q from a cron job each day at some time in the quiet interval. In systems with the nanokernel or microkernel performance enhancements, including Solaris, Tru64, Linux and FreeBSD, the kernel continuously disciplines the frequency so that the residual correction produced by ntpd is usually less than a few milliseconds.

Reliable source authentication requires the use of symmetric key Authentication Options page. In symmetric key cryptography servers and clients share session keys contained in a secret key file. In public key cryptography, the server has a private key, never shared, and a public key with unrestricted distribution. Symmetric kays can be produced by the ntpkeygen program.

Problems with symmetric key authentication are usually due to mismatched keys or improper use of the trustedkey command. A simple way to check for problems is to use the trace facility, which is enabled using the ntpd -d command line. As each packet is received a trace line is displayed which shows the authentication status in the auth field. A status of 1 indicates the packet was successful authenticated; otherwise it has failed.

If the ntpq or program does not show that messages are being received by the daemon or that received messages do not result in correct synchronization, verify the following: