Install, configure and troubleshoot NTP servers in Debian Linux

13 minute read

If you are reading this article, perhaps you have searched a lot about NTP server installation in Linux, issues and problems encountered and how to troubleshoot them! We know that there are so many articles about how to configure NTP servers but we could not find any which contains all the issues we faced in the same place!

Our story

A client of ours has a sever infrastructure where from the 40+ VMs in the tenant environment, only 3 are allowed to communicate with external NTP servers. Thus we had to install NTP servers in 3 Linux virtual machines in order to synchronise their clocks with the external pools and in the meantime act as internal environment primary NTP servers. The following diagram (kudos to AsciiFlow) summarizes the installation:

                  +---------------------+
                  |     External NTP    |
                  |1.europe.pool.ntp.org|
                  +----------+----------+
                             ^
                             |
                             |
        +---------------------------------------+
        |                    |                  |
        |                    |                  |
+-------+--------+  +--------+-------+  +-------+--------+
|      VM1       |  |      VM2       |  |      VM3       |
|  192.168.10.1  |  |  192.168.10.2  |  |  192.168.10.3  |
|    NTP srv1    |  |    NTP srv2    |  |    NTP srv3    |
+-------+--------+  +--------+-------+  +-----+----------+
        ^                    ^                ^
        |                    |                |
        +-------------------------------------+
                             |
                     +-------+--------+
                     |      VM4 (any) |
                     |  192.168.10.4  |
                     |   NTP client   |
                     +----------------+

Notes on diagram:

External NTP is one of the many NTP pools in the internet.
VM1, VM2 and VM3 are the Linux virtual machines which are allowed to connect to the internet using port 123 and synchronise their clocks with the external pool.
VM4 is any virtual machine (Linux or Windows) in the infrastructure which cannot communicate with external NTP servers, thus it will use VM1, VM2 and VM3 to update its clock.

IMPORTANT NOTES:

If you are a developer or a devops person and you have to communicate with network admins for any network issues you might encounter, please be meticulous and describe the problems in detail.

During this operation (yes, debugging was like war!), the majority of our problems were network related so we had to debug every step and inform network administrators in order to configure firewalls.

If your infrastructure is completely blocked by a firewall and you cannot access UDP port 123 (ntp) in any way, then you might consider using htpdate in VM1, VM2, VM3.

Install NTP Server in Debian Linux

So, to install NTP server in Debian Linux, you just execute:

sudo apt-get install ntp

in VM1, VM2, VM3 and VM4 . After it is installed, our efforts are focused on one single file: /etc/ntp.conf.

Configure NTP Server

As we mentioned, we open /etc/ntp.conf and edit the pools in VM1, VM2, VM3 according to our needs:

# /etc/ntp.conf, configuration for ntpd; see ntp.conf(5) for help

driftfile /var/lib/ntp/ntp.drift

# Enable this if you want statistics to be logged.
#statsdir /var/log/ntpstats/

statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable


# You do need to talk to an NTP server or two (or three).
#server ntp.your-provider.example

# pool.ntp.org maps to about 1000 low-stratum NTP servers.  Your server will
# pick a different set every time it starts up.  Please consider joining the
# pool: <http://www.pool.ntp.org/join.html>
server 3.gr.pool.ntp.org  iburst minpoll 6 maxpoll 8
server 1.europe.pool.ntp.org  iburst minpoll 6 maxpoll 8
server 2.europe.pool.ntp.org  iburst minpoll 6 maxpoll 8

# Undisciplined Local Clock. This is a fake driver intended for backup
# and when no outside source of synchronized time is available. 
server  127.127.1.0     # local clock
fudge   127.127.1.0 stratum 8

# Access control configuration; see /usr/share/doc/ntp-doc/html/accopt.html for
# details.  The web page <http://support.ntp.org/bin/view/Support/AccessRestrictions>
# might also be helpful.
#
# Note that "restrict" applies to both servers and clients, so a configuration
# that might be intended to block requests from certain clients could also end
# up blocking replies from your own upstream servers.

# By default, exchange time with everybody, but don't allow configuration.
restrict -4 default kod notrap nomodify nopeer noquery limited
restrict -6 default kod notrap nomodify nopeer noquery limited

# Local users may interrogate the ntp server more closely.
#restrict 127.0.0.1
#restrict ::1

# Needed for adding pool entries
restrict source notrap nomodify noquery

# Clients from this (example!) subnet have unlimited access, but only if
# cryptographically authenticated.
#restrict 192.168.123.0 mask 255.255.255.0 notrust

# If you want to provide time to your local subnet, change the next line.
# (Again, the address is an example only.)
#broadcast 192.168.123.255

# If you want to listen to time broadcasts on your local subnet, de-comment the
# next lines.  Please do this only if you trust everybody on the network!
#disable auth
#broadcastclient

NOTES:

We have put minpoll to 6 (2^6 = 64 sec) and maxpoll to 8 (2^8 = 256 sec) as we noticed that our local NTP servers tended to drift for minutes. Sometimes, +4 minutes in a day! You can find more information about minpoll and maxpoll here.
The following lines:
```
   server  127.127.1.0     # local clock
   fudge   127.127.1.0 stratum 8
```
are used in order to instruct the local server to act as a low stratum server and be preferred from the local clients. (more information : here)

Configure Linux NTP Client

We edit /etc/ntp.conf of VM4 and replace any external pools with our local NTP servers (VM1, VM2, VM3). Your configuration might look like the following:

# pool.ntp.org maps to about 1000 low-stratum NTP servers.  Your server will
# pick a different set every time it starts up.  Please consider joining the
# pool: <http://www.pool.ntp.org/join.html>
#pool 0.debian.pool.ntp.org iburst
#pool 1.debian.pool.ntp.org iburst
#pool 2.debian.pool.ntp.org iburst
#pool 3.debian.pool.ntp.org iburst
server 192.168.10.1 iburst minpoll 6 maxpoll 8
server 192.168.10.2 iburst minpoll 6 maxpoll 8
server 192.168.10.3 iburst minpoll 6 maxpoll 8

Again as you can you may notice we have put minpoll to 6 (2^6 = 64 sec) and maxpoll to 8 (2^8 = 256 sec) as we noticed that our local NTP servers tended to drift for minutes. Sometimes, +4 minutes in a day! You can find more information about minpoll and maxpoll here.

Configure Windows NTP Client

Our virtual machines use Windows Server 2012 R2 (for Windows 2003 and older see here). We followed the instructions from pool.ntp.org and this article. So we open a windows command line prompt with administrative privileges (WinKey + R + type “cmd” + hit enter key) and run :

w32tm /config /syncfromflags:manual /manualpeerlist:"192.168.10.1 192.168.10.2 192.168.10.3"

and then restart the time service so changes take effect and force it to resync:

net stop W32Time
net start W32Time

Troubleshooting

This is going to be the longest of all sections! Yes! Debugging was hard, we dare say! For our debugging purposes in Linux VMs we used ntpdate tool at first, although is deprecated. Also we have used ntpq and nmap. So this is what you have to do if you encounter the following errors both in your local NTP servers and clients.

NTP Server dropped: strata too high

If you see in syslog or in the output of the command:

sudo ntpdate -dv 2.europe.pool.ntp.org

the error:

91.228.108.200: Server dropped: strata too high

then, acccording to this and this article, the server is too far out of sync with the upstream servers, so it sets an artificially high stratum value to prevent other computers trusting it.

A possible solution is to set the time in the server manually as described here and then restart ntp service to see what happens.

No server suitable for synchronization found

If you use ntpdate to debug like this:

WARNING! NTP service has to be stopped in order to perform the following test.

sudo ntpdate -s -B -v 3.gr.pool.ntp.org

and the output in syslog is like this:

bobos@WEBSRV01:~$ sudo tail -f -n 10 /var/log/syslog
Nov 24 13:33:39 WEBWEBSRV01 ntpdate[26635]: ntpdate [email protected] Sat Nov 24 19:02:40 UTC 2017 (1)
Nov 24 13:33:48 WEBSRV01 ntpdate[26635]: no server suitable for synchronization found

then it might a network issue. To be sure, run nmap :

bobos@WEBSRV01:~$ sudo nmap -p123 -sU -P0 3.gr.pool.ntp.org
Starting Nmap 7.40 ( https://nmap.org ) at 2017-11-24 13:43 EEST
Nmap scan report for 3.gr.pool.ntp.org (194.177.210.54)
Host is up (0.00011s latency).
Other addresses for 3.gr.pool.ntp.org (not scanned): ::1
PORT    STATE  SERVICE
123/udp closed ntp

As you can see UTDP port 123 is closed, so we cannot communicate. Let’s run a second test:

WARNING! NTP service has to be stopped in order to perform the following test.

bobos@WEBSRV01:~$ sudo ntpdate -dv 2.europe.pool.ntp.org
24 Oct 12:02:20 ntpdate[26399]: ntpdate [email protected] Sat Sep 23 19:02:40 UTC 2017 (1)
transmit(87.118.124.35)
transmit(91.228.108.200)
transmit(194.177.4.1)
transmit(81.16.38.161)
transmit(87.118.124.35)
transmit(91.228.108.200)
transmit(194.177.4.1)
transmit(81.16.38.161)
transmit(87.118.124.35)
transmit(91.228.108.200)
transmit(194.177.4.1)
transmit(81.16.38.161)
transmit(87.118.124.35)
transmit(91.228.108.200)
transmit(194.177.4.1)
transmit(81.16.38.161)
transmit(87.118.124.35)
transmit(91.228.108.200)
transmit(194.177.4.1)
transmit(81.16.38.161)
87.118.124.35: Server dropped: no data
91.228.108.200: Server dropped: no data
194.177.4.1: Server dropped: no data
81.16.38.161: Server dropped: no data
server 87.118.124.35, port 123
stratum 0, precision 0, leap 00, trust 000
refid [87.118.124.35], delay 0.00000, dispersion 64.00000
transmitted 4, in filter 4
reference time:    00000000.00000000  Thu, Feb  7 2036  8:28:16.000
originate timestamp: 00000000.00000000  Thu, Feb  7 2036  8:28:16.000
transmit timestamp:  dd998022.d88eee1a  Tue, Oct 24 2017 12:02:26.845
filter delay:  0.00000  0.00000  0.00000  0.00000 
         0.00000  0.00000  0.00000  0.00000 
filter offset: 0.000000 0.000000 0.000000 0.000000
         0.000000 0.000000 0.000000 0.000000
delay 0.00000, dispersion 64.00000
offset 0.000000

server 91.228.108.200, port 123
stratum 0, precision 0, leap 00, trust 000
refid [91.228.108.200], delay 0.00000, dispersion 64.00000
transmitted 4, in filter 4
reference time:    00000000.00000000  Thu, Feb  7 2036  8:28:16.000
originate timestamp: 00000000.00000000  Thu, Feb  7 2036  8:28:16.000
transmit timestamp:  dd998023.0bc09a0a  Tue, Oct 24 2017 12:02:27.045
filter delay:  0.00000  0.00000  0.00000  0.00000 
         0.00000  0.00000  0.00000  0.00000 
filter offset: 0.000000 0.000000 0.000000 0.000000
         0.000000 0.000000 0.000000 0.000000
delay 0.00000, dispersion 64.00000
offset 0.000000

server 194.177.4.1, port 123
stratum 0, precision 0, leap 00, trust 000
refid [194.177.4.1], delay 0.00000, dispersion 64.00000
transmitted 4, in filter 4
reference time:    00000000.00000000  Thu, Feb  7 2036  8:28:16.000
originate timestamp: 00000000.00000000  Thu, Feb  7 2036  8:28:16.000
transmit timestamp:  dd998023.3ef4248c  Tue, Oct 24 2017 12:02:27.245
filter delay:  0.00000  0.00000  0.00000  0.00000 
         0.00000  0.00000  0.00000  0.00000 
filter offset: 0.000000 0.000000 0.000000 0.000000
         0.000000 0.000000 0.000000 0.000000
delay 0.00000, dispersion 64.00000
offset 0.000000

server 81.16.38.161, port 123
stratum 0, precision 0, leap 00, trust 000
refid [81.16.38.161], delay 0.00000, dispersion 64.00000
transmitted 4, in filter 4
reference time:    00000000.00000000  Thu, Feb  7 2036  8:28:16.000
originate timestamp: 00000000.00000000  Thu, Feb  7 2036  8:28:16.000
transmit timestamp:  dd998023.72276991  Tue, Oct 24 2017 12:02:27.445
filter delay:  0.00000  0.00000  0.00000  0.00000 
         0.00000  0.00000  0.00000  0.00000 
filter offset: 0.000000 0.000000 0.000000 0.000000
         0.000000 0.000000 0.000000 0.000000
delay 0.00000, dispersion 64.00000
offset 0.000000

24 Oct 12:02:29 ntpdate[26399]: no server suitable for synchronization found

As you can see from the output above, we cannot communicate with any server. Contact your network administrator.

NTPQ tool command shows zeroes in output

If you use ntpq tool and the output is the following:

bobos@WEBSRV01:~$ sudo date && ntpq -p
Tue Nov 24 13:26:45 EEST 2017
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 3.gr.pool.ntp.o .POOL.          16 p    -   64    0    0.000    0.000   0.000
 1.europe.pool.n .POOL.          16 p    -   64    0    0.000    0.000   0.000
 2.europe.pool.n .POOL.          16 p    -   64    0    0.000    0.000   0.000

then it means that you have no communication at all with your NTP servers. Contact your network administrator. A valid and working output would be:

bobos@WEBSRV01:~$ date && ntpq -p
Tue Nov 24 11:00:34 EEST 2017
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
* 3.gr.pool.ntp.o    .POOL.      4 u  988 1024  377    1.180   39.213  24.397
* 1.europe.pool.n    .POOL.      4 u  986 1024  377    1.080   38.984  24.397
* 2.europe.pool.n    .POOL.      4 u  985 1024  377    1.710   39.213  24.397

Bibliography

For a more “in depth dive” you can find more in the following links:

That’s it! We hope it helped you! It was a long story to debug and actually narrate afterwards!

Twitter Facebook LinkedIn

Install, configure and troubleshoot NTP servers in Debian Linux

Our story

Install NTP Server in Debian Linux

Configure NTP Server

Configure Linux NTP Client

Configure Windows NTP Client

Troubleshooting

NTP Server dropped: strata too high

No server suitable for synchronization found

NTPQ tool command shows zeroes in output

Bibliography

Comments

You May Also Enjoy

Linux Mint Mate configure screensaver and display sleep via command line

ORA-17026: Numeric overflow when calling Oracle SQL function using Hibernate 6

Kubernetes tail Spring Boot json logs in plain text Log4j (Logback) format using Stern

MS SQL Server Docker container restore database from .bak file created in Windows to a New Location