Handled a ntp issue these days and I want to share my learning here as I could not find the right and detail explanation until now.
The error message was like the title and when I checked the ntp service on the server, I found it worked well already while I needed to know the reason.
About the ntp configuration, we should follow below rules:
1. Better use 4 ntp servers as time source, at least 3 servers. 1 server is also OK while we should NOT use 2 servers.
2. If the time difference between the client and the server is more than 1000s, then the ntp service will exit and will not change the time. Have to run 'ntpdate' to set a closer time first.
3. If the time difference is less than 128ms, then client will slew the time with 0.5ms/s. If large than it, will reset the time directly. An option -x will increase the threshold to 600s.
4. NTP server was not designed to run inside of a virtual machine. It requires a high resolution system clock, with response times to clock interrupts that are serviced with a high level of accuracy.
Back to my issue and I found the client was configured with two ntp servers and one of them was restarted on the same day and the ntp server was a KVM VM.
After the reboot the time clock on the ntp server was a little different and I could confirm this from the client:
Aug 12 15:40:42 xxxxx0 daemon:info xntpd[17629290]: synchronized to 172.31.254.1, stratum=3 Aug 12 15:40:42 xxxxx0 daemon:notice xntpd[17629290]: time reset (step) -0.560307 s Aug 12 15:40:42 xxxxx0 daemon:info xntpd[17629290]: synchronisation lost Aug 12 15:40:47 xxxxx0 daemon:warn|warning xntpd[17629290]: Previous time adjustment didn't complete Aug 12 15:44:58 xxxxx0 daemon:info xntpd[17629290]: synchronized to 172.31.252.2, stratum=3 Aug 12 15:44:59 xxxxx0 daemon:notice xntpd[17629290]: time reset (step) 0.560628 s Aug 12 15:44:59 xxxxx0 daemon:info xntpd[17629290]: synchronisation lost
Such actions would last one hour then the ntp service would exit.
So there were two ntp servers while their time difference was more than 128ms and the client could not know which one it should select.
Our monitor team would start the ntp service manually per hour until the time difference was less than 128ms.
Aug 12 23:28:41 xxxxx0 daemon:info xntpd[19595298]: synchronized to 172.31.254.1, stratum=3 Aug 12 23:28:41 xxxxx0 daemon:notice xntpd[19595298]: time reset (step) 0.134670 s Aug 12 23:28:41 xxxxx0 daemon:info xntpd[19595298]: synchronisation lost Aug 12 23:28:43 xxxxx0 daemon:warn|warning xntpd[19595298]: Previous time adjustment didn't complete Aug 12 23:32:58 xxxxx0 daemon:info xntpd[19595298]: synchronized to 172.31.252.2, stratum=3 Aug 12 23:32:58 xxxxx0 daemon:notice xntpd[19595298]: time reset (step) -0.137917 s Aug 12 23:32:58 xxxxx0 daemon:info xntpd[19595298]: synchronisation lost Aug 12 23:36:22 xxxxx0 daemon:info xntpd[19595298]: SRC stop issued. Aug 12 23:36:22 xxxxx0 daemon:info xntpd[19595298]: exiting. Aug 12 23:36:22 xxxxx0 daemon:info src[22872246]: The xntpd subsystem was requested to STOPPED by user root Aug 12 23:36:24 xxxxx0 daemon:notice xntpd[19595310]: 3.4y Aug 12 23:36:24 xxxxx0 daemon:notice xntpd[19595310]: tickadj = 1000, tick = 10000, tvu_maxslew = 99000 Aug 12 23:36:24 xxxxx0 daemon:info xntpd[19595310]: precision = 10 usec Aug 12 23:36:24 xxxxx0 daemon:info src[22872252]: The xntpd subsystem was requested to STARTED by user root Aug 12 23:40:41 xxxxx0 daemon:info xntpd[19595310]: synchronized to 172.31.254.1, stratum=3
Now the client was synchronzied with the ntp server and the ntp service would not exit any more.
