Description
This article describes a guideline and commands to troubleshoot any NTP synchronization issue via CLI.
Solution
If the FortiGate is not able to sync the time with the configured NTP server, use the following commands to check the NTP server status:
get sys stat
execute date
execute time
diagnose sys ntp status
An example output of the ntp status command is seen below:
diagnose sys ntp status
synchronized: yes, ntpsync: enabled, server-mode: disabled
ipv4 server(time.google.com) 216.239.35.0 -- reachable(0xff) S:1 T:5
server-version=4, stratum=1
reference time is e18b5929.fc81eb59 -- UTC Fri Nov 29 08:45:29 2019
clock offset is 0.011700 sec, root delay is 0.000000 sec
root dispersion is 0.000183 sec, peer dispersion is 5 msec
ipv4 server(time.google.com) 216.239.35.4 -- reachable(0xff) S:1 T:5 selected
server-version=4, stratum=1
reference time is e18b5929.c0f08d85 -- UTC Fri Nov 29 08:45:29 2019
clock offset is 0.009796 sec, root delay is 0.000000 sec
root dispersion is 0.000214 sec, peer dispersion is 8 msec
ipv4 server(time.google.com) 216.239.35.12 -- reachable(0xff) S:1 T:5
server-version=4, stratum=1
reference time is e18b592a.21f2218 -- UTC Fri Nov 29 08:45:30 2019
clock offset is 0.009603 sec, root delay is 0.000000 sec
root dispersion is 0.000214 sec, peer dispersion is 5 msec
ipv6 server(time.google.com) 2001:4860:4806:8:: -- reachable(0xfe) S:0 T:4
no data
ipv6 server(time.google.com) 2001:4860:4806:4:: -- reachable(0xfe) S:0 T:4
no data
ipv6 server(time.google.com) 2001:4860:4806:: -- reachable(0xfe) S:0 T:4
no data
ipv6 server(time.google.com) 2001:4860:4806:c:: -- reachable(0xfe) S:0 T:4
no data
ipv4 server(time.google.com) 216.239.35.8 -- reachable(0xff) S:1 T:5
server-version=4, stratum=1
reference time is e18b592a.bcbcf -- UTC Fri Nov 29 08:45:30 2019
clock offset is 0.013359 sec, root delay is 0.000000 sec
root dispersion is 0.000198 sec, peer dispersion is 3 msec
execute time
current time is: 16:45:53
last ntp sync: Fri Nov 29 16:45:29 2019
If the NTP server is not reachable, change it to a different NTP server and verify afterwards if the time got synced properly.
To list the current NTP config run:
get system ntp
show full system ntp
To verify if the NTP service is running verify if this command returns a process ID (PID):
diagnose sys process pidof ntpd
If no process ID is returned the process is not running.
This can be double checked with the ps command which should show a process named 'ntpd':
fnsysctl ps
To verify if implicit firewall policy got added to accept remote ntp requests use the iprope command:
diagnose firewall iprope list | grep -f 123 -B11 -A1
diagnose firewall iprope list
To list open NTP sessions on port 123 run:
diagnose sys session filter clear
diagnose sys session filter dport 123
diagnose sys session list
diagnose sys session filter clear
Use the following command to create a network capture to verify if NTP packets are sent and received from the device:
diag sniffer packet any 'port 123' 6 0 l
To end the network capture after testing hit keys 'CTRL+C'.
To enable debug logging for the ntpd process on the application layer run:
diagnose debug reset
diagnose debug disable
diagnose debug console timestamp enable
diagnose debug application ntpd -1
Debug logging can be disabled after collecting the data with commands:
diagnose debug reset
diagnose debug disable
In case the ntpd process has a high CPU usage or a higher memory usage collect the following outputs while the issue is present:
First find the PID of the ntp process.
diagnose sys process pidof ntpd
Then dump details about the process IDs:
diagnose sys process pstack <PID>
diagnose sys process dump <PID>
fnsysctl ls -al /proc/<PID>
fnsysctl cat /proc/<PID>/status
fnsysctl cat /proc/<PID>/stack
fnsysctl cat /proc/<PID>/limits
fnsysctl cat /proc/<PID>/maps
fnsysctl ls -al /proc/<PID>/fd
In case of a high CPU usage terminate the ntpd process and create a backtrace. This backtrace can provide details about the function in which the process got stuck:
diag sys kill 11 <PID>
diag debug crashlog read
Where <PID> is the process ID previously found with command 'diagnose sys process pidof ntpd'.
It is also possible to keep track of the ntp status regularly using the API.
https://x.x.x.x/api/v2/monitor/system/ntp/status/?access_token='YourApiTokenHere'
This will provide similar data to 'diag sys ntp status' output which can be seen below:
