NTP (chrony) not starting on newly provisioned device

Greetings,

I have encountered an issue regarding NTP time synchronization while provisioning a new device. I have been able to replicate the issue on 3 separate raspberry pi devices. I have read other posts on this forum, yet none seemed to address my issue.

The setup:

  • When provisioning the device, I have set-up the config.json file as the tutorial suggests, with "ntpServers":"<my ntp server IP>" value that contains my local NTP server IP.
  • Upon connecting and booting the device, the device doesn’t get provisioned, with logs stating “Error: certificate is not yet valid”.
  • I am able to ssh onto the device.
  • timedatectl shows that no NTP service is not available:
root@fd779a4:~# timedatectl
               Local time: Sun 2020-09-20 14:06:44 UTC
           Universal time: Sun 2020-09-20 14:06:44 UTC
                 RTC time: n/a
                Time zone: n/a (UTC, +0000)
System clock synchronized: no
              NTP service: n/a
          RTC in local TZ: no
  • The system clock is set to the creation of the image.
  • chronyc tracking shows 506 cannot talk to daemon.
  • When starting that newly provisioned device and connecting to it, the systemctl status chronyd command states that it is “inactive (dead)”.
● chronyd.service - NTP client/server
     Loaded: loaded (/lib/systemd/system/chronyd.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/chronyd.service.d
             └─chronyd.conf
     Active: inactive (dead)
       Docs: man:chronyd(8)
             man:chrony.conf(5)
  • The device then remains unsynchronized forever, not fetching the configuration due to certificates being invalid.
  • This issue persists even when removing the “ntpServers” value from config.json file, leaving the ntp sources to default while provisioning.

The steps i’ve taken as workarounds:

  1. Check chronyd logs with journalctl:
root@fd779a4:~# journalctl -u chronyd             
-- Logs begin at Sun 2020-09-20 10:51:53 UTC, end at Sun 2020-09-20 14:12:16 UTC. --
-- No entries --
  1. Attempt to start chronyd. service:
    systemctl start chronyd command hangs indefinitely, chronyd service never starts. No logs are being output.

  2. Manually forcing the time update with my server

I have manually forced the system time update with chronyd -q

root@eb468d1:~# chronyd -q 'server <my time-server IP> iburst'                                
2022-04-06T07:11:52Z chronyd version 4.0 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH +IPV6 -DEBUG)
2022-04-06T07:11:56Z System clock wrong by 87905.389257 seconds (step)
2022-04-07T07:37:02Z chronyd exiting

This command works and time is stepped correctly (I would assume this also verifies that my NTP server is able to provide time synchronization). Afterwards, the device fetches the correct configuration and starts services as it should. However, upon losing power and reconnecting, the system clock is delayed for the amount of time the device remains powered off.

It is also quite tedious for provisioning, having to ssh onto each device separately and force time stepping in order to get the correct initial configuration.

The BalenaOS version i’m using is balenaOS 2.88.5+rev1

Any insight would be greatly appreciated. Thank you in advance!

Hi, thanks for your message.

Is your device connected to the network from boot, or does it take some time to connect (for example a slow to setup cellular connection or a captive portal)?

What I would expect to see is that the timesync-https service makes a one-off time synchronization to make sure certificates can be used, and after that happens chronyd is launched.

Could you please post the output of journalctl --no-pager -u timesync-https.service?

Also, could you please make sure that you can access the balenaCloud API with:

curl https://api.balena-cloud.com/ping

Just, just following up on this. Is this still a problem? If so, could you please answer the questions above?