Correct way to set up custom NTP servers

Device type: Raspberry Pi (v1 / Zero / Zero W)
OS version: balenaOS 2.32.0+rev1
Supervisor version: 9.15.7

I am using the configizer @ https://github.com/balena-io/configizer to set up custom NTP servers. I set up the following parameter in config.sh

NTPSERVERS=“time.google.com 0.resinio.pool.ntp.org 1.resinio.pool.ntp.org time.apple.com
SSHKEYS=(“ssh-rsa AAA…”)

Upon running the run.sh script with my test device as the only id in the batch file, I got:

14bf78f383234e273d4dd0860268f3a9 : =============================================================
14bf78f383234e273d4dd0860268f3a9 :     Welcome to balenaOS
14bf78f383234e273d4dd0860268f3a9 : =============================================================
14bf78f383234e273d4dd0860268f3a9 : Inserting ntpServers values
14bf78f383234e273d4dd0860268f3a9 : Inserting sshKeys values
14bf78f383234e273d4dd0860268f3a9 : Keyindex: 0
14bf78f383234e273d4dd0860268f3a9 : Stopping supervisor before updating the original config.json
14bf78f383234e273d4dd0860268f3a9 : Running ntpServers post insert tasks
14bf78f383234e273d4dd0860268f3a9 : resin-ntp-config: Found config.json in /mnt/boot/config.json .
14bf78f383234e273d4dd0860268f3a9 : 511 Source already present
14bf78f383234e273d4dd0860268f3a9 : 503 No such source
14bf78f383234e273d4dd0860268f3a9 : 200 OK
14bf78f383234e273d4dd0860268f3a9 : 503 No such source
14bf78f383234e273d4dd0860268f3a9 : 200 OK
14bf78f383234e273d4dd0860268f3a9 : 503 No such source
14bf78f383234e273d4dd0860268f3a9 : 200 OK
14bf78f383234e273d4dd0860268f3a9 : 503 No such source
14bf78f383234e273d4dd0860268f3a9 : Running sshKeys post insert tasks
14bf78f383234e273d4dd0860268f3a9 : Restarting supervisor.
14bf78f383234e273d4dd0860268f3a9 : DONE

Is the format for setting the NTP servers correct or do I need to be specific, e.g. time1.google.com.

I confirmed that the changes were made to /mnt/boot/config.json.

My understanding from Use custom NTP server(s) is that the entries would be added to /etc/chrony.conf at runtime but they were not.

So curious about the 503 No such source messages, and my understanding in general.

Thanks

Hi

Do you know which one of the URLs is throwing the 503?

Hi, I don’t but I can do a few experiments. General question: is it ok to use time.google.com or do I need to be more specific and use time1.google.com?

Hello @0xff

It should be ok to use time.google.com but it might be worth it to try with time1.google.com just in case.

Cheers,
Nico,

I changed the ntp servers to defaults and ran the configizer.

NTPSERVERS="0.resinio.pool.ntp.org 1.resinio.pool.ntp.org 2.resinio.pool.ntp.org "

14bf78f383234e273d4dd0860268f3a9 : =============================================================
14bf78f383234e273d4dd0860268f3a9 :     Welcome to balenaOS
14bf78f383234e273d4dd0860268f3a9 : =============================================================
14bf78f383234e273d4dd0860268f3a9 : Inserting ntpServers values
14bf78f383234e273d4dd0860268f3a9 : Inserting sshKeys values
14bf78f383234e273d4dd0860268f3a9 : Keyindex: 0
14bf78f383234e273d4dd0860268f3a9 : Stopping supervisor before updating the original config.json
14bf78f383234e273d4dd0860268f3a9 : Running ntpServers post insert tasks
14bf78f383234e273d4dd0860268f3a9 : resin-ntp-config: Found config.json in /mnt/boot/config.json .
14bf78f383234e273d4dd0860268f3a9 : 200 OK
14bf78f383234e273d4dd0860268f3a9 : 503 No such source
14bf78f383234e273d4dd0860268f3a9 : 200 OK
14bf78f383234e273d4dd0860268f3a9 : 503 No such source
14bf78f383234e273d4dd0860268f3a9 : 200 OK
14bf78f383234e273d4dd0860268f3a9 : 503 No such source
14bf78f383234e273d4dd0860268f3a9 : Running sshKeys post insert tasks
14bf78f383234e273d4dd0860268f3a9 : Restarting supervisor.
14bf78f383234e273d4dd0860268f3a9 : DONE

I don’t understand what the 503s are referring to, but there are 3 for the 3 NTP servers I set up. I then limited it to a single value resinio.pool.ntp.org and it ran w/o errors.

14bf78f383234e273d4dd0860268f3a9 : =============================================================
14bf78f383234e273d4dd0860268f3a9 :     Welcome to balenaOS
14bf78f383234e273d4dd0860268f3a9 : =============================================================
14bf78f383234e273d4dd0860268f3a9 : Inserting ntpServers values
14bf78f383234e273d4dd0860268f3a9 : Inserting sshKeys values
14bf78f383234e273d4dd0860268f3a9 : Keyindex: 0
14bf78f383234e273d4dd0860268f3a9 : Stopping supervisor before updating the original config.json
14bf78f383234e273d4dd0860268f3a9 : Running ntpServers post insert tasks
14bf78f383234e273d4dd0860268f3a9 : resin-ntp-config: Found config.json in /mnt/boot/config.json .
14bf78f383234e273d4dd0860268f3a9 : Running sshKeys post insert tasks
14bf78f383234e273d4dd0860268f3a9 : Restarting supervisor.
14bf78f383234e273d4dd0860268f3a9 : DONE

Setting it to 1.resinio.pool.ntp.org does result in the same 503 error + a supervisor error to boot:

configizer> ./run.sh
14bf78f383234e273d4dd0860268f3a9 : =============================================================
14bf78f383234e273d4dd0860268f3a9 :     Welcome to balenaOS
14bf78f383234e273d4dd0860268f3a9 : =============================================================
14bf78f383234e273d4dd0860268f3a9 : Inserting ntpServers values
14bf78f383234e273d4dd0860268f3a9 : Inserting sshKeys values
14bf78f383234e273d4dd0860268f3a9 : Keyindex: 0
14bf78f383234e273d4dd0860268f3a9 : Stopping supervisor before updating the original config.json
14bf78f383234e273d4dd0860268f3a9 : Running ntpServers post insert tasks
14bf78f383234e273d4dd0860268f3a9 : resin-ntp-config: Found config.json in /mnt/boot/config.json .
14bf78f383234e273d4dd0860268f3a9 : 200 OK
14bf78f383234e273d4dd0860268f3a9 : 503 No such source
14bf78f383234e273d4dd0860268f3a9 : Running sshKeys post insert tasks
14bf78f383234e273d4dd0860268f3a9 : Restarting supervisor.
14bf78f383234e273d4dd0860268f3a9 : FAIL: Supervisor not restarted properly after while.

So it seems that for some reason specifying the keys in the expanded format such as 1.resinio.pool.ntp.org results in the 503 errors.

  • The documentation example specifically indicates setting
    "ntpServers": "0.resinio.pool.ntp.org 1.resinio.pool.ntp.org"
    but in my case at least it seems to result in 503 errors

A few other observations:

  • The values in the SSHKEYS array specified in config.sh are added to the list in /mt/boot/config.json, even if it’s the same key. So running run.sh 4 times will result in 4 copies of the same key in the sshKeys array. Not operationally a problem perhaps, but not clean.
  • After reboot, my device never came back. I had to power cycle it, and then it came back. The only relevant log error seems to be May 19 23:36:58 14bf78f openvpn[908]: Tue May 19 23:36:58 2020 ERROR: Linux route add command failed: external program exited with error status: 2.
  • Post power cycle, the file /etc/chrony.conf is unchanged. Shouldn’t the specified NTP values in the config file be written here to be used by chrony?

So when you specify time.google.com, the actual assigned NTP source will be one of the pool servers. I think this is the value that should be used.

Right, this is consistent with my findings that it doesn’t generate a 503. However, the example in the documentation indicates setting "ntpServers": "0.resinio.pool.ntp.org 1.resinio.pool.ntp.org" which in my case resulted in 503 errors. Either I’m doing something wrong or it could be helpful to change the documentation.

May I ask again:

Hi, let me add some context here.

When an NTP source is added using config.json (the configizer tool in your case) they are added to the running chrony using chronyc, as seen in https://github.com/balena-os/meta-balena/blob/87dacf84d83d5bf6e999847da9e86d9026dde73b/meta-balena-common/recipes-connectivity/resin-ntp-config/resin-ntp-config/resin-ntp-config#L19, that is:

		/usr/bin/chronyc add server $ntp minpoll 14 maxpoll 14 || true
		/usr/bin/chronyc burst 4/10 "$ntp" || true

This happens at every boot, but no change is recorded to /etc/chrony.conf, which is stored in a read-only filesystem.

This works fine when the NTP address or hostname used is an NTP source. When it is an NTP pool that can be resolved to a random NTP source, the NTP source is added fine, but the burst setting in the second line above fails with the 503 as it needs a valid NTP source. So, the NTP source will work but it will take longer to synchronize as the burst setting did not apply.
We are discussing mentioning this corner case in the documentation.

Hi, I created an issue to fix this at https://github.com/balena-os/meta-balena/issues/1903. We will ping you back in this thread once the issue is resolved.

Thanks for the detailed explanation. Very helpful.

I may be confused (often am :slight_smile: ) but in my experiments above Correct way to set up custom NTP servers. when I added the specific ntp servers in the form of "ntpServers": "0.resinio.pool.ntp.org 1.resinio.pool.ntp.org", that’s what generated the 503 errors in the configizer output. When I provide pool addresses, it does not.

Would you consider it a bug that if I run the script multiple times with the same ssh key line, it results in multiple duplicate entries in the config.json os.sshKeys entry? Seems to me it should replace the sshKeys rather than append to it.

Hi @0xff – I’m not one of the maintainers, but I encourage you to file an issue for that; if nothing else, it sounds like it could be better documented.

All the best,
Hugh

Done. Thanks.

@alexgg this wrt to your explanation:

Hi, Both “0.resinio.pool.ntp.org” and “pool.ntp.org” are pools. If you ping them you will get them resolved to different IPs:

root@9e9e53c:~# ping pool.ntp.org
PING pool.ntp.org (162.159.200.123): 56 data bytes
64 bytes from 162.159.200.123: seq=0 ttl=57 time=8.870 ms

root@9e9e53c:~# ping pool.ntp.org
PING pool.ntp.org (162.159.200.1): 56 data bytes

And

root@9e9e53c:~# ping 0.resinio.pool.ntp.org
PING 0.resinio.pool.ntp.org (162.159.200.1): 56 data bytes
64 bytes from 162.159.200.1: seq=0 ttl=57 time=12.221 ms

root@9e9e53c:~# ping 0.resinio.pool.ntp.org
PING 0.resinio.pool.ntp.org (147.156.7.18): 56 data bytes
64 bytes from 147.156.7.18: seq=0 ttl=52 time=21.890 ms

Chrony will reply with a 503 if the burst refers to a non valid source. What happens is that then we do the:

               /usr/bin/chronyc add server $ntp minpoll 14 maxpoll 14 || true
                /usr/bin/chronyc burst 4/10 "$ntp" || true

The “$ntp” variable above can resolve to different IP address if the hostname of a pool is passed. And the burst will fail with a 503 if the source has not been added first.
Try it manually:

ping 0.resinio.pool.ntp.org
PING 0.resinio.pool.ntp.org (74.6.168.72): 56 data bytes
64 bytes from 74.6.168.72: seq=0 ttl=47 time=211.647 ms

And then use that IP address, not the pool hostname:

/usr/bin/chronyc add server 74.6.168.72
/usr/bin/chronyc burst 4/10 74.6.168.72

That should not fail. But if you do the add to one source and then the burst to a different one, that may fail is the second one has not been added.
As I mention above, we will review the script to use the NTP source IP address.

@alexgg Thanks for the awesome explanation! Got it! I had mistakenly thought that resinio.pool.ntp.org was the pool and 0.resinio.pool.ntp.org was a particular source but your example clarifies that.

Glad to hear it worked!
Let us know if you need any further help with that

We recently made an improvement to meta-balena that improves how NTP pools are handled - for more detail please see the pull request.
This change should appear in the next balenaOS release.