Hi, I don’t but I can do a few experiments. General question: is it ok to use time.google.com or do I need to be more specific and use time1.google.com?
14bf78f383234e273d4dd0860268f3a9 : =============================================================
14bf78f383234e273d4dd0860268f3a9 : Welcome to balenaOS
14bf78f383234e273d4dd0860268f3a9 : =============================================================
14bf78f383234e273d4dd0860268f3a9 : Inserting ntpServers values
14bf78f383234e273d4dd0860268f3a9 : Inserting sshKeys values
14bf78f383234e273d4dd0860268f3a9 : Keyindex: 0
14bf78f383234e273d4dd0860268f3a9 : Stopping supervisor before updating the original config.json
14bf78f383234e273d4dd0860268f3a9 : Running ntpServers post insert tasks
14bf78f383234e273d4dd0860268f3a9 : resin-ntp-config: Found config.json in /mnt/boot/config.json .
14bf78f383234e273d4dd0860268f3a9 : 200 OK
14bf78f383234e273d4dd0860268f3a9 : 503 No such source
14bf78f383234e273d4dd0860268f3a9 : 200 OK
14bf78f383234e273d4dd0860268f3a9 : 503 No such source
14bf78f383234e273d4dd0860268f3a9 : 200 OK
14bf78f383234e273d4dd0860268f3a9 : 503 No such source
14bf78f383234e273d4dd0860268f3a9 : Running sshKeys post insert tasks
14bf78f383234e273d4dd0860268f3a9 : Restarting supervisor.
14bf78f383234e273d4dd0860268f3a9 : DONE
I don’t understand what the 503s are referring to, but there are 3 for the 3 NTP servers I set up. I then limited it to a single value resinio.pool.ntp.org and it ran w/o errors.
14bf78f383234e273d4dd0860268f3a9 : =============================================================
14bf78f383234e273d4dd0860268f3a9 : Welcome to balenaOS
14bf78f383234e273d4dd0860268f3a9 : =============================================================
14bf78f383234e273d4dd0860268f3a9 : Inserting ntpServers values
14bf78f383234e273d4dd0860268f3a9 : Inserting sshKeys values
14bf78f383234e273d4dd0860268f3a9 : Keyindex: 0
14bf78f383234e273d4dd0860268f3a9 : Stopping supervisor before updating the original config.json
14bf78f383234e273d4dd0860268f3a9 : Running ntpServers post insert tasks
14bf78f383234e273d4dd0860268f3a9 : resin-ntp-config: Found config.json in /mnt/boot/config.json .
14bf78f383234e273d4dd0860268f3a9 : Running sshKeys post insert tasks
14bf78f383234e273d4dd0860268f3a9 : Restarting supervisor.
14bf78f383234e273d4dd0860268f3a9 : DONE
Setting it to 1.resinio.pool.ntp.org does result in the same 503 error + a supervisor error to boot:
configizer> ./run.sh
14bf78f383234e273d4dd0860268f3a9 : =============================================================
14bf78f383234e273d4dd0860268f3a9 : Welcome to balenaOS
14bf78f383234e273d4dd0860268f3a9 : =============================================================
14bf78f383234e273d4dd0860268f3a9 : Inserting ntpServers values
14bf78f383234e273d4dd0860268f3a9 : Inserting sshKeys values
14bf78f383234e273d4dd0860268f3a9 : Keyindex: 0
14bf78f383234e273d4dd0860268f3a9 : Stopping supervisor before updating the original config.json
14bf78f383234e273d4dd0860268f3a9 : Running ntpServers post insert tasks
14bf78f383234e273d4dd0860268f3a9 : resin-ntp-config: Found config.json in /mnt/boot/config.json .
14bf78f383234e273d4dd0860268f3a9 : 200 OK
14bf78f383234e273d4dd0860268f3a9 : 503 No such source
14bf78f383234e273d4dd0860268f3a9 : Running sshKeys post insert tasks
14bf78f383234e273d4dd0860268f3a9 : Restarting supervisor.
14bf78f383234e273d4dd0860268f3a9 : FAIL: Supervisor not restarted properly after while.
So it seems that for some reason specifying the keys in the expanded format such as 1.resinio.pool.ntp.org results in the 503 errors.
The documentation example specifically indicates setting "ntpServers": "0.resinio.pool.ntp.org 1.resinio.pool.ntp.org"
but in my case at least it seems to result in 503 errors
A few other observations:
The values in the SSHKEYS array specified in config.sh are added to the list in /mt/boot/config.json, even if it’s the same key. So running run.sh 4 times will result in 4 copies of the same key in the sshKeys array. Not operationally a problem perhaps, but not clean.
After reboot, my device never came back. I had to power cycle it, and then it came back. The only relevant log error seems to be May 19 23:36:58 14bf78f openvpn[908]: Tue May 19 23:36:58 2020 ERROR: Linux route add command failed: external program exited with error status: 2.
Post power cycle, the file /etc/chrony.conf is unchanged. Shouldn’t the specified NTP values in the config file be written here to be used by chrony?
Right, this is consistent with my findings that it doesn’t generate a 503. However, the example in the documentation indicates setting "ntpServers": "0.resinio.pool.ntp.org 1.resinio.pool.ntp.org" which in my case resulted in 503 errors. Either I’m doing something wrong or it could be helpful to change the documentation.
This happens at every boot, but no change is recorded to /etc/chrony.conf, which is stored in a read-only filesystem.
This works fine when the NTP address or hostname used is an NTP source. When it is an NTP pool that can be resolved to a random NTP source, the NTP source is added fine, but the burst setting in the second line above fails with the 503 as it needs a valid NTP source. So, the NTP source will work but it will take longer to synchronize as the burst setting did not apply.
We are discussing mentioning this corner case in the documentation.
Thanks for the detailed explanation. Very helpful.
I may be confused (often am ) but in my experiments above Correct way to set up custom NTP servers. when I added the specific ntp servers in the form of "ntpServers": "0.resinio.pool.ntp.org 1.resinio.pool.ntp.org", that’s what generated the 503 errors in the configizer output. When I provide pool addresses, it does not.
Would you consider it a bug that if I run the script multiple times with the same ssh key line, it results in multiple duplicate entries in the config.json os.sshKeys entry? Seems to me it should replace the sshKeys rather than append to it.
Hi @0xff – I’m not one of the maintainers, but I encourage you to file an issue for that; if nothing else, it sounds like it could be better documented.
The “$ntp” variable above can resolve to different IP address if the hostname of a pool is passed. And the burst will fail with a 503 if the source has not been added first.
Try it manually:
ping 0.resinio.pool.ntp.org
PING 0.resinio.pool.ntp.org (74.6.168.72): 56 data bytes
64 bytes from 74.6.168.72: seq=0 ttl=47 time=211.647 ms
And then use that IP address, not the pool hostname:
/usr/bin/chronyc add server 74.6.168.72
/usr/bin/chronyc burst 4/10 74.6.168.72
That should not fail. But if you do the add to one source and then the burst to a different one, that may fail is the second one has not been added.
As I mention above, we will review the script to use the NTP source IP address.
@alexgg Thanks for the awesome explanation! Got it! I had mistakenly thought that resinio.pool.ntp.org was the pool and 0.resinio.pool.ntp.org was a particular source but your example clarifies that.
We recently made an improvement to meta-balena that improves how NTP pools are handled - for more detail please see the pull request.
This change should appear in the next balenaOS release.
Thanks alexgg for the info on how config.json integrates into chronyc, because I was worried and also looking into it, because I saw no changes to the chronyc.conf file. But chronyc sources reveals that my custom servers are actually heared and used. Maybe an idea to add this info to the documentation?
) but in my experiments above