I was working with logging today and to test some things that happen at certain times of the day I was just changing the time on my test device. While I was happy that worked for testing purposes it raised some questions around how the supervisor deals with time.
I know Balena OS manages NTP servers and I think even uses it properly. (This comment here is how I learned about Balena in the first place)
When does the time get update via the NTP server? What if the device is offline? When will it retry and how often?
When does Balena update the hardware clock? If offline when does Balena use the hardware clock to offset time skew and set the time back to the hardware clock?
Thanks to anyone that can shine some light on this!
Q: What if the device is offline? When will it retry and how often?
A: The chrony.conf file that ships with recent versions of balenaOS seems use chrony’s default values. Therefore: “Their default values are 6 (64 seconds) for minpoll , 10 (1024 seconds) for maxpoll and 8 (samples) for polltarget,” as found in FAQ question “2.6 How can I improve the accuracy of the system clock with NTP sources?”
I assume that chrony keeps unsuccessfully retrying while offline, but no more often than minpoll seconds between retries.
Q: If offline when does Balena use the hardware clock to offset time skew and set the time back to the hardware clock?
A: I see that the rtcsync option is provided in chrony.conf, and the FAQ explains that:
5.1. What is the real-time clock (RTC)?
This is the clock which keeps the time even when your computer is turned off. It is used to initialise the system clock on boot. It normally doesn’t drift more than few seconds per day.
There are two approaches how chronyd can work with it. One is to use the rtcsync directive, which tells chronyd to enable a kernel mode which sets the RTC from the system clock every 11 minutes. chronyd itself won’t touch the RTC. If the computer is not turned off for a long time, the RTC should still be close to the true time when the system clock will be initialised from it on the next boot.
Whether the behaviour above is different while the device is offline… We may need to check chrony’s source code.
The only remaining question I have is how the hardware clock gets set in the first place. If chronyd won’t touch the RTC does balena?
I think this implies that if there is an RTC chronyd relies on it over NTP? It would be interesting to me if that means a device with an exsesive amount of uptime would start to skew off realtime.
The documentation balena provides seems to show a logical progression.
RTC --> first time
time file --> can overwrite to make sure we are going forward
NTP --> will update RTC
But does this ever happen again, and if not can I trigger it by restarting the supervisor from within my container?
But does this ever happen again, and if not can I trigger it by restarting the supervisor from within my container?
Adjustment of the hardware clock, as it’s stated in the docs, is done on boot only (before any application containers start).
And after chrony service is started, it syncs the clock using NTP, and on every successful sync it also updates the file timestamp.
Then, on next boot, the last update timestamp from the file is used to ensure the monotonic time. chrony service may be controlled with systemctl (check systemctl status chronyd).
Further reading has suggested that both the hardware clock and system clock will drift when offline. Going with the hardware clock doesn’t appear to be any more reliable than going with the system clock during that time. (From my early research at least.)
Precision between the different devices in a location is important to me and we are moving to a model where devices will only be online during a window in the evening. I was hoping to prevent time skew with the hardware clock.
However, it doesn’t seem like there are any improvements to be made.
