Immediately after my device comes online, it pulls from the Docker image repository. The pull keeps failing because the time is not synced (ubuntu - Docker pull error : x509: certificate has expired or is not yet valid - Stack Overflow). Is there a way to trigger a time update immediately, and monitor when it is synced so I can return a message to the user instead of an error?
1 Like
Hi there. To check if the system clock is synchronised from within a container, you could try installing chrony within that container, and then use the command chronyc sources | grep "*" which will return the ntp server that the devices clock is sync’d to - if the device hasn’t sync’s its time yet then this command will return nothing.
When you say your device is failing to pull from the docker image repository, do you mean that your device can’t pull the balena application’s container images? Or are you pulling from the registry as a part of your application?
From the registry as part of my app using http://docker-py.readthedocs.io. Definitely not Balena related.
Thanks for the pointer on the chronyc sources. Ideally it would be better to trigger the sync and then wait for a reply, as using the grep option would require a loop to listen. I read about some methods using dbus (Chronyc config is bad if device comes online without internet - #27 by adamshapiro0) but it’s pretty complex for something quite simple. Shame to see that chronyc doesn’t seem to have anything like this in its CLI.
There is another potential issue, presumably once it has synced it will return a source for the sync every other time. But if there is clock drift while offline and then it comes back online, it will be out of sync again and yet my test will return that it is in sync.
I will keep playing around, but if any other ideas spring to mind on how to trigger the sync do let me know.
Another way to go which I am going to try and explore is not having the Balena-engine rely on the SSL for the registry (ubuntu - Docker pull error : x509: certificate has expired or is not yet valid - Stack Overflow). Perhaps even more complex though? It would be a better option though, as then the time sync plays no part and the user will always be able to pull from the registry at any time.
Full details from the docker-py end here: 404 on containers.create for valid images · Issue #2101 · docker/docker-py · GitHub
It seems it doesn’t immediately pull the time update, or not at least according to the chronyc sources test you suggested. It takes a little while. And then when it does trigger it restarts the containers. Would be nice to have more control over this.
Hi, there is an open pull request in meta-balena to introduce a one-time https time synchronization service [1] that will make sure the time is synched to a good enough source at boot. Chrony will then manage the time synchronization. I expect this to solve the problem for your application.
I have linked the PR to this thread so we will notify you once it is merged. It will then need to be released to the specific device types you use.
[1] systemd/timeinit: add HTTPS time synchronisation service by markcorbinuk · Pull Request #2360 · balena-os/meta-balena · GitHub
1 Like
Thanks for the info.
It is not the boot process alone however that causes the issue. A device can boot and then connect to the internet at a later stage. A manual way to trigger the sync when that happens would be helpful.
Indeed, but it should be automatic and not manual. This is being tracked in Burst time syncing on network connectivity loss · Issue #2314 · balena-os/meta-balena · GitHub and hopefully we will get there soon enough.
Perfect, thanks. Will monitor the tracking issue.