Seeing some weird stuff on resinOS - I hope someone can help!
Sometimes, one of my device restarts (not sure why!) and it’s system time is totally off. E.g.
09.03.17 11:03:08 (-0800) Starting application 'registry.resin.io/...'
09.03.17 11:02:13 (-0800) Systemd init system enabled.
09.03.17 11:02:13 (-0800) systemd 215 running in system mode. (+PAM +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ -SECCOMP -APPARMOR)
09.03.17 11:02:13 (-0800) Detected virtualization 'other'.
09.03.17 11:02:13 (-0800) Detected architecture 'arm'.
09.03.17 11:02:13 (-0800) Set hostname to <010e05e-010e05e>.
// sometimes later
17.07.17 10:49:43 (-0700) Application is already running registry.resin.io/...'
Notice the date - 09-03-2017!
The system time is corrected sometime later, but we’ve built some logic around the datetime in our initialization script that gets tripped because of this. Any idea why this might be happening on few of our devices?
The HostOS version for this particular device is “Resin OS 2.0.0-rc1.rev2”
What device is that? It probably doesn’t have a battery, and the hardware clock resets itself if the board is powered down. Then, some time after boot, NTP sets the time over the network to a correct value.
Hey @michal and @shaunmulligan thanks for your replies! I’m running my apps in RPi 3 - I do wonder if this issue is on resinOS not the Pi based on @shaunmulligan’s comment. Do you guys have additional thoughts?
Hi @daniel.lee , as Shaun said, we believe this has been fixed starting with version v2.0.3.
Can you use the latest available version for your RPI3 and let us know if this issue is still persisting?
Hey @daniel.lee would probably suggest running the check through dbus-send (direct link in the docs: Checking if device time is NTP synchronized), either with dbus-send directly or with another language like Python or Node.js, or anything that dbus communication libraries.
Would not recommend running other ntp clients inside the application, having multiple ntp services trying to modify time potentially may cause all kinds of issues.
What resinOS version are you running now on your devices?
@imrehg Most of our devices are running resin OS 2.0.0.
We are using the dbus-send command to check whether NTP synchronization has completed or not. Usually, NTP sync within reasonable amount of time, thought it would be cool if we can FORCE the sync (I haven’t figured out how this can be done ).
We are working on resinOS 2.x device updates, and we have experimental tools to do that (e.g. potentially update your device from 2.0.0 to a newer version, likely the latest one, which has these fixes). Would you be interested maybe trying it out on some of your devices?
I’ve checked briefly, as a workaround might be able to set the time directly from the container, and be close enough for your use case so time syncing can happen on its own pace. E.g. using the trick of querying a public server, and getting the time from the HTTP response:
date -s "`curl -v --silent http://google.com/ 2>&1 \ | grep Date | sed -e 's/< Date: //'`"
This is obviously a hacky workaround and it’s not something I’d recommend, the best thing is using the newer version of resinOS, if that’s possible for you…
Sure thing, we love coming up with ways to help you get your work done.
Regarding the resin host OS updates - under these circumstances I would be hesitant to suggest to do that at this point if the network traffic amount is really sensitive. The update would be about ~150MB data transferred at the moment. We are working on reducing that, and apply delta updates to the host OS as well (ie. only transferring the changes), but it’s not there yet.
Regarding the 2. point, updating resinOS 2.x is should be pretty robust, it was designed with updates in mind, and it’s much simpler than how resinOS 1.x updates work at the moment. We are doing a writeup of what exactly happens when an update is run, so it’s more transparent and easier to make choices of update or not to update.
Hope this helps, and let us know what do you think! Will keep you posted about future host OS updates as well, that would suit your circumstances!
Confirmed, that time kept well on newer resinOS devices, just tried it on a 2.2.0+rev1 Raspberry Pi 3, and the host OS logs show this after a reboot:
Jul 28 05:23:00 raspberrypi3 systemd-timesyncd[560]: System clock time unset or jumped backwards, restoring from recorded timestamp: Fri 2017-08-04 10:16:53 UTC
Aug 04 10:16:53 raspberrypi3 systemd[1]: Time has been changed
Aug 04 10:16:53 raspberrypi3 systemd[1]: Started Network Time Synchronization.
And as mentioned, this is available since resinOS 2.0.3. If you have any devices to update to newer version, we are beta-testing updates, see more in the announcement
Does anyone has a simpler method (like a Python script) that returns 1 or 0 depending if the NTP synchronization has completed? (“NTPSynchronized” parameter)?
The repo by default runs an sh script that sets a var and then runs the python script but you can make your code invoke the NTPQuery().get_ntp_syncronized() method directly.
).
