I’ve come across quite a fundamental thing which I presumed I’d understood, but obviously haven’t at all - container persistence between restarts. Every time our device boots, it compares a checksum of the result of the GET /v1/apps/:appId api call, with the checksum of the call at the previous boot. Basically if the application notices that the commit has changed, or we have new env vars, then there’s some stuff it has to do. Pretty straight forward.
What I’ve noticed (by accident, because we initially included it in the checksum), is that the containerID is not always changing between reboots of the device. It does, however always change between restarts of the application.
Can someone explain the expected behaviour to me? Is this possibly a by-product of our docker-in-docker setup, which does a mount -o bind /data /var/lib/docker at app startup? Or is this actually how the supervisor always works?
When you restart the device, the application container is stopped but not removed; this enables the application to come up right after reboot (when docker restarts it just restarts the application container too), which is an important thing for most use cases. Also, usually it happens even before the supervisor starts, for the user application needs to be able to run, whatever might be up with the supervisor.
When you restart the application, the resin-supervisor stops and removes the application container, and starts up a new one, hence the container ID change.
Yes, that’s pretty much what I was expecting - thanks!
Is there any standard setting to force a new container start on device restart each time? For our use-case, this is actually not ideal, as we have a bunch of network bridges, and iptables rules which sometimes get screwed up when the container is just restarted, instead of stopped started.
Well, I guess you could call the supervisor to restart the application if you see that the container id is the same as on the last application start. Just have to make sure that you have sufficient safeguards (when the supervisor is not available, or other corner cases).
Regarding the iptables rules, likely it would be necessary to have some clean start rules (where you clean up rules on start or reapply them to always start from a known state, as much as possible). That would be my best guess.
Also, just to note, on resinOS 2.3.0 the docker0 interface has a fixed IP (as opposed to automatic assignment as before), which might or might not interfere with your iptables rules, would highly recommend checking it out and making sure that both 2.3.0 and your deployed resinOS versions work too, for futureproofness. See the changelog.
We see a lot of strange support requests which ultimately come back to not-quite-perfect iptables assignments by the user container.
