Container reinstalled on device reboot?

Product support
1

We are seeing that our application container is reinstalled after a device is rebooted and comes back online (see log output below). This isn’t necessarily a problem for our use case but we have multiple resin applications and do not see this behavior in all of them (2 reinstall the container on reboot, 2 do not), so we’re just trying to get a better understanding of this. I also don’t recall seeing this behavior until a couple days ago. Is this a configurable option?

Log Output after reboot:

17.10.18 13:21:39 (-0500) Service exited 'main sha256:50777aa0bc83a946a17fbcca983574143620fd52cdb118012850346bee6238a3'
17.10.18 13:21:39 (-0500) Killed service 'main sha256:50777aa0bc83a946a17fbcca983574143620fd52cdb118012850346bee6238a3'
17.10.18 13:21:40 (-0500) Installing service 'main sha256:50777aa0bc83a946a17fbcca983574143620fd52cdb118012850346bee6238a3'
17.10.18 13:21:40 (-0500) Installed service 'main sha256:50777aa0bc83a946a17fbcca983574143620fd52cdb118012850346bee6238a3'
17.10.18 13:21:40 (-0500) Starting service 'main sha256:50777aa0bc83a946a17fbcca983574143620fd52cdb118012850346bee6238a3'
17.10.18 13:21:42 (-0500) Started service 'main sha256:50777aa0bc83a946a17fbcca983574143620fd52cdb118012850346bee6238a3'
2

hi @againes_rr that would be good to have more context, was the device just rebooted? Or is it after a host OS update or similar?

Don’t think this should normally happen, normally we see the supervisor reporting back that “Application already running”, but will check with our team.

5

Yes this happens after just rebooting the device (via reboot command or physically power cycling it).

6

Any update on this? Still seeing this behavior in our applications. Below are some lines I pulled from journalctl on the host OS. Let me know if there is any other information I can provide or troubleshooting I can do to assist with this.

Oct 31 15:18:25 64fb500971d2 balenad[839]: [60B blob data]
Oct 31 15:18:25 64fb500971d2 balenad[839]: [60B blob data]
Oct 31 15:18:25 64fb500971d2 balenad[839]: [60B blob data]
Oct 31 15:18:26 64fb500971d2 ModemManager[632]: Couldn't check support for device '/sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.1': not supported by any plugin
Oct 31 15:18:26 64fb500971d2 balenad[839]: [60B blob data]
Oct 31 15:18:26 64fb500971d2 systemd-udevd[1603]: Process '/usr/sbin/crda' failed with exit code 234.
Oct 31 15:18:27 64fb500971d2 balenad[839]: [60B blob data]
Oct 31 15:18:27 64fb500971d2 ModemManager[632]: (net/wlan0): released by device '/sys/devices/platform/soc/3f300000.mmc/mmc_host/mmc1/mmc1:0001/mmc1:0001:1'
Oct 31 15:18:27 64fb500971d2 ModemManager[632]: Couldn't check support for device '/sys/devices/platform/soc/3f300000.mmc/mmc_host/mmc1/mmc1:0001/mmc1:0001:1': Operation was cancelled
Oct 31 15:18:27 64fb500971d2 ModemManager[632]: (net/eth0): released by device '/sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.1'
Oct 31 15:18:27 64fb500971d2 ModemManager[632]: Couldn't check support for device '/sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.1': Operation was cancelled
Oct 31 15:18:31 64fb500971d2 healthdog[1155]: [2018-10-31T15:18:31.372Z] Event: Supervisor start {}
Oct 31 15:18:31 64fb500971d2 balenad[839]: [2018-10-31T15:18:31.372Z] Event: Supervisor start {}
Oct 31 15:18:31 64fb500971d2 balenad[839]: [2018-10-31T15:18:31.819Z] Event: Service already running {"service":{"appId":712935,"serviceId":7806,"serviceName":"main","releaseId":651424,"imag>
Oct 31 15:18:31 64fb500971d2 healthdog[1155]: [2018-10-31T15:18:31.819Z] Event: Service already running {"service":{"appId":712935,"serviceId":7806,"serviceName":"main","releaseId":651424,"i>
Oct 31 15:18:31 64fb500971d2 balenad[839]: [2018-10-31T15:18:31.888Z] Connectivity check enabled: true
Oct 31 15:18:31 64fb500971d2 healthdog[1155]: [2018-10-31T15:18:31.888Z] Connectivity check enabled: true
Oct 31 15:18:31 64fb500971d2 balenad[839]: [2018-10-31T15:18:31.900Z] Starting periodic check for IP addresses
Oct 31 15:18:31 64fb500971d2 healthdog[1155]: [2018-10-31T15:18:31.900Z] Starting periodic check for IP addresses
Oct 31 15:18:31 64fb500971d2 balenad[839]: [2018-10-31T15:18:31.921Z] Reporting initial state, supervisor version and API info
Oct 31 15:18:31 64fb500971d2 healthdog[1155]: [2018-10-31T15:18:31.921Z] Reporting initial state, supervisor version and API info
Oct 31 15:18:31 64fb500971d2 balenad[839]: [2018-10-31T15:18:31.926Z] Starting API server
Oct 31 15:18:31 64fb500971d2 healthdog[1155]: [2018-10-31T15:18:31.926Z] Starting API server
Oct 31 15:18:31 64fb500971d2 balenad[839]: [2018-10-31T15:18:31.970Z] Ensuring device is provisioned
Oct 31 15:18:31 64fb500971d2 healthdog[1155]: [2018-10-31T15:18:31.970Z] Ensuring device is provisioned
Oct 31 15:18:31 64fb500971d2 balenad[839]: [2018-10-31T15:18:31.997Z] Applying target state
Oct 31 15:18:31 64fb500971d2 healthdog[1155]: [2018-10-31T15:18:31.997Z] Applying target state
Oct 31 15:18:32 64fb500971d2 balenad[839]: [2018-10-31T15:18:32.030Z] VPN status path exists.
Oct 31 15:18:32 64fb500971d2 healthdog[1155]: [2018-10-31T15:18:32.030Z] VPN status path exists.
Oct 31 15:18:32 64fb500971d2 balenad[839]: [2018-10-31T15:18:32.162Z] Internet Connectivity: OK
Oct 31 15:18:32 64fb500971d2 healthdog[1155]: [2018-10-31T15:18:32.162Z] Internet Connectivity: OK
Oct 31 15:18:32 64fb500971d2 balenad[839]: [2018-10-31T15:18:32.172Z] Event: Device bootstrap {}
Oct 31 15:18:32 64fb500971d2 healthdog[1155]: [2018-10-31T15:18:32.172Z] Event: Device bootstrap {}
Oct 31 15:18:32 64fb500971d2 balenad[839]: [2018-10-31T15:18:32.870Z] Starting current state report
Oct 31 15:18:32 64fb500971d2 healthdog[1155]: [2018-10-31T15:18:32.887Z] Starting target state poll
Oct 31 15:18:32 64fb500971d2 balenad[839]: [2018-10-31T15:18:32.887Z] Starting target state poll
Oct 31 15:18:33 64fb500971d2 healthdog[1155]: [2018-10-31T15:18:33.545Z] Event: Service kill {"service":{"appId":712935,"serviceId":7806,"serviceName":"main","releaseId":651424,"image":"sha256:f6fd663490e2375b008035816d47c439a1ebcf00d6a49e6d2cbee24ade2e90ec","labels":{"io.resin.app-id":"712935","io.resin.architecture":"armv7hf","io.resin.device-type":"raspberry-pi2","io.resin.features.dbus":"1","io.resin.features.firmware":"1","io.resin.features.kernel-modules":"1","io.resin.features.resin-api":"1","io.resin.features.supervisor-api":"1","io.resin.qemu.version":"3.0.0+resin-arm","io.resin.service-id":"7806","io.resin.service-name":"main","io.resin.supervised":"true"}}}
Oct 31 15:18:33 64fb500971d2 balenad[839]: [2018-10-31T15:18:33.545Z] Event: Service kill {"service":{"appId":712935,"serviceId":7806,"serviceName":"main","releaseId":651424,"image":"sha256:f6fd663490e2375b008035816d47c439a1ebcf00d6a49e6d2cbee24ade2e90ec","labels":{"io.resin.app-id":"712935","io.resin.architecture":"armv7hf","io.resin.device-type":"raspberry-pi2","io.resin.features.dbus":"1","io.resin.features.firmware":"1","io.resin.features.kernel-modules":"1","io.resin.features.resin-api":"1","io.resin.features.supervisor-api":"1","io.resin.qemu.version":"3.0.0+resin-arm","io.resin.service-id":"7806","io.resin.service-name":"main","io.resin.supervised":"true"}}}
Oct 31 15:18:35 64fb500971d2 healthdog[839]: time="2018-10-31T15:18:35.694395267Z" level=info msg="Container failed to stop after sending signal 37 to the process, force killing"
Oct 31 15:18:36 64fb500971d2 healthdog[839]: time="2018-10-31T15:18:36.203087621Z" level=warning msg="unknown container" container=befeb89a1dcd9e3a9cd8b1e14362c7a0dc301733d9f14f2109e9c9bc549ccde6 module=libcontainerd namespace=plugins.moby
Oct 31 15:18:36 64fb500971d2 healthdog[839]: time="2018-10-31T15:18:36Z" level=info msg="shim reaped" id=befeb89a1dcd9e3a9cd8b1e14362c7a0dc301733d9f14f2109e9c9bc549ccde6 module=containerd/tasks
Oct 31 15:18:36 64fb500971d2 healthdog[839]: time="2018-10-31T15:18:36.544532083Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Oct 31 15:18:36 64fb500971d2 healthdog[839]: time="2018-10-31T15:18:36.546588030Z" level=info msg="ignoring event" module=libcontainerd namespace=plugins.moby topic=/tasks/delete type="*events.TaskDelete"
Oct 31 15:18:36 64fb500971d2 healthdog[839]: time="2018-10-31T15:18:36.770217115Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/containers/delete type="*events.ContainerDelete"
Oct 31 15:18:36 64fb500971d2 balenad[839]: [2018-10-31T15:18:36.824Z] Event: Service exit {"service":{"appId":712935,"serviceId":7806,"serviceName":"main","releaseId":651424,"image":"sha256:f6fd663490e2375b008035816d47c439a1ebcf00d6a49e6d2cbee24ade2e90ec","labels":{"io.resin.app-id":"712935","io.resin.architecture":"armv7hf","io.resin.device-type":"raspberry-pi2","io.resin.features.dbus":"1","io.resin.features.firmware":"1","io.resin.features.kernel-modules":"1","io.resin.features.resin-api":"1","io.resin.features.supervisor-api":"1","io.resin.qemu.version":"3.0.0+resin-arm","io.resin.service-id":"7806","io.resin.service-name":"main","io.resin.supervised":"true"}}}
Oct 31 15:18:36 64fb500971d2 healthdog[1155]: [2018-10-31T15:18:36.824Z] Event: Service exit {"service":{"appId":712935,"serviceId":7806,"serviceName":"main","releaseId":651424,"image":"sha256:f6fd663490e2375b008035816d47c439a1ebcf00d6a49e6d2cbee24ade2e90ec","labels":{"io.resin.app-id":"712935","io.resin.architecture":"armv7hf","io.resin.device-type":"raspberry-pi2","io.resin.features.dbus":"1","io.resin.features.firmware":"1","io.resin.features.kernel-modules":"1","io.resin.features.resin-api":"1","io.resin.features.supervisor-api":"1","io.resin.qemu.version":"3.0.0+resin-arm","io.resin.service-id":"7806","io.resin.service-name":"main","io.resin.supervised":"true"}}}
Oct 31 15:18:37 64fb500971d2 healthdog[1155]: [2018-10-31T15:18:37.160Z] Event: Service stop {"service":{"appId":712935,"serviceId":7806,"serviceName":"main","releaseId":651424,"image":"sha256:f6fd663490e2375b008035816d47c439a1ebcf00d6a49e6d2cbee24ade2e90ec","labels":{"io.resin.app-id":"712935","io.resin.architecture":"armv7hf","io.resin.device-type":"raspberry-pi2","io.resin.features.dbus":"1","io.resin.features.firmware":"1","io.resin.features.kernel-modules":"1","io.resin.features.resin-api":"1","io.resin.features.supervisor-api":"1","io.resin.qemu.version":"3.0.0+resin-arm","io.resin.service-id":"7806","io.resin.service-name":"main","io.resin.supervised":"true"}}}
Oct 31 15:18:37 64fb500971d2 balenad[839]: [2018-10-31T15:18:37.160Z] Event: Service stop {"service":{"appId":712935,"serviceId":7806,"serviceName":"main","releaseId":651424,"image":"sha256:f6fd663490e2375b008035816d47c439a1ebcf00d6a49e6d2cbee24ade2e90ec","labels":{"io.resin.app-id":"712935","io.resin.architecture":"armv7hf","io.resin.device-type":"raspberry-pi2","io.resin.features.dbus":"1","io.resin.features.firmware":"1","io.resin.features.kernel-modules":"1","io.resin.features.resin-api":"1","io.resin.features.supervisor-api":"1","io.resin.qemu.version":"3.0.0+resin-arm","io.resin.service-id":"7806","io.resin.service-name":"main","io.resin.supervised":"true"}}}
Oct 31 15:18:38 64fb500971d2 balenad[839]: [2018-10-31T15:18:38.499Z] Event: Service install {"service":{"appId":712935,"serviceId":7806,"serviceName":"main","releaseId":651424,"image":"sha256:f6fd663490e2375b008035816d47c439a1ebcf00d6a49e6d2cbee24ade2e90ec","labels":{"io.resin.features.dbus":"1","io.resin.features.firmware":"1","io.resin.features.kernel-modules":"1","io.resin.features.resin-api":"1","io.resin.features.supervisor-api":"1","io.resin.architecture":"armv7hf","io.resin.device-type":"raspberry-pi2","io.resin.qemu.version":"3.0.0+resin-arm","io.resin.supervised":"true","io.resin.app-id":"712935","io.resin.service-id":"7806","io.resin.service-name":"main"}}}
Oct 31 15:18:38 64fb500971d2 healthdog[1155]: [2018-10-31T15:18:38.499Z] Event: Service install {"service":{"appId":712935,"serviceId":7806,"serviceName":"main","releaseId":651424,"image":"sha256:f6fd663490e2375b008035816d47c439a1ebcf00d6a49e6d2cbee24ade2e90ec","labels":{"io.resin.features.dbus":"1","io.resin.features.firmware":"1","io.resin.features.kernel-modules":"1","io.resin.features.resin-api":"1","io.resin.features.supervisor-api":"1","io.resin.architecture":"armv7hf","io.resin.device-type":"raspberry-pi2","io.resin.qemu.version":"3.0.0+resin-arm","io.resin.supervised":"true","io.resin.app-id":"712935","io.resin.service-id":"7806","io.resin.service-name":"main"}}}
Oct 31 15:18:39 64fb500971d2 balenad[839]: [2018-10-31T15:18:39.072Z] Event: Service installed {"service":{"appId":712935,"serviceId":7806,"serviceName":"main","releaseId":651424,"image":"sha256:f6fd663490e2375b008035816d47c439a1ebcf00d6a49e6d2cbee24ade2e90ec","labels":{"io.resin.features.dbus":"1","io.resin.features.firmware":"1","io.resin.features.kernel-modules":"1","io.resin.features.resin-api":"1","io.resin.features.supervisor-api":"1","io.resin.architecture":"armv7hf","io.resin.device-type":"raspberry-pi2","io.resin.qemu.version":"3.0.0+resin-arm","io.resin.supervised":"true","io.resin.app-id":"712935","io.resin.service-id":"7806","io.resin.service-name":"main"}}}
Oct 31 15:18:39 64fb500971d2 healthdog[1155]: [2018-10-31T15:18:39.072Z] Event: Service installed {"service":{"appId":712935,"serviceId":7806,"serviceName":"main","releaseId":651424,"image":"sha256:f6fd663490e2375b008035816d47c439a1ebcf00d6a49e6d2cbee24ade2e90ec","labels":{"io.resin.features.dbus":"1","io.resin.features.firmware":"1","io.resin.features.kernel-modules":"1","io.resin.features.resin-api":"1","io.resin.features.supervisor-api":"1","io.resin.architecture":"armv7hf","io.resin.device-type":"raspberry-pi2","io.resin.qemu.version":"3.0.0+resin-arm","io.resin.supervised":"true","io.resin.app-id":"712935","io.resin.service-id":"7806","io.resin.service-name":"main"}}}
Oct 31 15:18:39 64fb500971d2 balenad[839]: [2018-10-31T15:18:39.088Z] Event: Service start {"service":{"appId":712935,"serviceId":7806,"serviceName":"main","releaseId":651424,"image":"sha256:f6fd663490e2375b008035816d47c439a1ebcf00d6a49e6d2cbee24ade2e90ec","labels":{"io.resin.features.dbus":"1","io.resin.features.firmware":"1","io.resin.features.kernel-modules":"1","io.resin.features.resin-api":"1","io.resin.features.supervisor-api":"1","io.resin.architecture":"armv7hf","io.resin.device-type":"raspberry-pi2","io.resin.qemu.version":"3.0.0+resin-arm","io.resin.supervised":"true","io.resin.app-id":"712935","io.resin.service-id":"7806","io.resin.service-name":"main"}}}
Oct 31 15:18:39 64fb500971d2 healthdog[1155]: [2018-10-31T15:18:39.088Z] Event: Service start {"service":{"appId":712935,"serviceId":7806,"serviceName":"main","releaseId":651424,"image":"sha256:f6fd663490e2375b008035816d47c439a1ebcf00d6a49e6d2cbee24ade2e90ec","labels":{"io.resin.features.dbus":"1","io.resin.features.firmware":"1","io.resin.features.kernel-modules":"1","io.resin.features.resin-api":"1","io.resin.features.supervisor-api":"1","io.resin.architecture":"armv7hf","io.resin.device-type":"raspberry-pi2","io.resin.qemu.version":"3.0.0+resin-arm","io.resin.supervised":"true","io.resin.app-id":"712935","io.resin.service-id":"7806","io.resin.service-name":"main"}}}
Oct 31 15:18:39 64fb500971d2 healthdog[839]: time="2018-10-31T15:18:39.221037842Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/containers/create type="*events.ContainerCreate"
9

Hi,
This does look odd indeed, as if the update was pending and applied when the device was rebooted. Were no environment variables changed? What OS version are you using? Could you point us to the device’s uuid and grant support access for us to take a look?
Thanks!

11

Correct, no environment variables were changed, just a reboot of the device. It is running OS 2.24.0 rev 1 prod. Device UUID is 3fcf79bc5b5a92458fec94be38141d53 and I’ve granted access for a day. Feel free to reboot the device or do whatever you need.

12

Thanks @againes_rr - I’ve detected that the problem has to do with the hostname of the device changing during the reboot, so the container has to be restarted for the hostname to match the expected one. We’ll investigate how this happened (might be a race condition with how the hostname is set by the host), but just in case, does your application try to set a hostname or do anything else related to it?

17

Hi @againes_rr any thoughts on the above? It’d be useful to know if your application does anything with the device’s hostname, or anything else that might be related.

19

Apologies on the late reply here. Yes our team looked into it and it seems we were setting the hostname of the device at each boot. We have modified this behavior so the hostname is not set every boot. So far we are no longer seeing the container get reinstalled at boot so that appears to have fixed the issue. Thank you for the assistance!

1 Like
20

Brilliant, glad it got solved, cheers!