Download and install of images hangs

Pushing to a newly-provisioned device seems to hang indefinitely with the images part-downloaded / installed.

If I try to reboot through the dashboard I get:

This seems similar to the topic “Problems to Download Resin Images” but that did not seem to have been resolved. Is there anything obvious that can be done to force it to work?

I tried this from the host os but it just restarts installing etc. and gets stuck again.

systemctl stop resin-supervisor
systemctl stop balena
rm -rf /var/lib/docker/{aufs,containers,image,tmp}
systemctl start balena
update-resin-supervisor
systemctl start resin-supervisor

I’m only using an 8GB SD card, so wonder if this could be the problem, but there are no obvious errors reported about running out of space and doing df -h through the host OS shows 5.6GB+ free.

This could be a problem related to the device network, like limited bandwidth.

Do you see any error logs ? If you have hostOS access you can also try running journalctl -f in case this reveals any insights.

See below; there seem to be a couple of errors relating to the supervisor.

– Logs begin at Sun 2018-08-05 16:54:58 UTC. –
Aug 06 08:12:39 e556370 kernel[651]: [235780.808425] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
Aug 06 08:12:39 e556370 kernel[651]: [235780.817408] brcmfmac: power management disabled
Aug 06 08:12:39 e556370 NetworkManager[680]: [1533543159.8195] device (wlan0): supplicant interface state: inactive -> disabled
Aug 06 08:12:39 e556370 NetworkManager[680]: [1533543159.8216] device (wlan0): supplicant interface state: disabled -> inactive
Aug 06 08:12:39 e556370 wpa_supplicant[770]: wlan0: Reject scan trigger since one is already pending
Aug 06 08:16:36 e556370 systemd[1]: Started SSH Per-Connection Server (52.4.252.97:60784).
Aug 06 08:16:36 e556370 dropbear[13989]: Failed loading /etc/dropbear/dropbear_dss_host_key
Aug 06 08:16:36 e556370 dropbear[13989]: Failed loading /etc/dropbear/dropbear_ecdsa_host_key
Aug 06 08:16:36 e556370 dropbear[13989]: Child connection from ::ffff:52.4.252.97:60784
Aug 06 08:16:37 e556370 dropbear[13989]: Auth succeeded with blank password for ‘root’ from ::ffff:52.4.252.97:60784
Aug 06 08:17:16 e556370 systemd[1]: resin-supervisor.service: Watchdog timeout (limit 1min)!
Aug 06 08:17:16 e556370 systemd[1]: resin-supervisor.service: Killing process 12861 (start-resin-sup) with signal SIGABRT.
Aug 06 08:17:16 e556370 systemd[1]: resin-supervisor.service: Killing process 12862 (exe) with signal SIGABRT.
Aug 06 08:17:16 e556370 systemd[1]: resin-supervisor.service: Killing process 12890 (balena) with signal SIGABRT.
Aug 06 08:17:16 e556370 systemd[1]: resin-supervisor.service: Main process exited, code=dumped, status=6/ABRT

Looks like it’s failing on parsing environment variables - it may be the update to handle multiline ones or that mine aren’t all caps - will change this and try again.

Aug 06 08:19:16 e556370 balenad[723]: [2018-08-06T08:19:16.544Z] Unhandled rejection TypeError: Cannotread property ‘0’ of null
Aug 06 08:19:16 e556370 balenad[723]: at /usr/src/app/dist/app.js:310:69723
Aug 06 08:19:16 e556370 balenad[723]: at Qt (/usr/src/app/dist/app.js:9:5482)
Aug 06 08:19:16 e556370 balenad[723]: at Function.dr.filter (/usr/src/app/dist/app.js:9:50861)
Aug 06 08:19:16 e556370 balenad[723]: at p (/usr/src/app/dist/app.js:9:67155)
Aug 06 08:19:16 e556370 balenad[723]: at Fa (/usr/src/app/dist/app.js:9:39713)
Aug 06 08:19:16 e556370 balenad[723]: at /usr/src/app/dist/app.js:9:25590
Aug 06 08:19:16 e556370 balenad[723]: at en (/usr/src/app/dist/app.js:9:5919)
Aug 06 08:19:16 e556370 balenad[723]: at Ui (/usr/src/app/dist/app.js:9:25557)
Aug 06 08:19:16 e556370 balenad[723]: at yr.value (/usr/src/app/dist/app.js:9:68864)
Aug 06 08:19:16 e556370 balenad[723]: at Ui (/usr/src/app/dist/app.js:9:25548)
Aug 06 08:19:16 e556370 balenad[723]: at vr.dr.toJSON.dr.valueOf.dr.value (/usr/src/app/dist/app.js:9:69887)
Aug 06 08:19:16 e556370 balenad[723]: at Object.t.envArrayToObject (/usr/src/app/dist/app.js:310:69752)
Aug 06 08:19:16 e556370 balenad[723]: at e.exports.e.fromContainer (/usr/src/app/dist/app.js:310:83857)
Aug 06 08:19:16 e556370 balenad[723]: at u (/usr/src/app/dist/app.js:9:72666)
Aug 06 08:19:16 e556370 balenad[723]: at A._settlePromiseFromHandler (/usr/src/app/dist/app.js:310:122637)
Aug 06 08:19:16 e556370 balenad[723]: at A._settlePromise (/usr/src/app/dist/app.js:310:123437)

Hey @WillFG, supervisor maintainer here. Could you provide me with the dashboard url and enable support access please? I believe that this is because of an invalid env var, and later versions of the supervisor should in fact warn you of this.

I’ve granted support access to 12 hours:

https://dashboard.resin.io/devices/ead3ad5f6813690cff69898e92d788c8

Let me know if you need anything else.

I can try updating the supervisor in case that’s an old version (I’m using an image with 7.4.3).

I took a look at the device, and the logs do contain an error message, which says that one of your environment variables cannot be parsed as it contains newlines. If you remove these newlines, the supervisor should then be happy to continue.

Thanks Cameron, my understanding was that the system was updated to allow multi-line environment variables to now be used; I am guessing that while the dashboard now supports this and no longer strips newlines, the backend does not.

I’ll replace newlines with spaces for the moment as I worked around this previous dashboard behaviour in the past but would it be a good idea to revert the dashboard to prevent this?

@WillFG that’s exactly it, the frontend supports this, but the docker daemon itself (and by extension balena) does not support this.

A way to mitigate this is to base 64 encode your environment variable before insertion into the dashboard, and decode inside your user container.