Raspberry Pi 4 - Keeps restarting upload

walpoletim · October 25, 2019, 6:59am

Hi

Device has just been rebooted…

Tim

walpoletim · October 26, 2019, 6:50am

Hi guys

Any progress on the issues with the Pi Firmware - I’m keen to get my containers deployed so I can start doing some testing…

BTW - I would like to do a blog article on privacy first AI solutions delivered using Balena on Pi 4’s once this is up and running if you are interested in this (Will be doing it anyway, but thought you could be interested)

Thanks

Tim

shaunmulligan · October 27, 2019, 3:38pm

Hi @walpoletim I don’t think @wrboyce has managed to get much further into the investigation. Do you have a simple reproduction / or repo we could push to our devices to debug this faster. i’ve been using rpi4 for a while with multiple containers and haven’t hit anything like this, so will be interesting to see what is going on

shaunmulligan · October 27, 2019, 3:39pm

Also a blog article on privacy first AI will be very interesting to us indeed

walpoletim · October 27, 2019, 4:22pm

I will see if I can put together a set of containers this week which do not have any private code so you can investigate…

Be great to get to the bottom of what’s going on…

Tim

shaunmulligan · October 27, 2019, 4:24pm

Awesome, thanks Tim that will be very useful.

walpoletim · October 28, 2019, 3:34pm

Hi

I have discovered something that may be of use….

Deploy Containers [Mongo] -> Worked Correctly
Add a new container [Mongo, API (Koa)] -> Worked Correctly
Add a new container [Mongo, API (Koa), UI (Vue)] -> Worked Correctly
Updated UI -> Constantly Failed to Download (Kept failing at low percentages (2%, 5% etc)
Removed UI Container -> PI Updated Correctly
Added UI Container Again -> Downloaded much better - only restarted once approx 70%

Seems like having the container running is causing the download to fail…?

Let me know if you want me to experiment more…

Tim

imrehg · October 28, 2019, 6:15pm

Yes, that is possible. We have a number of watchdog services to try to keep a device running, for example restarting balenaEngine if it’s not responding for a while (in case it is stuck). But when there are these downloads, maybe the engine is slow to respond, and gets restarted when it shouldn’t have been. This can happen either due to slow SD card or busy CPU, in our experience, but not totally clear what resource utilization would trigger it.

What would be useful, is to see whether the supervisor is restarted around that time. For example looking at the logs, when the download fails, by logging into the host OS, and running systemctl logs -f -a -u resin-supervisor -u balena This would show logs from the supervisor service and the balena engine service. I would suspect one or the other would show a “watchdog timeout” or similar error and a restart.
How’s this?

walpoletim · October 28, 2019, 6:38pm

I can check that for you. I do know then when the pi is downloading images and you ssh onto the pi then normal commands lock most of the time for 20+ seconds at a time so something is blocking the user kernel …

walpoletim · October 28, 2019, 7:49pm

Hi

Here you go

Does this mean anything…??

-- Logs begin at Mon 2019-10-28 16:54:52 UTC. --
Oct 28 19:38:42 a01bdc6 resin-supervisor[7885]: [info]    Applying target state
Oct 28 19:38:44 a01bdc6 f07fbb3a7150[6981]: [event]   Event: Docker image download {"image":{"name":"registry2.balena-cloud.com/v2/7c779e6a95b10ba5e30a93f7bdafc283@sha256:641b1378b20735bde688fc03fd2553b4ec577f68a20412317f7fb4bd29abb820","appId":1521922,"serviceId":335393,"serviceName":"runtime","imageId":1631937,"releaseId":1125521,"dependent":0,"dockerImageId":null}}
Oct 28 19:38:44 a01bdc6 resin-supervisor[7885]: [event]   Event: Docker image download {"image":{"name":"registry2.balena-cloud.com/v2/7c779e6a95b10ba5e30a93f7bdafc283@sha256:641b1378b20735bde688fc03fd2553b4ec577f68a20412317f7fb4bd29abb820","appId":1521922,"serviceId":335393,"serviceName":"runtime","imageId":1631937,"releaseId":1125521,"dependent":0,"dockerImageId":null}}
Oct 28 19:39:28 a01bdc6 balenad[6981]: time="2019-10-28T19:39:28.935273105Z" level=info msg="shim balena-engine-containerd-shim started" address=/containerd-shim/moby/3455b422d21a66d4115d9befef8188b8bb18a107af0578d8a046de92fdcf2cfc/shim.sock debug=false pid=14478
Oct 28 19:39:30 a01bdc6 balenad[6981]: time="2019-10-28T19:39:30.199088156Z" level=info msg="shim reaped" id=3455b422d21a66d4115d9befef8188b8bb18a107af0578d8a046de92fdcf2cfc
Oct 28 19:39:30 a01bdc6 balenad[6981]: time="2019-10-28T19:39:30.202353531Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Oct 28 19:39:35 a01bdc6 59d5f899deab[6981]: 2019/10/28 19:39:35 [error] 9#9: *21 open() "/usr/share/nginx/html/bots" failed (2: No such file or directory), client: 52.4.252.97, server: localhost, request: "GET /bots HTTP/1.1", host: "a01bdc618bba3602677c8119e31d8002.balena-devices.com", referrer: "https://a01bdc618bba3602677c8119e31d8002.balena-devices.com/"
Oct 28 19:39:35 a01bdc6 59d5f899deab[6981]: 52.4.252.97 - - [28/Oct/2019:19:39:35 +0000] "GET /bots HTTP/1.1" 404 153 "https://a01bdc618bba3602677c8119e31d8002.balena-devices.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15" "-"
Oct 28 19:40:12 a01bdc6 59d5f899deab[6981]: 2019/10/28 19:40:12 [error] 9#9: *21 open() "/usr/share/nginx/html/bots" failed (2: No such file or directory), client: 52.4.252.97, server: localhost, request: "GET /bots HTTP/1.1", host: "a01bdc618bba3602677c8119e31d8002.balena-devices.com", referrer: "https://a01bdc618bba3602677c8119e31d8002.balena-devices.com/"
Oct 28 19:40:12 a01bdc6 59d5f899deab[6981]: 52.4.252.97 - - [28/Oct/2019:19:40:12 +0000] "GET /bots HTTP/1.1" 404 153 "https://a01bdc618bba3602677c8119e31d8002.balena-devices.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15" "-"
^[[A^C
root@a01bdc6:~# journalctl -f -a -u resin-supervisor -u balena
-- Logs begin at Mon 2019-10-28 16:54:52 UTC. --
Oct 28 19:38:42 a01bdc6 resin-supervisor[7885]: [info]    Applying target state
Oct 28 19:38:44 a01bdc6 f07fbb3a7150[6981]: [event]   Event: Docker image download {"image":{"name":"registry2.balena-cloud.com/v2/7c779e6a95b10ba5e30a93f7bdafc283@sha256:641b1378b20735bde688fc03fd2553b4ec577f68a20412317f7fb4bd29abb820","appId":1521922,"serviceId":335393,"serviceName":"runtime","imageId":1631937,"releaseId":1125521,"dependent":0,"dockerImageId":null}}
Oct 28 19:38:44 a01bdc6 resin-supervisor[7885]: [event]   Event: Docker image download {"image":{"name":"registry2.balena-cloud.com/v2/7c779e6a95b10ba5e30a93f7bdafc283@sha256:641b1378b20735bde688fc03fd2553b4ec577f68a20412317f7fb4bd29abb820","appId":1521922,"serviceId":335393,"serviceName":"runtime","imageId":1631937,"releaseId":1125521,"dependent":0,"dockerImageId":null}}
Oct 28 19:39:28 a01bdc6 balenad[6981]: time="2019-10-28T19:39:28.935273105Z" level=info msg="shim balena-engine-containerd-shim started" address=/containerd-shim/moby/3455b422d21a66d4115d9befef8188b8bb18a107af0578d8a046de92fdcf2cfc/shim.sock debug=false pid=14478
Oct 28 19:39:30 a01bdc6 balenad[6981]: time="2019-10-28T19:39:30.199088156Z" level=info msg="shim reaped" id=3455b422d21a66d4115d9befef8188b8bb18a107af0578d8a046de92fdcf2cfc
Oct 28 19:39:30 a01bdc6 balenad[6981]: time="2019-10-28T19:39:30.202353531Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Oct 28 19:39:35 a01bdc6 59d5f899deab[6981]: 2019/10/28 19:39:35 [error] 9#9: *21 open() "/usr/share/nginx/html/bots" failed (2: No such file or directory), client: 52.4.252.97, server: localhost, request: "GET /bots HTTP/1.1", host: "a01bdc618bba3602677c8119e31d8002.balena-devices.com", referrer: "https://a01bdc618bba3602677c8119e31d8002.balena-devices.com/"
Oct 28 19:39:35 a01bdc6 59d5f899deab[6981]: 52.4.252.97 - - [28/Oct/2019:19:39:35 +0000] "GET /bots HTTP/1.1" 404 153 "https://a01bdc618bba3602677c8119e31d8002.balena-devices.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15" "-"
Oct 28 19:40:12 a01bdc6 59d5f899deab[6981]: 2019/10/28 19:40:12 [error] 9#9: *21 open() "/usr/share/nginx/html/bots" failed (2: No such file or directory), client: 52.4.252.97, server: localhost, request: "GET /bots HTTP/1.1", host: "a01bdc618bba3602677c8119e31d8002.balena-devices.com", referrer: "https://a01bdc618bba3602677c8119e31d8002.balena-devices.com/"
Oct 28 19:40:12 a01bdc6 59d5f899deab[6981]: 52.4.252.97 - - [28/Oct/2019:19:40:12 +0000] "GET /bots HTTP/1.1" 404 153 "https://a01bdc618bba3602677c8119e31d8002.balena-devices.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15" "-"
Oct 28 19:42:40 a01bdc6 f07fbb3a7150[6981]: [api]     GET /v1/healthy 200 - 9.878 ms
Oct 28 19:42:40 a01bdc6 resin-supervisor[7885]: [api]     GET /v1/healthy 200 - 9.878 ms
Oct 28 19:43:34 a01bdc6 resin-supervisor[7885]: [error]   Non-200 response from the API! Status code: 503 - message: t: The API cannot handle your request right now
Oct 28 19:43:34 a01bdc6 f07fbb3a7150[6981]: [error]   Non-200 response from the API! Status code: 503 - message: t: The API cannot handle your request right now
Oct 28 19:43:34 a01bdc6 resin-supervisor[7885]: [error]         at /usr/src/app/dist/app.js:465:5539
Oct 28 19:43:34 a01bdc6 resin-supervisor[7885]: [error]       at c (/usr/src/app/dist/app.js:9:75680)
Oct 28 19:43:34 a01bdc6 resin-supervisor[7885]: [error]       at A._settlePromiseFromHandler (/usr/src/app/dist/app.js:293:133124)
Oct 28 19:43:34 a01bdc6 f07fbb3a7150[6981]: [error]         at /usr/src/app/dist/app.js:465:5539
Oct 28 19:43:34 a01bdc6 f07fbb3a7150[6981]: [error]       at c (/usr/src/app/dist/app.js:9:75680)
Oct 28 19:43:34 a01bdc6 f07fbb3a7150[6981]: [error]       at A._settlePromiseFromHandler (/usr/src/app/dist/app.js:293:133124)
Oct 28 19:43:34 a01bdc6 f07fbb3a7150[6981]: [error]       at A._settlePromise (/usr/src/app/dist/app.js:293:133924)
Oct 28 19:43:34 a01bdc6 f07fbb3a7150[6981]: [error]       at A._settlePromise0 (/usr/src/app/dist/app.js:293:134623)
Oct 28 19:43:34 a01bdc6 resin-supervisor[7885]: [error]       at A._settlePromise (/usr/src/app/dist/app.js:293:133924)
Oct 28 19:43:34 a01bdc6 resin-supervisor[7885]: [error]       at A._settlePromise0 (/usr/src/app/dist/app.js:293:134623)
Oct 28 19:43:34 a01bdc6 resin-supervisor[7885]: [error]       at A._settlePromises (/usr/src/app/dist/app.js:293:135975)
Oct 28 19:43:34 a01bdc6 resin-supervisor[7885]: [error]       at A._fulfill (/usr/src/app/dist/app.js:293:134993)
Oct 28 19:43:34 a01bdc6 resin-supervisor[7885]: [error]       at U._callback (/usr/src/app/dist/app.js:103:92989)
Oct 28 19:43:34 a01bdc6 resin-supervisor[7885]: [error]       at U.t._callback.t.callback.t.callback (/usr/src/app/dist/app.js:525:5327)
Oct 28 19:43:34 a01bdc6 resin-supervisor[7885]: [error]       at U.emit (events.js:189:13)
Oct 28 19:43:34 a01bdc6 resin-supervisor[7885]: [error]       at U.<anonymous> (/usr/src/app/dist/app.js:525:18140)
Oct 28 19:43:34 a01bdc6 resin-supervisor[7885]: [error]       at U.emit (events.js:189:13)
Oct 28 19:43:34 a01bdc6 resin-supervisor[7885]: [error]       at IncomingMessage.<anonymous> (/usr/src/app/dist/app.js:525:16982)
Oct 28 19:43:34 a01bdc6 resin-supervisor[7885]: [error]       at Object.onceWrapper (events.js:277:13)
Oct 28 19:43:34 a01bdc6 resin-supervisor[7885]: [error]       at IncomingMessage.emit (events.js:194:15)
Oct 28 19:43:34 a01bdc6 resin-supervisor[7885]: [error]       at endReadableNT (_stream_readable.js:1125:12)
Oct 28 19:43:34 a01bdc6 resin-supervisor[7885]: [error]       at process._tickCallback (internal/process/next_tick.js:63:19)
Oct 28 19:43:34 a01bdc6 f07fbb3a7150[6981]: [error]       at A._settlePromises (/usr/src/app/dist/app.js:293:135975)
Oct 28 19:43:34 a01bdc6 f07fbb3a7150[6981]: [error]       at A._fulfill (/usr/src/app/dist/app.js:293:134993)
Oct 28 19:43:34 a01bdc6 f07fbb3a7150[6981]: [error]       at U._callback (/usr/src/app/dist/app.js:103:92989)
Oct 28 19:43:34 a01bdc6 f07fbb3a7150[6981]: [error]       at U.t._callback.t.callback.t.callback (/usr/src/app/dist/app.js:525:5327)
Oct 28 19:43:34 a01bdc6 f07fbb3a7150[6981]: [error]       at U.emit (events.js:189:13)
Oct 28 19:43:34 a01bdc6 f07fbb3a7150[6981]: [error]       at U.<anonymous> (/usr/src/app/dist/app.js:525:18140)
Oct 28 19:43:34 a01bdc6 f07fbb3a7150[6981]: [error]       at U.emit (events.js:189:13)
Oct 28 19:43:34 a01bdc6 f07fbb3a7150[6981]: [error]       at IncomingMessage.<anonymous> (/usr/src/app/dist/app.js:525:16982)
Oct 28 19:43:34 a01bdc6 f07fbb3a7150[6981]: [error]       at Object.onceWrapper (events.js:277:13)
Oct 28 19:43:34 a01bdc6 f07fbb3a7150[6981]: [error]       at IncomingMessage.emit (events.js:194:15)
Oct 28 19:43:34 a01bdc6 f07fbb3a7150[6981]: [error]       at endReadableNT (_stream_readable.js:1125:12)
Oct 28 19:43:34 a01bdc6 f07fbb3a7150[6981]: [error]       at process._tickCallback (internal/process/next_tick.js:63:19)
Oct 28 19:44:38 a01bdc6 balenad[6981]: time="2019-10-28T19:44:38.530510567Z" level=info msg="shim balena-engine-containerd-shim started" address=/containerd-shim/moby/acd7d54b5324f23b2fdcc9ba2f81ec0daf6cd3e96dea22c4ff6c02dc2e359473/shim.sock debug=false pid=14710
Oct 28 19:45:03 a01bdc6 balenad[6981]: time="2019-10-28T19:45:03.042409287Z" level=info msg="shim reaped" id=acd7d54b5324f23b2fdcc9ba2f81ec0daf6cd3e96dea22c4ff6c02dc2e359473
Oct 28 19:45:03 a01bdc6 balenad[6981]: time="2019-10-28T19:45:03.049824134Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Oct 28 19:46:43 a01bdc6 resin-supervisor[7885]: [debug]   Attempting container log timestamp flush...
Oct 28 19:46:43 a01bdc6 f07fbb3a7150[6981]: [debug]   Attempting container log timestamp flush...
Oct 28 19:46:43 a01bdc6 f07fbb3a7150[6981]: [debug]   Container log timestamp flush complete
Oct 28 19:46:43 a01bdc6 resin-supervisor[7885]: [debug]   Container log timestamp flush complete
Oct 28 19:47:53 a01bdc6 f07fbb3a7150[6981]: [api]     GET /v1/healthy 200 - 5.079 ms
Oct 28 19:47:53 a01bdc6 resin-supervisor[7885]: [api]     GET /v1/healthy 200 - 5.079 ms

walpoletim · October 28, 2019, 9:19pm

Hi - It definitely seems to be something with uploading a new version of a container when one is already there…

I have just downloaded my runtime (Koa) container, and NLP container (2.2g), with no download errors - as a first time container download - This never happens when its updating a container - it retries multiple times (sometimes for 24 hours) till it succeeds…

Tim

walpoletim · October 29, 2019, 7:51am

As expected, the issue started occurring when I updated existing images…

I have attached the device log file for you . The device is again in support mode if you want to take a look - It has 3 images that are trying to download, but failing…

UUID

a01bdc618bba3602677c8119e31d8002

Tim

(Attachment pi.log.zip is missing)

walpoletim · October 29, 2019, 8:02am

Some more logs as the pi restarted downloads…

Tim

Oct 29 07:59:39 a01bdc6 balenad[2809]: time=“2019-10-29T07:59:39.340164937Z” level=info msg=“loading plugin “io.containerd.grpc.v1.version”…” type=io.containerd.grpc.v1
Oct 29 07:59:39 a01bdc6 balenad[2809]: time=“2019-10-29T07:59:39.340552690Z” level=info msg=“loading plugin “io.containerd.grpc.v1.introspection”…” type=io.containerd.grpc.v1
Oct 29 07:59:39 a01bdc6 balenad[2809]: time=“2019-10-29T07:59:39.341894336Z” level=info msg=serving… address=/var/run/balena-engine/containerd/balena-engine-containerd-debug.sock
Oct 29 07:59:39 a01bdc6 balenad[2809]: time=“2019-10-29T07:59:39.342938690Z” level=info msg=serving… address=/var/run/balena-engine/containerd/balena-engine-containerd.sock
Oct 29 07:59:39 a01bdc6 balenad[2809]: time=“2019-10-29T07:59:39.343082244Z” level=info msg=“containerd successfully booted in 0.033828s”
Oct 29 08:00:19 a01bdc6 balenad[2809]: time=“2019-10-29T08:00:19.926765341Z” level=info msg=“shim balena-engine-containerd-shim started” address=/containerd-shim/moby/f07fbb3a71509b77a17a233bd42ae104045b9193352e7575c57f01a1765c23d0/shim.sock debug=false pid=3320
Oct 29 08:00:20 a01bdc6 balenad[2809]: time=“2019-10-29T08:00:20.326549773Z” level=info msg=“shim reaped” id=f07fbb3a71509b77a17a233bd42ae104045b9193352e7575c57f01a1765c23d0
Oct 29 08:00:20 a01bdc6 balenad[2809]: time=“2019-10-29T08:00:20.334087971Z” level=error msg=“stream copy error: reading from a closed fifo”
Oct 29 08:00:20 a01bdc6 balenad[2809]: time=“2019-10-29T08:00:20.334151118Z” level=error msg=“stream copy error: reading from a closed fifo”
Oct 29 08:00:32 a01bdc6 balenad[2809]: time=“2019-10-29T08:00:32.256141299Z” level=info msg=“shim balena-engine-containerd-shim started” address=/containerd-shim/moby/617f90e4ab026df4d0e6e0c28c8526b5de1e314c9c4098d9a9a5c36427572a2e/shim.sock debug=false pid=3351
yarn run v1.17.3
$ node lib/index.js
Oct 29 08:00:39 a01bdc6 617f90e4ab02[2809]: Connecting to mongodb on: mongodb://mongo:27017/accelerator
Oct 29 08:00:42 a01bdc6 617f90e4ab02[2809]: Failed trying to connect to mongodb. Trying again in 10 seconds.
Oct 29 08:00:45 a01bdc6 balenad[2809]: time=“2019-10-29T08:00:45.322313742Z” level=error msg=“f07fbb3a71509b77a17a233bd42ae104045b9193352e7575c57f01a1765c23d0 cleanup: failed to delete container from containerd: no such container”
Oct 29 08:00:45 a01bdc6 balenad[2809]: time=“2019-10-29T08:00:45.322477850Z” level=error msg=“Failed to start container f07fbb3a71509b77a17a233bd42ae104045b9193352e7575c57f01a1765c23d0: OCI runtime create failed: container_linux.go:345: starting container process caused “process_linux.go:424: container init caused \“rootfs_linux.go:58: mounting \\\”/var/run/balena-engine.sock\\\” to rootfs \\\”/var/lib/docker/overlay2/016f7b591797e5a988a2990801214475f269720ef1b4acd2a526372dc553b7bf/merged\\\" at \\\"/var/lib/docker/overlay2/016f7b591797e5a988a2990801214475f269720ef1b4acd2a526372dc553b7bf/merged/run/balena-engine.sock\\\" caused \\\“not a directory\\\”\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type"
Oct 29 08:00:52 a01bdc6 617f90e4ab02[2809]: Connecting to mongodb on: mongodb://mongo:27017/accelerator
Oct 29 08:00:55 a01bdc6 balenad[2809]: time=“2019-10-29T08:00:55.619454871Z” level=info msg=“shim balena-engine-containerd-shim started” address=/containerd-shim/moby/61a1a25592533ae270543be30bdbb3d3de0a0fa9a9c5671c05cd55343e045ade/shim.sock debug=false pid=3516
Oct 29 08:00:55 a01bdc6 617f90e4ab02[2809]: Failed trying to connect to mongodb. Trying again in 10 seconds.
Oct 29 08:00:59 a01bdc6 systemd[1]: balena.service: State ‘stop-sigterm’ timed out. Killing.
Oct 29 08:00:59 a01bdc6 systemd[1]: balena.service: Killing process 2809 (balenad) with signal SIGKILL.
Oct 29 08:00:59 a01bdc6 systemd[1]: balena.service: Killing process 3202 (balena-engine-c) with signal SIGKILL.
Oct 29 08:00:59 a01bdc6 systemd[1]: balena.service: Killing process 3293 (balena-engine-p) with signal SIGKILL.
Oct 29 08:00:59 a01bdc6 systemd[1]: balena.service: Killing process 3307 (balena-engine-p) with signal SIGKILL.
Oct 29 08:00:59 a01bdc6 systemd[1]: balena.service: Killing process 3351 (balena-engine-c) with signal SIGKILL.
Oct 29 08:00:59 a01bdc6 systemd[1]: balena.service: Killing process 3488 (balena-engine-p) with signal SIGKILL.
Oct 29 08:00:59 a01bdc6 systemd[1]: balena.service: Killing process 3508 (balena-engine-p) with signal SIGKILL.
Oct 29 08:00:59 a01bdc6 systemd[1]: balena.service: Killing process 3516 (balena-engine-c) with signal SIGKILL.
Oct 29 08:00:59 a01bdc6 systemd[1]: balena.service: Killing process 3583 (balena-engine-r) with signal SIGKILL.
Oct 29 08:01:05 a01bdc6 systemd[1]: balena.service: Main process exited, code=killed, status=9/KILL
Oct 29 08:01:05 a01bdc6 systemd[1]: balena.service: Failed with result ‘timeout’.
Oct 29 08:01:05 a01bdc6 systemd[1]: Failed to start Balena Application Container Engine.
Oct 29 08:01:05 a01bdc6 resin-supervisor[3589]: Cannot connect to the balenaEngine daemon at unix:///var/run/balena-engine.sock. Is the balenaEngine daemon running?
Oct 29 08:01:05 a01bdc6 systemd[1]: Dependency failed for Balena Application Container Engine.
Oct 29 08:01:05 a01bdc6 systemd[1]: balena.service: Job balena.service/start failed with result ‘dependency’.
Oct 29 08:01:05 a01bdc6 resin-supervisor[3599]: inactive
Oct 29 08:01:05 a01bdc6 systemd[1]: resin-supervisor.service: Control process exited, code=exited, status=3/NOTIMPLEMENTED
Oct 29 08:01:05 a01bdc6 systemd[1]: resin-supervisor.service: Failed with result ‘exit-code’.
Oct 29 08:01:05 a01bdc6 systemd[1]: Failed to start Resin supervisor.

walpoletim · October 29, 2019, 8:07am

This may be of interest as well…

Tim

Oct 29 08:04:48 a01bdc6 systemd[1]: balena-engine.socket: Failed to listen on sockets: Address already in use
Oct 29 08:04:48 a01bdc6 systemd[1]: balena-engine.socket: Failed with result ‘resources’.
Oct 29 08:04:48 a01bdc6 systemd[1]: Failed to listen on Docker Socket for the API.
Oct 29 08:04:48 a01bdc6 systemd[1]: Dependency failed for Balena Application Container Engine.
Oct 29 08:04:48 a01bdc6 systemd[1]: balena.service: Job balena.service/start failed with result ‘dependency’.
Oct 29 08:04:48 a01bdc6 resin-supervisor[3803]: Cannot connect to the balenaEngine daemon at unix:///var/run/balena-engine.sock. Is the balenaEngine daemon running?
Oct 29 08:04:48 a01bdc6 resin-supervisor[3811]: inactive
Oct 29 08:04:48 a01bdc6 systemd[1]: resin-supervisor.service: Control process exited, code=exited, status=3/NOTIMPLEMENTED
Oct 29 08:04:48 a01bdc6 systemd[1]: resin-supervisor.service: Failed with result ‘exit-code’.
Oct 29 08:04:48 a01bdc6 systemd[1]: Failed to start Resin supervisor.
Oct 29 08:04:59 a01bdc6 systemd[1]: balena-engine.socket: Failed to create listening socket (/var/run/balena-engine.sock): Address already in use
Oct 29 08:04:59 a01bdc6 systemd[1]: balena-engine.socket: Failed to listen on sockets: Address already in use
Oct 29 08:04:59 a01bdc6 systemd[1]: balena-engine.socket: Failed with result ‘resources’.
Oct 29 08:04:59 a01bdc6 systemd[1]: Failed to listen on Docker Socket for the API.
Oct 29 08:04:59 a01bdc6 systemd[1]: Dependency failed for Balena Application Container Engine.
Oct 29 08:04:59 a01bdc6 systemd[1]: balena.service: Job balena.service/start failed with result ‘dependency’.
Oct 29 08:04:59 a01bdc6 resin-supervisor[3813]: Cannot connect to the balenaEngine daemon at unix:///var/run/balena-engine.sock. Is the balenaEngine daemon running?
Oct 29 08:04:59 a01bdc6 resin-supervisor[3821]: inactive
Oct 29 08:04:59 a01bdc6 systemd[1]: resin-supervisor.service: Control process exited, code=exited, status=3/NOTIMPLEMENTED
Oct 29 08:04:59 a01bdc6 systemd[1]: resin-supervisor.service: Failed with result ‘exit-code’.
Oct 29 08:04:59 a01bdc6 systemd[1]: Failed to start Resin supervisor.
Oct 29 08:05:09 a01bdc6 systemd[1]: balena-engine.socket: Failed to create listening socket (/var/run/balena-engine.sock): Address already in use
Oct 29 08:05:09 a01bdc6 systemd[1]: balena-engine.socket: Failed to listen on sockets: Address already in use
Oct 29 08:05:09 a01bdc6 systemd[1]: balena-engine.socket: Failed with result ‘resources’.
Oct 29 08:05:09 a01bdc6 systemd[1]: Failed to listen on Docker Socket for the API.
Oct 29 08:05:09 a01bdc6 systemd[1]: Dependency failed for Balena Application Container Engine.
Oct 29 08:05:09 a01bdc6 systemd[1]: balena.service: Job balena.service/start failed with result ‘dependency’.
Oct 29 08:05:09 a01bdc6 resin-supervisor[3822]: Cannot connect to the balenaEngine daemon at unix:///var/run/balena-engine.sock. Is the balenaEngine daemon running?
Oct 29 08:05:09 a01bdc6 resin-supervisor[3831]: inactive
Oct 29 08:05:09 a01bdc6 systemd[1]: resin-supervisor.service: Control process exited, code=exited, status=3/NOTIMPLEMENTED
Oct 29 08:05:09 a01bdc6 systemd[1]: resin-supervisor.service: Failed with result ‘exit-code’.
Oct 29 08:05:09 a01bdc6 systemd[1]: Failed to start Resin supervisor.
Oct 29 08:05:19 a01bdc6 systemd[1]: balena-engine.socket: Failed to create listening socket (/var/run/balena-engine.sock): Address already in use
Oct 29 08:05:19 a01bdc6 systemd[1]: balena-engine.socket: Failed to listen on sockets: Address already in use
Oct 29 08:05:19 a01bdc6 systemd[1]: balena-engine.socket: Failed with result ‘resources’.
Oct 29 08:05:19 a01bdc6 systemd[1]: Failed to listen on Docker Socket for the API.
Oct 29 08:05:19 a01bdc6 systemd[1]: Dependency failed for Balena Application Container Engine.
Oct 29 08:05:19 a01bdc6 systemd[1]: balena.service: Job balena.service/start failed with result ‘dependency’.
Oct 29 08:05:19 a01bdc6 resin-supervisor[3832]: Cannot connect to the balenaEngine daemon at unix:///var/run/balena-engine.sock. Is the balenaEngine daemon running?
Oct 29 08:05:19 a01bdc6 resin-supervisor[3840]: inactive
Oct 29 08:05:19 a01bdc6 systemd[1]: resin-supervisor.service: Control process exited, code=exited, status=3/NOTIMPLEMENTED
Oct 29 08:05:19 a01bdc6 systemd[1]: resin-supervisor.service: Failed with result ‘exit-code’.
Oct 29 08:05:19 a01bdc6 systemd[1]: Failed to start Resin supervisor.
Oct 29 08:05:29 a01bdc6 systemd[1]: balena-engine.socket: Failed to create listening socket (/var/run/balena-engine.sock): Address already in use
Oct 29 08:05:29 a01bdc6 systemd[1]: balena-engine.socket: Failed to listen on sockets: Address already in use
Oct 29 08:05:29 a01bdc6 systemd[1]: balena-engine.socket: Failed with result ‘resources’.
Oct 29 08:05:29 a01bdc6 systemd[1]: Failed to listen on Docker Socket for the API.
Oct 29 08:05:29 a01bdc6 systemd[1]: Dependency failed for Balena Application Container Engine.
Oct 29 08:05:29 a01bdc6 systemd[1]: balena.service: Job balena.service/start failed with result ‘dependency’.
Oct 29 08:05:29 a01bdc6 resin-supervisor[3841]: Cannot connect to the balenaEngine daemon at unix:///var/run/balena-engine.sock. Is the balenaEngine daemon running?
Oct 29 08:05:29 a01bdc6 resin-supervisor[3849]: inactive
Oct 29 08:05:29 a01bdc6 systemd[1]: resin-supervisor.service: Control process exited, code=exited, status=3/NOTIMPLEMENTED
Oct 29 08:05:29 a01bdc6 systemd[1]: resin-supervisor.service: Failed with result ‘exit-code’.
Oct 29 08:05:29 a01bdc6 systemd[1]: Failed to start Resin supervisor.

walpoletim · October 29, 2019, 8:17am

Once its in this state, it never seems to recove, and even through the downloads say they are downloading, they never seem to complete…

Anything you want me to try ?

Tim

walpoletim · October 29, 2019, 9:08am

Hi

I can’t upload the zip logs to the forum - who can I email them to

Tim

zubairlk · October 29, 2019, 1:28pm

In your application, are you using the balena-engine socket or trying to restart balena/containers?

The reason I’m asking is that for some reason, in the HostOS, the balena-engine.socket file has become a folder. This happens when the socket file doesn’t exist and something(application code?) tries to mount it as a volume such as a label inside.

root@a01bdc6:~# ls /var/run/balena-engine.sock/
root@a01bdc6:~# rm -rf /var/run/balena-engine.sock/
root@a01bdc6:~# systemctl start balena-engine.socket
root@a01bdc6:~# systemctl start balena

I’ve seen this before in one instance as well https://github.com/balena-os/meta-balena/issues/1633

Its rare and we would like to mitigate the issue. Need more information on how this happened.

Regards
ZubairLK

zubairlk · October 29, 2019, 1:28pm

The device is downloading the application now.

walpoletim · October 29, 2019, 2:39pm

Hi

My application is not doing anything with Balena ita pure docker node.js or python containeras

It’s

Mongo
Vue with nginx reverse proxy port 80
Koa api port 3000
Koa runtime port 3001
Snips NLP server port 5000

Tim

walpoletim · October 29, 2019, 3:16pm

Let me know if there is anything you want me to try when I get home tonight

I don’t do anything with Balina in my containers

Tim

Topic		Replies	Views
Persistent "Failed to download image due to 'connect ECONNREFUSED /var/run/balena-engine.sock" error Product support	30	2755	April 13, 2022
Issues downloading images to boards Product support	20	2463	March 9, 2019
Failed to download image when disk usage very high Product support	10	949	October 6, 2020
Download error, URL wrong Product support	24	2569	April 28, 2020
Device stuck in Updating loop Product support	16	1784	July 4, 2019

Raspberry Pi 4 - Keeps restarting upload

Related topics