balenaEngine fails on Raspbian

Hello everyone,

if I install the balenaEngine on Raspbian, running any image fails. Do you know how to solve this issue?

Steps to reproduce:

  1. Flash Raspbian on a micro SD card using the Raspberry Pi Imager (Raspbian Release 2020-02-13)
  2. Add an empty file called SSH to the micro SD card to enable SSH
  3. Insert the SD card into a Raspberry Pi 4 and connect using SSH (credentials: User: pi, Password: raspberry)
  4. For skipping authentication, log in as root using sudo -i
  5. Install balenaEngine by running curl -sfL https://balena.io/engine/install.sh | sh
  6. Start the demon (backgrounded) using balena-engine-daemon & (see this post)
  7. Try to run an image: balena-engine run armhf/hello-world (Docker Hub)

Expected Results:

A “Hello World” text should be printed to the terminal. Similar to the one produced by Docker’s official hello-world image.

Actual results:

root@raspberrypi:~# balena-engine run armhf/hello-world
INFO[2020-04-20T14:55:07.086962609+01:00] ignoring event                                module=libcontainerd namespace=moby topic=/containers/create type="*events.ContainerCreate"
INFO[0132] shim balena-engine-containerd-shim started    address=/containerd-shim/moby/348e13e3da036faa15b398625aa21a97efa23c5ce79ba3aa5d4799b7727f64d5/shim.sock debug=false module=containerd/tasks pid=901
INFO[0132] shim reaped                                   id=348e13e3da036faa15b398625aa21a97efa23c5ce79ba3aa5d4799b7727f64d5 module=containerd/tasks
INFO[2020-04-20T14:55:07.603121457+01:00] ignoring event                                module=libcontainerd namespace=moby topic=/containers/delete type="*events.ContainerDelete"
ERRO[2020-04-20T14:55:08.116249750+01:00] 348e13e3da036faa15b398625aa21a97efa23c5ce79ba3aa5d4799b7727f64d5 cleanup: failed to delete container from containerd: no such container 
ERRO[2020-04-20T14:55:08.116375027+01:00] Handler for POST /v1.35/containers/348e13e3da036faa15b398625aa21a97efa23c5ce79ba3aa5d4799b7727f64d5/start returned error: unable to find "net_prio" in controller set: unknown 
balena-engine: Error response from daemon: unable to find "net_prio" in controller set: unknown.
ERRO[0001] error waiting for container: context canceled 

Notes:

I checked that the image works as expected if I connect to a Raspberry Pi running balenaOS through the webconsole provided by the balenaCloud. The error only occurs it I install balenaEngine on Raspbian. The balenaEngine on Raspbian also fails for images provided by the balenalib, e.g. balena-engine run -it balenalib/raspberrypi4-64-alpine /bin/bash.

Hello Christian, thanks for the detailed note and troubleshooting!

Looking at that error, it seems that it may be related to this issue: https://github.com/docker/for-linux/issues/545

The solutions, according to those comments, is to downgrade systemd (seems the error began appearing at version 240), or to recompile the kernel with CONFIG_CGROUP_NET_PRIO enabled.

I have pinged the balenaEngine maintainer though, to see if there is anything else that we can do to work around the issue. Thanks!

Thank you for the suggestion. Since the images work as expected using a current version of Docker, I hoped this could be fixed by updating or fixing the balenaEngine.

I’m not really a big fan of downgrading systemd or recompiling the kernel. Downgrading parts of the system may always cause new side effects and due to the current Corona crisis, I unfortunately have no easy physical access to most of my devices, which means reflashing the SD cards with a self-compiled kernel is not really an option for me.