Hi! I tried to use balenalib/rpi-raspbian:buster , balenalib/raspberry-pi2:buster , balenalib/raspberrypi3:buster. The first one works fine, but the latter two don’t work. I can’t complete apt update command. My hosts are on several places (Europe and Asia).
Could you help me with it, please?
reproduce:
# on amd64 host
$ docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
$ docker run --rm -it balenalib/raspberrypi3:buster bash
...
apt update
<geting stuck here>
Hello, we’ve been able to reproduce this on true armv7 hardware so it seems to not be related to emulation.
balena run --rm -it balenalib/raspberrypi3:buster apt-get update
Digest: sha256:5d7ff4a4990c2d60663683626c9be7e6ff3b78272aa8ba07fe118cd910d5c0c4
Status: Downloaded newer image for balenalib/raspberrypi3:buster
Get:1 http://deb.debian.org/debian buster InRelease [122 kB]
Get:2 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:4 http://archive.raspberrypi.org/debian buster InRelease [32.6 kB]
Get:5 http://archive.raspbian.org/raspbian buster InRelease [15.0 kB]
0% [Working]
However, the same base image when used in a Dockerfile as FROM command seems to work, so something about the docker build vs run context seems to matter.
FROM balenalib/raspberrypi3:buster
RUN apt-get update
Sending build context to Docker daemon 2.048kB
Step 1/2 : FROM balenalib/raspberrypi3:buster
---> e9b6b7ca4bec
Step 2/2 : RUN apt-get update
---> [Warning] The requested image's platform (linux/arm/v7) does not match the detected host platform (linux/amd64) and no specific platform was requested
---> Running in f312d0626b8e
Here are a few details about this Docker image (For more information please visit https://www.balena.io/docs/reference/base-images/base-images/):
Architecture: ARM v7
OS: Debian Buster
Variant: run variant
Default variable(s): UDEV=off
Extra features:
- Easy way to install packages with `install_packages <package-name>` command
- Run anywhere with cross-build feature (for ARM only)
- Keep the container idling with `balena-idle` command
- Show base image details with `balena-info` command
Get:1 http://deb.debian.org/debian buster InRelease [122 kB]
Get:2 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:4 http://archive.raspbian.org/raspbian buster InRelease [15.0 kB]
Get:5 http://archive.raspberrypi.org/debian buster InRelease [32.6 kB]
Get:6 http://security.debian.org/debian-security buster/updates/main armhf Packages [306 kB]
Get:7 http://deb.debian.org/debian buster/main armhf Packages [7698 kB]
Get:8 http://deb.debian.org/debian buster-updates/main armhf Packages [14.4 kB]
Get:9 http://archive.raspbian.org/raspbian buster/main armhf Packages [13.0 MB]
Get:10 http://archive.raspbian.org/raspbian buster/rpi armhf Packages [1360 B]
Get:11 http://archive.raspbian.org/raspbian buster/contrib armhf Packages [58.8 kB]
Get:12 http://archive.raspbian.org/raspbian buster/non-free armhf Packages [104 kB]
Get:13 http://archive.raspbian.org/raspbian buster/firmware armhf Packages [1232 B]
Get:14 http://archive.raspberrypi.org/debian buster/main armhf Packages [393 kB]
Fetched 21.9 MB in 17s (1288 kB/s)
Reading package lists...
Removing intermediate container f312d0626b8e
---> d2546a2d6436
Successfully built d2546a2d6436
I’m still investigating but perhaps someone in the community has seen this.
I’ve narrowed it down to bash vs sh in the base image. If you specify sh as your shell command it seems to work. Likewise if you change your shell in your Dockerfile to SHELL ["/bin/bash, "-c"] it will break.
So I need to keep investigating, but it seems that the error is avoided when using /bin/sh as the cmd since this is actually a wrapper script that prints balena-info before executing /bin/sh.real. So the workaround is to not use bash as your command but instead use sh until we figure out why this is.
This is also fully reproducible on amd64 images, so this seems to be an issue with apt running as pid 1, but we will have to see why that is.
Hey @antmak, we’ve narrowed this down to an issue with the temporary shim we install over /bin/sh to print image details before restoring the original sh binary. apt-get update calls apt-key which execs another process (via the function ExecGPGV) which is the one that needs sh to be a shell. If sh is not an actual shell, thousands of processes will be spawned all trying to do the same thing (verify a key), and this could/does halt the system. My laptop crashed a few times while testing this if I forgot to kill the container process.
I have opened a GH issue to re-evaluate how we have implemented this shim in our base images. In the meantime, some of the known workarounds are:
docker run --rm -it balenalib/raspberrypi:buster sh apt-get update)docker run --rm -it balenalib/raspberrypi:buster apt-get update)docker run --rm -it balenalib/raspberrypi:buster sh)Again this only impacts the very first execution of /bin/sh. If any other commands like an interactive shell run first, the apt command will end up using the real binary. It’s only when skipping the initial sh shell that apt has to deal with it and fails.
Hey @klutchell ,
I’m having the same issue with the balenalib/raspberry-pi2-debian-node:18-run image on docker build. Might be related, although @antmak had the issue with docker run.
Was this ever fixed?
I tried to add SHELL [ "/bin/sh", "-c" ] to the top of my Dockerfile but it did not help.
Cheers
Fred
Hey @frederikheld, here is the GitHub issue which is not closed, but I had left a note that it should have been resolved by an upstream change.
Can you provide a stripped down version of your Dockerfile, and the exact issue you are seeing?