Building ARM Containers on x86 Machines

I’m following these instructions to be able to run ARM code on x86: https://www.balena.io/docs/reference/base-images/base-images/#building-arm-containers-on-x86-machines

Unfortunately after loading up a fresh NUC image and loading that Dockerfile into the image, the container just restarts over and over. Produces the familiar error message: ‘standard_init_linux.go:207: exec user process caused “exec format error”’.

Any ideas?

Host OS Version: BalenaOS 2.39.0+rev3
Supervisor Version: 10.0.3

Dockerfile:

FROM balenalib/armv7hf-debian

RUN [ "cross-build-start" ]

RUN apt-get update
RUN apt-get install python-pip
RUN pip install virtualenv

RUN [ "cross-build-end" ]

Hello @spacelite
You need to tell the kernel to run arm binaries with qemu. This can be achieved with:

mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc  
echo ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm-static:' > /proc/sys/fs/binfmt_misc/register

Hi @zvin. I’m not totally clear. Do I add those as Docker ‘RUN’ commands?

At least from another official example, I don’t see those lines in the Dockerfile. Here’s the repo I’m talking about: https://github.com/petrosagg/armv7hf-python-dockerhub

No, you have to do this on your host computer as root.

I may have misundestood the problem.
Why are you trying to run an arm docker image on a NUC ?

Forget my comment above about binfmt_misc, it shouldn’t be necessary.
The idea of running ARM code on x86 is to be able to build an arm image on x86.
If you have a nuc, you should use an x86 base image.

Gotcha! It was my understanding that this was to run ARM code on x86, but it appears I was mistaken. Thanks for the super quick replies!!

You might be able to run arm code on x86 with the binfmt_misc trick above.
You’d need to run it in your Dockerfile.
You still need an x86 base image.

Something like:

FROM balenalib/intel-nuc-debian

COPY qemu-arm-static /usr/bin/qemu-arm-static

RUN mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc  

RUN echo ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm-static:' > /proc/sys/fs/binfmt_misc/register

RUN /some/arm/executable/file

The container probably needs to be privileged.
I’m not sure this will work as I don’t know if /proc/sys/fs/binfmt_misc is accessible from containers.

@zvin You beat me to it :wink:

I did end up playing with it a bit more and ended up simplifying some. I found out the following packages take care of all that work! So this dockerfile allows for transparently running ARM binaries on x86:

FROM balenalib/amd64-debian

# Installs packages needed to transparently run ARM binaries on x64
RUN install_packages qemu-user-static binfmt-support crossbuild-essential-armhf

# Keep container alive so we can SSH in to experiment
CMD sleep infinity

Awesome! Thank you for sharing your findings :slight_smile:

I’m using the balenalib/raspberrypi3:latest image from dockerhub as a base image.
Building works fine without installing qemu-user-static and binfmt-support on my x86_64 host because of the marvelous RUN [ "cross-build-start" ] prologue and RUN [ "cross-build-end" ] epilogue, which is brilliantly explained in this balena.io blog article:

https://www.balena.io/blog/building-arm-containers-on-any-x86-machine-even-dockerhub/

… however it seems that this mechanism can’t be leveraged when running arm binaries via docker runas well (not just docker build). I have to resort to installing qemu-user-static and binfmt-support again on the x86_64 host.

Is there a simple modification to the command I give to docker run that will get it to execute in the same way as arm binaries do during docker build?

You can install binfmt on your host computer, so that all arm binaries get ran with qemu. Depending on your host OS the process could be quite different.

What is the reason that you want to run arm containers on your host? Often you have just change the base image of the docker build and most things will work out of the box.

I’m running balena images on a CI server to create sustainable build environments that compile executables for a range of different systems and architectures, irrespective of the host doing the build.

The CI server builds the docker images and then instantiates each of them using docker run --rm -t -v <image_name> make all to compile architecture/platform-specific executables, which are written out to a mounted volume.

Unfortunately, I don’t have permission to install qemu-user-static and binfmt-support on the CI server itself. This isn’t a problem for docker build which you’ve provided a nice solution for, but do you have any advice for using ``docker run``` on the same CI server?

For example, is the solution to prefix each build command with /usr/bin/qemu-user-static e.g.:

docker run --rm -it <image_name> /usr/bin/qemu-user-static make all

Or perhaps I should omit the RUN [ "cross-build-end" ] statement in the Dockerfile to leave the /bin/sh shim in place?

Ideally, I’d like to strive for the same clean syntax you’ve made possible for docker build.