Non-deterministic failures on armv7hf

We’ve seen some non-deterministic failures on armv7hf (Balena Fin/Raspberry Pi 3) starting circa 2018-12-26 (yesterday).

The relevant part of our Dockerfile:

RUN cd ~ &&
wget https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-armhf-32bit-static.tar.xz &&
tar xvf ./ffmpeg-release-armhf-32bit-static.tar.xz &&
cd ~/ffmpeg--static &&
mv ./ffmpeg /usr/bin &&
ffmpeg -version &&
rm -rf ~/ffmpeg-
-static ~/ffmpeg*.tar.xz

Example crash (excerpt):

[main] ffmpeg-4.1-armhf-32bit-static/ffmpeg-10bit
[main] ffmpeg-4.1-armhf-32bit-static/qt-faststart
[main] ffmpeg-4.1-armhf-32bit-static/ffmpeg
[main] ffmpeg-4.1-armhf-32bit-static/GPLv3.txt
[main] qemu: uncaught target signal 4 (Illegal instruction) - core dumped
[main]
[main] Illegal instruction (core dumped)

When a build fails this way, just re-running it often seems to do the trick.

Why is qemu involved here? I thought this stuff was running on ARM HW, as per https://www.balena.io/blog/docker-builds-on-arm-servers-youre-not-crazy-your-builds-really-are-5x-faster/ ?

@jschon, you’re right, image builds for ARM devices normally take place on native ARM servers. However, if the builder fails to get hold of an ARM server from a pool (because of high latency/demand, or any other issues), the builders falls back to using QEMU. I can confirm observing intermittent latency issues with the ARM servers and some of my test builds occasionally revert to QEMU – but they always succeed, whether QEMU or native ARM hardware. It’s interesting to see that the “illegal instruction” error you get may be coming from the Dockerfile line that executes ffmpeg -version – if I was to hazard a guess, I’d say ffmpeg is using video-optimised processor instructions that are supported by ARM hardware but not by QEMU.

I am raising this with our ops team for high priority investigation - thank you for reporting it.

Meanwhile, consider if local device builds could be a better or workaround option for you, using the balena CLI push command followed perhaps by the deploy command with a pre-built image:

balena push <device_IP_address> -s <source directory>
balena deploy myApp myApp/myImage
balena help push
balena help deploy

https://www.balena.io/docs/reference/cli/

Kind regards,
Paulo

1 Like

Hi @jschon, just an update for you, as all of our arm builders instances, are available. Indeed some instances were unhealthy hence not included in our arm builder pools for some hours, so in some cases, fallback to QEMU builds could be encountered.
Best regards!