# Non-deterministic failures on armv7hf

**URL:** <https://forums.balena.io/t/non-deterministic-failures-on-armv7hf/4836>\
**Category:** Product support\
**Created:** [December 27, 2018, 5:51pm UTC](https://forums.balena.io/t/non-deterministic-failures-on-armv7hf/4836 "2018-12-27T17:51:57Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![jschon](https://avatars.discourse-cdn.com/v4/letter/j/4da419/32.png) [@jschon](https://forums.balena.io/u/jschon)\
**Post date:** [December 27, 2018, 5:51pm UTC](https://forums.balena.io/t/non-deterministic-failures-on-armv7hf/4836/1 "2018-12-27T17:51:58Z")

</div>

We’ve seen some non-deterministic failures on armv7hf (Balena Fin/Raspberry Pi 3) starting circa 2018-12-26 (yesterday).

The relevant part of our Dockerfile:

> RUN cd ~ &&   
> wget [https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-armhf-32bit-static.tar.xz](https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-armhf-32bit-static.tar.xz) &&   
> tar xvf ./ffmpeg-release-armhf-32bit-static.tar.xz &&   
> cd ~/ffmpeg-_-static &&   
> mv ./ffmpeg /usr/bin &&   
> ffmpeg -version &&   
> rm -rf ~/ffmpeg-_-static ~/ffmpeg\*.tar.xz

Example crash (excerpt):

> [main] ffmpeg-4.1-armhf-32bit-static/ffmpeg-10bit  
> [main] ffmpeg-4.1-armhf-32bit-static/qt-faststart  
> [main] ffmpeg-4.1-armhf-32bit-static/ffmpeg  
> [main] ffmpeg-4.1-armhf-32bit-static/GPLv3.txt  
> [main] qemu: uncaught target signal 4 (Illegal instruction) - core dumped  
> [main]  
> [main] Illegal instruction (core dumped)

When a build fails this way, just re-running it often seems to do the trick.

Why is qemu involved here? I thought this stuff was running on ARM HW, as per [Docker builds on ARM servers: you’re not crazy, your builds really are 5x faster](https://www.balena.io/blog/docker-builds-on-arm-servers-youre-not-crazy-your-builds-really-are-5x-faster/) ?

---

<div class="post-metadata">

**Author:** ![pdcastro](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/pdcastro/32/1264_2.png) [@pdcastro](https://forums.balena.io/u/pdcastro)\
**Post date:** [December 27, 2018, 10:17pm UTC](https://forums.balena.io/t/non-deterministic-failures-on-armv7hf/4836/6 "2018-12-27T22:17:06Z")

</div>

@jschon, you’re right, image builds for ARM devices normally take place on native ARM servers. However, if the builder fails to get hold of an ARM server from a pool (because of high latency/demand, or any other issues), the builders falls back to using QEMU. I can confirm observing intermittent latency issues with the ARM servers and some of my test builds occasionally revert to QEMU – but they always succeed, whether QEMU or native ARM hardware. It’s interesting to see that the “illegal instruction” error you get may be coming from the Dockerfile line that executes `ffmpeg -version` – if I was to hazard a guess, I’d say ffmpeg is using video-optimised processor instructions that are supported by ARM hardware but not by QEMU.

I am raising this with our ops team for high priority investigation - thank you for reporting it.

Meanwhile, consider if local device builds could be a better or workaround option for you, using the balena CLI push command followed perhaps by the deploy command with a pre-built image:

```auto
balena push <device_IP_address> -s <source directory>
balena deploy myApp myApp/myImage
balena help push
balena help deploy

```

[https://www.balena.io/docs/reference/cli/](https://www.balena.io/docs/reference/cli/)

Kind regards,  
Paulo

---

<div class="post-metadata">

**Author:** ![mikesimos](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/mikesimos/32/1418_2.png) [@mikesimos](https://forums.balena.io/u/mikesimos)\
**Post date:** [December 28, 2018, 10:52am UTC](https://forums.balena.io/t/non-deterministic-failures-on-armv7hf/4836/12 "2018-12-28T10:52:11Z")

</div>

Hi @jschon, just an update for you, as all of our arm builders instances, are available. Indeed some instances were unhealthy hence not included in our arm builder pools for some hours, so in some cases, fallback to QEMU builds could be encountered.  
Best regards!