Cloud build fails, but local device build works on Raspberry Pi Zero

I have an application on balenaClould that targets the Raspberry Pi Zero.

My dockerfile downloads and runs a prebuilt binary (of Swift and the Swift Package Manager) that was built for the Raspberry Pi Zero with Raspbian.

This Swift binary is then used to compile my application.

The docker image builds and runs successfully using balena local push.

It also builds and runs successfully using balena deploy --build --source . --dockerHost <shortUUID>.local --dockerPort 2375 AppName as discussed here: Are you still running some image builders on an emulator?

Unfortunately, the build fails using the native arm cloud builders (it hangs indefinitely when asking the Swift Package Manager binary what version it is).

The build also fails using the emulated cloud builders as suggested here: Resin.io's cloud fails to compile properly (the build gets a little further than the native builders, but still ultimately fails).

Using the local Raspberry Pi Zero as the builder is a workaround, but it’s very slow compared to the cloud builders. Any suggestions / context as to why the cloud builders could fail here would be greatly appreciated.

Probably unrelated – but for some additional context: I have a different Swift application that targets the Raspberry Pi 3 that is able to build with the native arm cloud builders (this app uses a different Swift binary that’s built for the Raspberry Pi 3 architecture).

Hello ! Could you share the Dockerfile and the build output for us to be able to reproduce ? Thank you

@cyplo here’s an example that hangs during swift package --version on the native arm builders.

Output: https://gist.github.com/wlisac/96275d4d286a5c476095e54da0b6ceb1

Sorry for the messy dockerfile – I’m happy to simplify the steps to reproduce if it would make things easier.

FROM balenalib/raspberry-pi-debian:stretch

USER root

RUN apt-get update
RUN apt-get upgrade -y

RUN apt-get install -y wget git cmake ninja-build clang-3.8 python uuid-dev libicu-dev icu-devtools libbsd-dev libedit-dev libxml2-dev libsqlite3-dev swig libpython-dev libncurses5-dev pkg-config libblocksruntime-dev libcurl4-openssl-dev autoconf libtool systemtap-sdt-dev libcurl4-openssl-dev libz-dev

RUN curl -s https://packagecloud.io/install/repositories/swift-arm/release/script.deb.sh | bash

RUN apt-get install -y libxml2

RUN apt-get install swift4rpi01

WORKDIR /package

COPY . ./

RUN swift --version
RUN swift package --version

RUN swift package resolve
RUN swift package clean
RUN swift build

ENTRYPOINT ["./.build/debug/Run"]

I’ve done a little investigation into this and it seems that it’s something to do with the emulation of armv6, both on our emulated builders (running qemu) and our native arm builders (which are actually aarch64 with hardware arm emulation). Running strace on the command doesn’t reveal all that much, and I’m unsure what exactly is going wrong.

Whilst we look into why this may be, is it possible that you only use the swift binary at runtime? If not, I’d say that the only other option you have currently is to build remotely on a pi zero (or rpi1 if you have one around) and balena deploy the result.

I’ve done a little investigation into this and it seems that it’s something to do with the emulation of armv6, both on our emulated builders (running qemu) and our native arm builders (which are actually aarch64 with hardware arm emulation). Running strace on the command doesn’t reveal all that much, and I’m unsure what exactly is going wrong.
Whilst we look into why this may be, is it possible that you only use the swift binary at runtime? If not, I’d say that the only other option you have currently is to build remotely on a pi zero (or rpi1 if you have one around) and balena deploy the result.

The swift binary is the compiler – so it needs to be used at build time.

It’s definitely nice to be able to work around this using the Pi Zero as the builder, so I’ll continue to do that for now – it’s just incredibly slow (hours vs minutes). I think I can reduce this time with some cache magic, so that’s my next step for now.

Let me know if I can help with the emulator debugging process at all.

Cheers,
Will

Thanks for the offer. Hopefully it will just be a case of adding some compatibility flags (as we’ve needed to this before for armv6).

@CameronDiver Any luck determining what compatibility flags may be needed to support this?

Hey @wlisac we’re currently testing some compatibility kernel flags currently for other (similar) issues with the ARM builders, so hopefully this will also help the issue that you’re seeing.

Interestingly, my colleague tried to reproduce the issue that you were having on a local qemu binary (of a lower version which we have on our builders) and it worked fine. Until we manage to find the root cause of the failure on both the emulated builders and native builders, I would recommend using balena build, which will use qemu locally to build your containers, and then you can use balena deploy to push it to a device.

1 Like

@CameronDiver Thanks for the update.

I also tested using balena build and balena deploy and you’re right – that does appear to be another workaround to the problem. Thank you :smiley:

For what it’s worth, I’ve found the workaround to be a bit unreliable (sometimes it still fails to build), but it definitely works sometimes.

I’ve also made a new base image (willlisac/rpi-01-swift:4.1.3) to make testing easier if you’re interested in using it.

Source for base image: https://github.com/helje5/dockSwiftOnARM/blob/9f6876ac32b3da44b557b0ed384d503c7f52a02a/rpi-armv6-swift-4.1.3-uraimo-ctx/rpi-armv6-debian-stretch-swift-4.1.3.dockerfile

Example usage:

FROM willlisac/rpi-01-swift:4.1.3

USER root

WORKDIR /package

COPY . ./

RUN swift --version
RUN swift package --version

RUN swift package resolve
RUN swift package clean
RUN swift build

ENTRYPOINT ["./.build/debug/Run"]

@wlisac I think Swift only supports 64-bit Linux so I’m not surprised there’s no compatibility flags that would make this work. I’m basing this on what is on their site (scroll down to Linux): https://swift.org/download/#using-downloads

@dfunckt yeah – official Swift binaries for Linux are only supported on x86-64.

However, there is a community using Swift on ARM Linux and producing binaries for ARM 32bit and ARM 64bit.

I’m currently using Swift on Raspberry Pi Zero / 3 with Balena.

If you’re curious, here’s a recent update on the status of Swift on ARM: https://www.uraimo.com/2019/04/11/A-long-overdue-update-on-Swift-5-0-for-raspberry-pi-zero-1-2-3/

The issue above is specific to armv6 on the native ARM cloud builders. I haven’t confirmed this is still an issue in a while – I’ll try and find time to do that this week using the latest Swift ARM binaries.

Cheers,
Will

I tested this again tonight and the issue appears to have been resolved.

I tested multiple Swift versions (4.1.3, 4.2.2, 4.2.3, and 5.0) against the same project and all of the builds worked for Raspberry Pi Zero using Balena’s native arm cloud builders.

Maybe some of the compatibility flags have been updated on the builders?

It’s exciting to see these builds working now.

Thanks for all of the help,
Will

@wlisac just to give you some context … It’s a longer story, I’ll try to make it short. ARMv6 has CP15 barrier instructions. These are deprecated since ARMv7. Linux kernel can emulate or HW exec them (if CPU supports it). Some details here. Our builders have ARMv8 CPU and the kernel was configured to emulate these instructions. Unfortunately, we’ve found that the emulation doesn’t work. I experienced same issue (hang) with Rust installer and other ARMv6 binaries running on our builders. Started digging, found the kernel issue and the fix was deployed this Tuesday (23rd April). Here’re other details if you’re interested. In other words, any ARMv6 binary using CP15 barrier instructions was affected. That’s the reason why it started to work for you.

1 Like

Fantastic. Thanks for the additional details.

1 Like