Unfortunately, the build fails using the native arm cloud builders (it hangs indefinitely when asking the Swift Package Manager binary what version it is).
The build also fails using the emulated cloud builders as suggested here: Resin.io's cloud fails to compile properly (the build gets a little further than the native builders, but still ultimately fails).
Using the local Raspberry Pi Zero as the builder is a workaround, but it’s very slow compared to the cloud builders. Any suggestions / context as to why the cloud builders could fail here would be greatly appreciated.
Probably unrelated – but for some additional context: I have a different Swift application that targets the Raspberry Pi 3 that is able to build with the native arm cloud builders (this app uses a different Swift binary that’s built for the Raspberry Pi 3 architecture).
Sorry for the messy dockerfile – I’m happy to simplify the steps to reproduce if it would make things easier.
FROM balenalib/raspberry-pi-debian:stretch
USER root
RUN apt-get update
RUN apt-get upgrade -y
RUN apt-get install -y wget git cmake ninja-build clang-3.8 python uuid-dev libicu-dev icu-devtools libbsd-dev libedit-dev libxml2-dev libsqlite3-dev swig libpython-dev libncurses5-dev pkg-config libblocksruntime-dev libcurl4-openssl-dev autoconf libtool systemtap-sdt-dev libcurl4-openssl-dev libz-dev
RUN curl -s https://packagecloud.io/install/repositories/swift-arm/release/script.deb.sh | bash
RUN apt-get install -y libxml2
RUN apt-get install swift4rpi01
WORKDIR /package
COPY . ./
RUN swift --version
RUN swift package --version
RUN swift package resolve
RUN swift package clean
RUN swift build
ENTRYPOINT ["./.build/debug/Run"]
I’ve done a little investigation into this and it seems that it’s something to do with the emulation of armv6, both on our emulated builders (running qemu) and our native arm builders (which are actually aarch64 with hardware arm emulation). Running strace on the command doesn’t reveal all that much, and I’m unsure what exactly is going wrong.
Whilst we look into why this may be, is it possible that you only use the swift binary at runtime? If not, I’d say that the only other option you have currently is to build remotely on a pi zero (or rpi1 if you have one around) and balena deploy the result.
The swift binary is the compiler – so it needs to be used at build time.
It’s definitely nice to be able to work around this using the Pi Zero as the builder, so I’ll continue to do that for now – it’s just incredibly slow (hours vs minutes). I think I can reduce this time with some cache magic, so that’s my next step for now.
Let me know if I can help with the emulator debugging process at all.
Hey @wlisac we’re currently testing some compatibility kernel flags currently for other (similar) issues with the ARM builders, so hopefully this will also help the issue that you’re seeing.
Interestingly, my colleague tried to reproduce the issue that you were having on a local qemu binary (of a lower version which we have on our builders) and it worked fine. Until we manage to find the root cause of the failure on both the emulated builders and native builders, I would recommend using balena build, which will use qemu locally to build your containers, and then you can use balena deploy to push it to a device.
FROM willlisac/rpi-01-swift:4.1.3
USER root
WORKDIR /package
COPY . ./
RUN swift --version
RUN swift package --version
RUN swift package resolve
RUN swift package clean
RUN swift build
ENTRYPOINT ["./.build/debug/Run"]
@wlisac I think Swift only supports 64-bit Linux so I’m not surprised there’s no compatibility flags that would make this work. I’m basing this on what is on their site (scroll down to Linux): https://swift.org/download/#using-downloads
The issue above is specific to armv6 on the native ARM cloud builders. I haven’t confirmed this is still an issue in a while – I’ll try and find time to do that this week using the latest Swift ARM binaries.
I tested this again tonight and the issue appears to have been resolved.
I tested multiple Swift versions (4.1.3, 4.2.2, 4.2.3, and 5.0) against the same project and all of the builds worked for Raspberry Pi Zero using Balena’s native arm cloud builders.
Maybe some of the compatibility flags have been updated on the builders?
@wlisac just to give you some context … It’s a longer story, I’ll try to make it short. ARMv6 has CP15 barrier instructions. These are deprecated since ARMv7. Linux kernel can emulate or HW exec them (if CPU supports it). Some details here. Our builders have ARMv8 CPU and the kernel was configured to emulate these instructions. Unfortunately, we’ve found that the emulation doesn’t work. I experienced same issue (hang) with Rust installer and other ARMv6 binaries running on our builders. Started digging, found the kernel issue and the fix was deployed this Tuesday (23rd April). Here’re other details if you’re interested. In other words, any ARMv6 binary using CP15 barrier instructions was affected. That’s the reason why it started to work for you.