Can't compile GOLANG on armv7 via balena cross compile or docker edge arm support

Product support
1

I’m trying to build an image of logspout on MacOS for RPi3/armv7 from this repo/branch:

I’ve added a simple hello test to run through, because it also doesn’t build correctly but I can’t identify the problem (whether it’s docker or go or what). I’m on MacOS 10.14.6, running Docker Desktop Edge 2.1.4.0 (arm support).

If you checkout that repo/branch and run this, it will build fine:

docker buildx build --platform linux/arm64 --no-cache -f Dockerfile.hello .

If you run this however:

docker buildx build --platform linux/arm/v7 --no-cache -f Dockerfile.hello .

It throws this error:


#7 16.03 Linux buildkitsandbox 4.14.131-linuxkit #1 SMP Fri Jul 19 12:31:17 UTC 2019 armv7l Linux

#7 17.32 # command-line-arguments

#7 17.32 runtime: pointer 0xa00020 to unused region of span span.base()=0x800000 span.limit=0x801ff0 span.state=1

#7 17.32 runtime: found in object at *(0xfffc5008+0x7fe8)

#7 17.32 object=0xfffc5008 s=nil

#7 17.32 fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

So I wanted to see if I could use one of Balena’s images using cross-build-start, this one runs fine:

docker build --no-cache -f Dockerfile.balenacross .

So then I made one using cross-build-start to run/build the Go program:

docker build --no-cache -f Dockerfile.hellobalenacross .

And it throws the same error:

Linux 7f692e18a736 4.14.131-linuxkit #1 SMP Fri Jul 19 12:31:17 UTC 2019 armv7l Linux

# command-line-arguments

runtime: pointer 0xa00020 to unused region of span span.base()=0x800000 span.limit=0x809f00 span.state=1

runtime: found in object at *(0xfffc5008+0x7fe8)

object=0xfffc5008 s=nil

fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

Any ideas on what’s going on? Or even where the problem is so I can open a ticket there?

2

Hello,

This seems to be a qemu problem, this thing builds fine on a real armv7.
You can test it by pushing this code to balena which will do a native build.

4

Yes, I noticed that, just wanted to flag this here because all of the arm/docker code is built around qemu and I had never heard anyone talking about incompatibilities within it. Should I raise this issue with qemu then?

5

I guess.
Not sure if it should be on go or qemu.
I couldn’t find anyone reporting this issue on the Internet.
The closest thing I’ve found is https://github.com/golang/go/issues/19135 but it’s on an older version of go (1.8) and cross compiling from mac amd64 to mac i386.

11

Hi, I searched for similar issues reported on GitHub and I could find plenty reported on the Go lang repo. Please check whether you are running latest Go version and if that still does not work, I think the Go lang repo is the best place to report this.

Thanks,
Zahari

13

Just to add two things I discovered yesterday while poking around a little bit more. I decided to try this with ubuntu instead of alpine, and everything built/compiled correctly on the same MacOS/Docker combo that doesn’t work with the alpine image. I created another branch here:

docker buildx build --platform linux/arm/v7 --no-cache -f Dockerfile .

I also created a real armv7 server at scaleway and ran the ORIGINAL alpine branch code, and as noted by @zvin above, everything ran and compiled correctly. I even checked across that alpine docker image and the image that is being pulled by docker on MacOS, and it’s pulling the same one:

sha256:c19173c5ada610a5989151111163d28a67368362762534d8a8121ce95cf2bd5a

That server was running this version of Ubuntu:
Distributor ID: Ubuntu
Description: Ubuntu 16.04.5 LTS
Release: 16.04
Codename: xenial
Linux scw-elastic-mclaren 4.4.127-mainline-rev1 #1 SMP Sun Apr 8 10:33:17 UTC 2018 armv7l armv7l armv7l GNU/Linux
Docker version 18.09.0, build 4d60db4

I also tried changing the build image to arm32v7/alpine:latest on MacOS/Docker buildx, but that also failed (I was reading some places that alpine didn’t have real armv7 support?)

So not sure what this narrows it down to? An issue between Alpine/qemu and the linux kernel that comes with MacOS Docker? Linux buildkitsandbox 4.14.131-linuxkit #1 SMP Fri Jul 19 12:31:17 UTC 2019 armv7l Linux

Maybe the next step I will try is to get it to compile using the Docker CE edge buildx stuff on x86 ubuntu to see if it has the same problems? If it works correctly there, then it’s not a qemu problem but something specific to the MacOS docker kernel?

14

Possibly the difference is because Alpine uses musl libc instead of glibc. It is worth trying to build that on Ubuntu indeed. I still think it falls into the area of Go language itself though because it deals with those low level allocations. Here is an example recent commit: https://github.com/golang/go/commit/813d8e886208e5e001df3b55e8bc1c65ca4249ff

16

Ok, I will open an issue with Go lang.

Also, I tried it on Ubuntu at Scaleway on amd64/x86 and it failed the same as on MacOS. Steps to reproduce…

Spin up Ubuntu version: Ubuntu 18.04.2 LTS

I installed edge docker from .deb for this test:

wget https://download.docker.com/linux/ubuntu/dists/bionic/pool/edge/amd64/containerd.io_1.2.6-3_amd64.deb
wget https://download.docker.com/linux/ubuntu/dists/bionic/pool/edge/amd64/docker-ce_19.03.3~3-0~ubuntu-bionic_amd64.deb
wget https://download.docker.com/linux/ubuntu/dists/bionic/pool/edge/amd64/docker-ce-cli_19.03.4~3-0~ubuntu-bionic_amd64.deb

dpkg -i *.deb

Then follow these instructions to get buildx and gemu setup:

I did the buildx binary