Crosscompile go for arm from amd64 (with CGO)

dolanor · July 18, 2017, 12:18am

Hi,

before jumping into the action, I want to say that I was really impressed with your blog post . It was tempting not to build on rpi anymore. But, I got problems:

I want to build a go program for arm from amd64.
The problem is that go program rely on a library that uses a normal .so (the libltdl.so).
I got an error from the go build and I don’t know what the problem is. I suspect that the qemu env is limited in the number of threads it can deal with?

The Dockerfile (multistage, it fails on the stage 0, at the RUN go build …)

# nodered-peer App building stage
#FROM golang:1.8 as app-build
FROM resin/raspberrypi3-golang:1.8 as app-build


RUN [ "cross-build-start" ]
RUN apt-get update && apt-get install -y libltdl-dev && apt-get clean 

COPY . /go/src/github.com/tuxago/mychain/hyperledger

WORKDIR /go/src/github.com/tuxago/mychain/hyperledger

RUN go get -d -v github.com/tuxago/mychain/hyperledger/nodered-peer
RUN go build -o release/nodered-peer github.com/tuxago/mychain/hyperledger/nodered-peer
RUN [ "cross-build-end" ]

# Image creation stage
FROM alpine:latest
RUN apk --no-cache add ca-certificates libtool
COPY --from=app-build /go/src/github.com/tuxago/mychain/hyperledger/release/nodered-peer /nodered-peer
CMD [ "/nodered-peer" ]

The docker build log

 docker build  -t nodered-peer .
Sending build context to Docker daemon  150.6MB
Step 1/12 : FROM resin/raspberrypi3-golang:1.8 as app-build
 ---> 56ce92074b4b
Step 2/12 : RUN cross-build-start
 ---> Using cache
 ---> 914eacb0e230
Step 3/12 : RUN apt-get update && apt-get install -y libltdl-dev && apt-get clean
 ---> Using cache
 ---> b3896d16a42b
Step 4/12 : COPY . /go/src/github.com/tuxago/mychain/hyperledger
 ---> 2747e2500a29
Removing intermediate container a27fe73babb3
Step 5/12 : WORKDIR /go/src/github.com/tuxago/mychain/hyperledger
 ---> f4f53e85fb8e
Removing intermediate container 9a911e2a70ac
Step 6/12 : RUN go get -d -v github.com/tuxago/mychain/hyperledger/nodered-peer
 ---> Running in 12fc686d3724
Fetching https://gopkg.in/gin-gonic/gin.v1?go-get=1
Parsing meta tags from https://gopkg.in/gin-gonic/gin.v1?go-get=1 (status code 200)
get "gopkg.in/gin-gonic/gin.v1": found meta tag main.metaImport{Prefix:"gopkg.in/gin-gonic/gin.v1", VCS:"git", RepoRoot:"https://gopkg.in/gin-gonic/gin.v1"} at https://gopkg.in/gin-gonic/gin.v1?go-get=1
gopkg.in/gin-gonic/gin.v1 (download)
github.com/gin-contrib/sse (download)
github.com/gin-gonic/gin (download)
github.com/golang/protobuf (download)
github.com/json-iterator/go (download)
github.com/ugorji/go (download)
Fetching https://gopkg.in/go-playground/validator.v8?go-get=1
Parsing meta tags from https://gopkg.in/go-playground/validator.v8?go-get=1 (status code 200)
get "gopkg.in/go-playground/validator.v8": found meta tag main.metaImport{Prefix:"gopkg.in/go-playground/validator.v8", VCS:"git", RepoRoot:"https://gopkg.in/go-playground/validator.v8"} at https://gopkg.in/go-playground/validator.v8?go-get=1
gopkg.in/go-playground/validator.v8 (download)
Fetching https://gopkg.in/yaml.v2?go-get=1
Parsing meta tags from https://gopkg.in/yaml.v2?go-get=1 (status code 200)
get "gopkg.in/yaml.v2": found meta tag main.metaImport{Prefix:"gopkg.in/yaml.v2", VCS:"git", RepoRoot:"https://gopkg.in/yaml.v2"} at https://gopkg.in/yaml.v2?go-get=1
gopkg.in/yaml.v2 (download)
github.com/mattn/go-isatty (download)
 ---> 2b34c6efb505
Removing intermediate container 12fc686d3724
Step 7/12 : RUN go build -o release/nodered-peer github.com/tuxago/mychain/hyperledger/nodered-peer
 ---> Running in 63a0a05afbe4
# github.com/tuxago/mychain/hyperledger/nodered-peer/vendor/github.com/golang/protobuf/proto
runtime: failed to create new OS thread (have 2 already; errno=22)
fatal error: newosproc

runtime stack:
runtime.throw(0x602c20, 0x9)
        /go/src/runtime/panic.go:596 +0x70
runtime.newosproc(0x10c26000, 0x10c36000)
        /go/src/runtime/os_linux.go:163 +0x158
runtime.newm(0x61d28c, 0x0)
        /go/src/runtime/proc.go:1628 +0x10c
runtime.main.func1()
        /go/src/runtime/proc.go:126 +0x24
runtime.systemstack(0x8a6700)
        /go/src/runtime/asm_arm.s:264 +0x80
runtime.mstart()
        /go/src/runtime/proc.go:1132

goroutine 1 [running]:
runtime.systemstack_switch()
        /go/src/runtime/asm_arm.s:209 +0x4 fp=0x10c227bc sp=0x10c227b8
runtime.main()
        /go/src/runtime/proc.go:127 +0x58 fp=0x10c227e4 sp=0x10c227bc
runtime.goexit()
        /go/src/runtime/asm_arm.s:1017 +0x4 fp=0x10c227e4 sp=0x10c227e4
# github.com/tuxago/mychain/hyperledger/mychain
runtime: failed to create new OS thread (have 2 already; errno=22)
fatal error: newosproc

runtime stack:
runtime.throw(0x602c20, 0x9)
        /go/src/runtime/panic.go:596 +0x70
runtime.newosproc(0x10c24000, 0x10c34000)
        /go/src/runtime/os_linux.go:163 +0x158
runtime.newm(0x61d28c, 0x0)
        /go/src/runtime/proc.go:1628 +0x10c
runtime.main.func1()
        /go/src/runtime/proc.go:126 +0x24
runtime.systemstack(0x8a6700)
        /go/src/runtime/asm_arm.s:264 +0x80
runtime.mstart()
        /go/src/runtime/proc.go:1132

goroutine 1 [running]:
runtime.systemstack_switch()
        /go/src/runtime/asm_arm.s:209 +0x4 fp=0x10c207bc sp=0x10c207b8
runtime.main()
        /go/src/runtime/proc.go:127 +0x58 fp=0x10c207e4 sp=0x10c207bc
runtime.goexit()
        /go/src/runtime/asm_arm.s:1017 +0x4 fp=0x10c207e4 sp=0x10c207e4
# github.com/tuxago/mychain/hyperledger/nodered-peer/vendor/github.com/op/go-logging
runtime: failed to create new OS thread (have 2 already; errno=22)
fatal error: newosproc

runtime stack:
runtime.throw(0x602c20, 0x9)
        /go/src/runtime/panic.go:596 +0x70
runtime.newosproc(0x10c26000, 0x10c36000)
        /go/src/runtime/os_linux.go:163 +0x158
runtime.newm(0x61d28c, 0x0)
        /go/src/runtime/proc.go:1628 +0x10c
runtime.main.func1()
        /go/src/runtime/proc.go:126 +0x24
runtime.systemstack(0x8a6700)
        /go/src/runtime/asm_arm.s:264 +0x80
runtime.mstart()
        /go/src/runtime/proc.go:1132
# … It goes on for 1771 lines with the same idea

willnewton · July 18, 2017, 8:22am

Hi,

I think the problem you are hitting is this one: https://bugs.launchpad.net/qemu/+bug/1696353

Unfortunately it looks like the only way to fix the problem is to downgrade/patch qemu or patch golang and neither of these is likely to be very easy. We are planning to get some dedicated hardware for doing ARM builds which should make problems like this go away for good but I don’t have an ETA for that yet.

dolanor · July 18, 2017, 8:59am

Thanks for the quick reply!
And more than that, it’s exactly that and I have a view of what will happen.
I can wait for 1.10 and use an rpi to build in the meantime. We’re not to build every day anyway, so it is okay!

imrehg · July 19, 2017, 9:16am

Hey @dolanor Just a sidenote to clarify, the final container you want to run on a Raspberry Pi 3? Asking because in the the “image creation” stage you are using alpine:latest, which is x86 image, not resin/raspberrypi3-alpine (from resin) or even arm32v6/alpine (which is the official alpine ARM upstream image). Or have I misunderstood something?

dolanor · July 19, 2017, 10:15am

You’re totally right. I was building on amd64 before and since the first stage of the Dockerfile was not working I didn’t get to the fact that it wouldn’t have liked an amd64 alpine version either