Error running CI build: server error - invalid reference format

I suspect this isn’t an issue directly with the Balena CLI, but I’m hoping you can help me out.
I’ve got a balena application that builds just fine when pushed to balena, or emulated build on my OSX machine, but fails when run on our CI server.

The error message is:
DEBUG=1 balena build --arch armv7hf --deviceType raspberrypi3 --logs --emulated [Debug] Parsing input... [Debug] Loading project... [Debug] Resolving project... [Info] Compose file detected [Debug] Creating project... [Info] Building for armv7hf/raspberrypi3 [Build] Building services... [Build] jenkins Preparing... [Build] tinc Preparing... [Build] yoctopuce-virtualhub Preparing... [Info] Installing qemu for armv7hf emulation... [Info] Emulation is enabled [Debug] Found build tasks: [Debug] jenkins: build [./jenkins] [Debug] tinc: build [./tinc] [Debug] yoctopuce-virtualhub: build [./yoctopuce-virtualhub] [Debug] Resolving services with [raspberrypi3|armv7hf] [Debug] Found project types: [Debug] jenkins: Dockerfile.template [Debug] tinc: Dockerfile.template [Debug] yoctopuce-virtualhub: Dockerfile.template [Debug] Prepared tasks; building... [Build] Built 3 services in 0:10 [Error] Build failed Error: (HTTP code 500) server error - invalid reference format at /snapshot/balena-cli/node_modules/docker-modem/lib/modem.js:257:17 at IncomingMessage.<anonymous> (/snapshot/balena-cli/node_modules/docker-modem/lib/modem.js:284:9) at emitNone (events.js:91:20) at IncomingMessage.emit (events.js:185:7) at endReadableNT (_stream_readable.js:974:12) at _combinedTickCallback (internal/process/next_tick.js:80:11) at process._tickDomainCallback (internal/process/next_tick.js:128:9)

The balena CLI is invoked via a docker image:
FROM ubuntu:16.04 ENV BUILD_DEPS curl unzip docker.io ca-certificates RUN apt-get update && \ apt-get -y --no-install-recommends install ${BUILD_DEPS} && \ update-ca-certificates && \ apt-get clean && \ rm -rf /var/lib/apt/lists/* ENV VERSION v10.0.0 RUN curl https://github.com/balena-io/balena-cli/releases/download/${VERSION}/balena-cli-${VERSION}-linux-x64.zip \ --verbose -L -o /tmp/balena-cli-${VERSION}-linux-x64.zip && \ unzip /tmp/balena-cli-${VERSION}-linux-x64.zip -d /tmp/ && \ ln -s /tmp/balena-cli/balena /usr/bin/balena ENV DEBUG 1 CMD ["balena", "build", "--arch", "armv7hf", "--deviceType", "raspberrypi3", "--logs", "--emulated"]

The one thing that stands out to me is that because of the Docker-over-Docker deployment, the docker sock is running an older version of docker than the local client.

I also found some hints that the error message could be due to YAML issues, but I appears to be ok. The application code can be found here: https://dashboard.balena-cloud.com/apps/1384945/devices

Hi there. If the problem is indeed the version of Docker, you could try running a docker-in-docker image: https://hub.docker.com/_/docker
I understand that it does not perform as well as simply sharing the host’s docker daemon socket, but if it solves the problem…
As you probably know, the balena build command has options to specify a docker daemon socket or ip address and port number pair:

--docker, -P <docker>
Path to a local docker socket (e.g. /var/run/docker.sock)

--dockerHost, -h <dockerHost>
Docker daemon hostname or IP address (dev machine or balena device)

--dockerPort, -p <dockerPort>
Docker daemon TCP port number (hint: 2375 for balena devices)

And more: https://www.balena.io/docs/reference/cli/#build-source

So if you were running a dind container listening on a different port number or socket path than the host docker, you could direct the balena CLI to use them.

Do you know what the docker version is in your CI’s host machine? We could try to reproduce the “invalid reference format” error by installing that specific version of docker.

Hi.
The versions of docker are:
server is v17.05.0
client is v18.09.2

Thanks for the DinD link. I’ll give that a go.
Daniel.

Is there an example of this? I use dind for gitlab-ci, but the usage of this for building items within Balena is not understood.

I have tried, using all Balena images, to build a basic django app with zero luck. Always on Build Server or with Balena CLI:

(HTTP code 500) server error - {“message”:“invalid reference format”}

or

Error: (HTTP code 500) server error - invalid reference format