Cannot build anything with Balena Base Images | exec user process caused "exec format error"

Winkelmann · November 17, 2020, 10:47pm

Hello again,

It has been a while since I last updated the software on my Balena devices.
Today I made a change, then ran the usual build command with the Balena CLI: balena deploy <appName> --logs --source . --emulated --build
However, I cannot build anything anymore.

Any command put in the Dockerfile.template will fail. Even a RUN ["echo", "'Hello there'"]
I tried many Balena base images, but none worked.
Then I tried a debian base image FROM debian:stretch, and it works as expected, but obviously I can’t use that image to build for my Balena devices.

Here is my Dockerfile.template:

# base-image for node on any machine using a template variable,
# see more about dockerfile templates here: https://www.balena.io/docs/learn/develop/dockerfile/#dockerfile-templates
# and about balena base images here: https://www.balena.io/docs/reference/base-images/base-images/
FROM balenalib/%%BALENA_MACHINE_NAME%%-node:10-stretch-build

# use `install_packages` if you need to install dependencies,
RUN install_packages python sqlite3 libsqlite3-dev build-essential curl tar libtool automake

# Enable udevd so that plugged dynamic hardware devices show up in our container.
ENV UDEV=1

# Compile libmbus

# Defines our working directory in container
WORKDIR /usr/src/app

RUN mkdir libmbus
RUN cd libmbus && curl -sL https://github.com/rscada/libmbus/archive/0.9.0.tar.gz | tar zx
WORKDIR /usr/src/app/libmbus/libmbus-0.9.0
RUN bash ./clean.sh
RUN bash ./build.sh
RUN make install
RUN ln -s /usr/local/lib/libmbus.so.0 /usr/lib/libmbus.so.0
RUN chmod +s /usr/local/bin/mbus*

# Install nodejs app

# Defines our working directory in container
WORKDIR /usr/src/app

# Copies the package.json first for better cache on later pushes
COPY package.json package.json

# This install npm dependencies on the balena build server,
# making sure to clean up the artifacts it creates in order to reduce the image size.
RUN JOBS=MAX npm install --production --unsafe-perm && npm cache verify && rm -rf /tmp/*

# This will copy all files in our root to the working  directory in the container
COPY . ./

# will run when container starts up on the device
CMD ["bash", "./init.sh"]

Obiously the only interresting part are the two first lines, because it fails at the second one, not matter what it is.
Here is the full DEBUG=1 output:

scott@scott-asus-pop:~/Documents/my-private-path$ balena deploy nt1-mbus --logs --source . --emulated --build
[debug] original argv0="node" argv=[/home/scott/.nvm/versions/node/v12.14.1/bin/node,/home/scott/.nvm/versions/node/v12.14.1/bin/balena,deploy,nt1-mbus,--logs,--source,.,--emulated,--build] length=9
[Debug]   Parsing input...
[Debug]   Loading project...
[Debug]   Resolving project...
[Debug]   Failed to resolve project: 
[Debug]   AggregateError of:
[Debug]       Error: ENOENT: no such file or directory, open '/home/scott/Documents/my-private-path/docker-compose.yml'
[Debug]       Error: ENOENT: no such file or directory, open '/home/scott/Documents/my-private-path/docker-compose.yaml'
[Info]    Creating default composition with source: /home/scott/Documents/my-private-path
[Debug]   Creating project...
[Info]    Building for rpi/raspberry-pi
[Build]   Building services...
[Build]   main Preparing...
[Info]    Emulation is enabled
[Debug]   Found build tasks:
[Debug]       main: build [.]
[Debug]   Resolving services with [raspberry-pi|rpi]
[Debug]   Found project types:
[Debug]       main: Dockerfile.template
[Debug]   Prepared tasks; building...
[Build]   main Step 1/18 : FROM balenalib/raspberry-pi-node:10-stretch-build
[Build]   main  ---> fb569da22fe2
[Build]   main Step 2/18 : COPY [".balena/qemu-execve","/tmp/qemu-execve"]
[Build]   main  ---> Using cache
[Build]   main  ---> cdcf042e15f0
[Build]   Built 1 service in 0:01
[Error]   Deploy failed
The command '/tmp/qemu-execve -execve /bin/sh -c install_packages python sqlite3 libsqlite3-dev build-essential curl tar libtool automake' returned a non-zero code: 1
Error: The command '/tmp/qemu-execve -execve /bin/sh -c install_packages python sqlite3 libsqlite3-dev build-essential curl tar libtool automake' returned a non-zero code: 1
    at Stream.<anonymous> (/home/scott/.nvm/versions/node/v12.14.1/lib/node_modules/balena-cli/node_modules/resin-docker-build/build/builder.js:181:23)
    at Stream.stream.write (/home/scott/.nvm/versions/node/v12.14.1/lib/node_modules/balena-cli/node_modules/through/index.js:26:11)
    at Stream.ondata (internal/streams/legacy.js:17:31)
    at Stream.emit (events.js:223:5)
    at Stream.EventEmitter.emit (domain.js:475:20)
    at drain (/home/scott/.nvm/versions/node/v12.14.1/lib/node_modules/balena-cli/node_modules/through/index.js:36:16)
    at Stream.stream.queue.stream.push (/home/scott/.nvm/versions/node/v12.14.1/lib/node_modules/balena-cli/node_modules/through/index.js:45:5)
    at Parser.parser.onToken (/home/scott/.nvm/versions/node/v12.14.1/lib/node_modules/balena-cli/node_modules/JSONStream/index.js:132:18)
    at Parser.proto.write (/home/scott/.nvm/versions/node/v12.14.1/lib/node_modules/balena-cli/node_modules/jsonparse/jsonparse.js:135:34)
    at Stream.<anonymous> (/home/scott/.nvm/versions/node/v12.14.1/lib/node_modules/balena-cli/node_modules/JSONStream/index.js:23:12)
    at Stream.stream.write (/home/scott/.nvm/versions/node/v12.14.1/lib/node_modules/balena-cli/node_modules/through/index.js:26:11)
    at IncomingMessage.ondata (_stream_readable.js:728:22)
    at IncomingMessage.emit (events.js:223:5)
    at IncomingMessage.EventEmitter.emit (domain.js:475:20)
    at addChunk (_stream_readable.js:309:12)
    at readableAddChunk (_stream_readable.js:290:11)
    at IncomingMessage.Readable.push (_stream_readable.js:224:10)
    at HTTPParser.parserOnBody (_http_common.js:128:24)
    at Socket.socketOnData (_http_client.js:456:22)
    at Socket.emit (events.js:223:5)
    at Socket.EventEmitter.emit (domain.js:475:20)
    at addChunk (_stream_readable.js:309:12)
    at readableAddChunk (_stream_readable.js:290:11)
    at Socket.Readable.push (_stream_readable.js:224:10)
    at Pipe.onStreamRead (internal/stream_base_commons.js:181:23)
    at /home/scott/.nvm/versions/node/v12.14.1/lib/node_modules/balena-cli/node_modules/resin-docker-build/build/builder.js:73:20
    at Builder.createBuildStream (/home/scott/.nvm/versions/node/v12.14.1/lib/node_modules/balena-cli/node_modules/resin-docker-build/build/builder.js:72:97)
    at /home/scott/.nvm/versions/node/v12.14.1/lib/node_modules/balena-cli/node_modules/resin-multibuild/build/build.js:82:17
    at Object.runBuildTask (/home/scott/.nvm/versions/node/v12.14.1/lib/node_modules/balena-cli/node_modules/resin-multibuild/build/build.js:59:12)
    at /home/scott/.nvm/versions/node/v12.14.1/lib/node_modules/balena-cli/node_modules/resin-multibuild/build/index.js:157:34
    at Generator.next (<anonymous>)
    at /home/scott/.nvm/versions/node/v12.14.1/lib/node_modules/balena-cli/node_modules/resin-multibuild/build/index.js:8:71
    at new Promise (<anonymous>)
    at __awaiter (/home/scott/.nvm/versions/node/v12.14.1/lib/node_modules/balena-cli/node_modules/resin-multibuild/build/index.js:4:12)
    at performSingleBuild (/home/scott/.nvm/versions/node/v12.14.1/lib/node_modules/balena-cli/node_modules/resin-multibuild/build/index.js:155:12)
    at /home/scott/.nvm/versions/node/v12.14.1/lib/node_modules/balena-cli/node_modules/resin-multibuild/build/index.js:140:20
    at processImmediate (internal/timers.js:439:21)
    at process.topLevelDomainCallback (domain.js:130:23)
    at Object.<anonymous> (/home/scott/.nvm/versions/node/v12.14.1/lib/node_modules/balena-cli/node_modules/resin-multibuild/build/index.js:139:39)
    at Generator.next (<anonymous>)
    at fulfilled (/home/scott/.nvm/versions/node/v12.14.1/lib/node_modules/balena-cli/node_modules/resin-multibuild/build/index.js:5:58)
From previous event:
    at runCommand (/home/scott/.nvm/versions/node/v12.14.1/lib/node_modules/balena-cli/build/app-capitano.js:181:14)
    at Object.exports.run (/home/scott/.nvm/versions/node/v12.14.1/lib/node_modules/balena-cli/build/app-capitano.js:193:39)
    at routeCliFramework (/home/scott/.nvm/versions/node/v12.14.1/lib/node_modules/balena-cli/build/preparser.js:39:79)
    at async Object.run (/home/scott/.nvm/versions/node/v12.14.1/lib/node_modules/balena-cli/build/app.js:10:5)

If you need help, don't hesitate in contacting our support forums at
https://forums.balena.io

For CLI bug reports or feature requests, have a look at the GitHub issues or
create a new one at: https://github.com/balena-io/balena-cli/issues/

(node:16210) [DEP0066] DeprecationWarning: OutgoingMessage.prototype._headers is deprecated

The only real error that is always there witht the Balena Base Images is exec user process caused "exec format error", but it seems to be quite generic and Googling it around did not give me any clues.

My OpenBalena is self hosted on DigitalOcean and has never had this kind of problem.
I dev on Pop!_OS 20.04 LTS, with freshly installed Docker from the repositories.
Balena cli version: 11.23.0
Node version: v12.14.1

Please help, I can’t update any of my devices at the moment ^^
I can provide any extra info you need.

Thanks in advance
Tim Winkelmann

alanb128 · November 18, 2020, 2:42am

Hello, it looks like you may be running up against this issue: https://github.com/balena-io/balena-cli/issues/1788
Can you try either pinning the base image to a previous version such as adding -20200327 to the end of the FROM line or upgrade your CLI to v11.31.26 or higher (but likely no higher than v12.2.2 - see https://github.com/balena-io/balena-cli/issues/1998)

Winkelmann · November 18, 2020, 9:11am

Hey,

Thank you so much for the answer. I’m a little sad I didn’t find that myself and had to bother you with a forum post ^^
I’ll test later today the proposed solutions, but it is very likely it will fix my issue.

While I’m at it, the only thing that worries me with OpenBalena is how to update it safely. How should I go about updating to v3.1.1 now that it’s out ?

Thanks in advance
Tim

richbayliss · November 18, 2020, 2:14pm

No worries.

It should be a case of pulling down the latest version from the repo and re-running the ./scripts/compose up step to update the running containers.

Winkelmann · November 18, 2020, 2:25pm

Hey,

Okay sounds logic. But isn’t there any concern about compatibility with the supervisors on my devices ?

Tim

dfunckt · November 18, 2020, 2:42pm

Older devices are not guaranteed to work with version 3.0.0. This thread has lots of info: Upgrading from v2.x.x to v3.x.x

Topic		Replies	Views
github actions: standard_init_linux.go:211: exec user process caused "exec format error" balenaOS raspberrypi3	5	3317	March 4, 2020
Standard_init_linux.go:211: exec user process caused "exec format error" balenaOS	2	13687	July 23, 2019
Image build failure log details Product support	10	457	January 19, 2020
Cross Build Error During Balena Push balenaOS	13	783	August 25, 2020
Docker build-arg for BALENA_MACHINE_NAME instead of using Dockerfile.template Product support dockerfile	3	1673	December 23, 2021

Cannot build anything with Balena Base Images | exec user process caused "exec format error"

Related topics