Balena-cli, balena-engine and SecretRemovalError

I am trying to build an image by use of balena cli.

The following works as expected when docker socket is used:

balena build --deviceType intel-nuc --arch amd64 --docker ‘/var/run/docker.sock’

The following does not work as expected when using balena-engine socket:

DEBUG=1 balena build --deviceType intel-nuc --arch amd64 --docker ‘/var/run/balena-engine.sock’

The above throw during the build step building and produce the following error output:

SecretRemovalError: read ECONNRESET
at Object. (/snapshot/versioned-source/node_modules/resin-multibuild/build/index.js:124:19)
at Generator.throw ()
at rejected (/snapshot/versioned-source/node_modules/resin-multibuild/build/index.js:5:65)

This seems to be related to Pull access denied error and [Error] Deploy failed where it is described that downgrade of balena-cli solves this issue.

I am trying to use the balena engine to add secrets, hence downgrade seems to be counter productive.

I am using balena-cli 11.9.7 and balena-engine Docker version v18.9.7, build 87ef8e0dca.

Git issue balena build: SecretRemovalError: The command '/bin/sh -c apk update && apk add jq' returned a non-zero code: 1 · Issue #1355 · balena-io/balena-cli · GitHub states that this issue is solved in balena-cli 11.9.0 which is not the case.

Anyone got an idea to get forward from this?

Hello and thanks for reporting this, I raised this internally with the CLI maintainer, will update you as soon as we figure out what may cause this

Hello again @aliasbits can you try running the same command with the --emulated flag and let us know if you are still getting the same error?

@aliasbits, I have tested the command line you mentioned with CLI v11.9.7 (standalone zip package), but I have not got any errors:

$ balena build --deviceType intel-nuc --arch amd64 --docker /var/run/balena-engine.sock
[Info]    Creating default composition with source: /mnt/data/balena-idling
[Info]    Building for amd64/intel-nuc
[Build]   Built 1 service in 0:16
[Build]   main Image size: 41.15 MB
[Success] Build succeeded!

Is the issue you are having consistent, or intermittent? The SecretRemovalError error can have several underlying the causes. The issues you have linked, and the output you have shared, seem to have different causes:

In order to find out where the ECONNRESET error you are getting is coming from, it would be helpful to be able to reproduce the error. Perhaps you could share a bit more about your “environment” (machine where you’re running the CLI, how you have installed the CLI, OS version, balenaOS version, if you’re using a VirtualBox VM, etc) and a sample Dockerfile or docker-compose.yml file that reproduces the error.

By the way, I think that the --docker /var/run/balena-engine.sock option you are using is not a common workflow; I imagine you’re either:

  • Running balenaEngine on your non-balenaOS development machine (laptop or desktop, e.g. running Ubuntu Linux), or
  • Running balena CLI in a container on balenaOS with balenaEngine.

I think neither option above is a common workflow, but I was using the second option to try to use the command line you shared:

# on a balenaOS host OS prompt on a NUC device or VirtualBox VM:
$ mkdir /mnt/data/cli
$ cd /mnt/data/cli
$ curl -LO https://github.com/balena-io/balena-cli/releases/download/v11.9.7/balena-cli-v11.9.7-linux-x64-standalone.zip
$ unzip balena-cli-v11.9.7-linux-x64-standalone.zip
$ mv balena-cli balena-cli-11.9.7

# now put the device in local mode (https://www.balena.io/docs/learn/develop/local-mode/)
# and create and run an ubuntu:bionic container (on balenaOS)
$ balena-engine run -it -v /mnt/data:/mnt/data -v /var/run/balena-engine.sock:/var/run/balena-engine.sock --network host ubuntu:bionic /bin/bash

# then inside the ubuntu:bionic container powered by balenaEngine/balenaOS
$ cd /mnt/data/
$ git clone https://github.com/balena-io-projects/balena-idling
$ cd balena-idling
$ /mnt/data/cli/balena-cli-11.9.7/balena build --deviceType intel-nuc --arch amd64 --docker /var/run/balena-engine.sock
[Info]    Creating default composition with source: /mnt/data/balena-idling
[Info]    Building for amd64/intel-nuc
[Build]   Built 1 service in 0:16
[Build]   main Image size: 41.15 MB
[Success] Build succeeded!

I’ve got “build succeeded” using both a “real” NUC computer (running balenaOS 2.32.0+rev4) and also running the intel-nuc image on a VirtualBox VM with balenaOS 2.41.1+rev1.

A more common workflow would be to run balenaCLI on your development machine (Mac, Windows or Linux) and select a remote balenaEngine daemon with the --dockerHost and --dockerPort command-line options (where “remote” could be a development/lab device like a NUC computer on the local network, or a VirtualBox VM running an intel-nuc balenaOS image). In theory however the results should be similar.

Barebone installation:
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION=“Ubuntu 18.04.1 LTS”

I have installed the balena-cli and balena-engine from standalone packages

--emulated error

Error: Cannot install emulator for architecture intel-nuc
at balenaArchToQemuArch (/snapshot/versioned-source/build/utils/qemu.js:112:13)
at exports.installQemu (/snapshot/versioned-source/build/utils/qemu.js:83:18)
at Promise.module.exports.Promise._execute (/snapshot/versioned-source/node_modules/bluebird/js/release/debuggability.js:313:9)
at Promise.module.exports.Promise._resolveFromExecutor (/snapshot/versioned-source/node_modules/bluebird/js/release/promise.js:488:18)
at new Promise (/snapshot/versioned-source/node_modules/bluebird/js/release/promise.js:79:10)
at /snapshot/versioned-source/build/utils/qemu.js:80:12
at tryCatcher (/snapshot/versioned-source/node_modules/bluebird/js/release/util.js:16:23)
at Promise.module.exports.Promise._settlePromiseFromHandler (/snapshot/versioned-source/node_modules/bluebird/js/release/promise.js:517:31)
at Promise.module.exports.Promise._settlePromise (/snapshot/versioned-source/node_modules/bluebird/js/release/promise.js:574:18)
at Promise.module.exports.Promise._settlePromise0 (/snapshot/versioned-source/node_modules/bluebird/js/release/promise.js:619:10)
at Promise.module.exports.Promise._settlePromises (/snapshot/versioned-source/node_modules/bluebird/js/release/promise.js:699:18)
at _drainQueueStep (/snapshot/versioned-source/node_modules/bluebird/js/release/async.js:138:12)
at _drainQueue (/snapshot/versioned-source/node_modules/bluebird/js/release/async.js:131:9)
at Async._drainQueues (/snapshot/versioned-source/node_modules/bluebird/js/release/async.js:147:5)
at Immediate.Async.drainQueues [as _onImmediate] (/snapshot/versioned-source/node_modules/bluebird/js/release/async.js:17:14)
at runCallback (timers.js:705:18)
at tryOnImmediate (timers.js:676:5)
at processImmediate (timers.js:658:5)
at process.topLevelDomainCallback (domain.js:120:23)

Tried to reduce complexity and executed balena-engine only on a simple Dockerfile which may hint about the issue since this produce “unexpected EOF”.

The output are as follows:

balena-engine build --file Dockerfile .
Sending build context to Docker daemon 50.18kB
Step 1/2 : FROM balenalib/intel-nuc-ubuntu:bionic
—> f111f749c49e
Step 2/2 : RUN apt-get update && apt-get install -y --no-install-recommends wget
—> Running in c1de4ef01e58
Get:1 Index of /ubuntu bionic-security InRelease [88.7 kB]
Get:2 Index of /ubuntu bionic InRelease [242 kB]
Get:3 Index of /ubuntu bionic-updates InRelease [88.7 kB]
Get:4 Index of /ubuntu bionic-backports InRelease [74.6 kB]
Get:5 Index of /ubuntu bionic-security/main amd64 Packages [649 kB]
Get:6 Index of /ubuntu bionic-security/multiverse amd64 Packages [4952 B]
Get:7 Index of /ubuntu bionic/multiverse amd64 Packages [186 kB]
Get:8 Index of /ubuntu bionic/restricted amd64 Packages [13.5 kB]
Get:9 Index of /ubuntu bionic-security/universe amd64 Packages [766 kB]
Get:10 Index of /ubuntu bionic-security/restricted amd64 Packages [8385 B]
Get:11 Index of /ubuntu bionic/main amd64 Packages [1344 kB]
Get:12 Index of /ubuntu bionic/universe amd64 Packages [11.3 MB]
Get:13 Index of /ubuntu bionic-updates/main amd64 Packages [946 kB]
Get:14 Index of /ubuntu bionic-updates/restricted amd64 Packages [19.0 kB]
Get:15 Index of /ubuntu bionic-updates/universe amd64 Packages [1284 kB]
Get:16 Index of /ubuntu bionic-updates/multiverse amd64 Packages [7998 B]
Get:17 Index of /ubuntu bionic-backports/main amd64 Packages [2496 B]
Get:18 Index of /ubuntu bionic-backports/universe amd64 Packages [4227 B]
Fetched 17.1 MB in 4s (4577 kB/s)
Reading package lists…
Reading package lists…
Building dependency tree…
Reading state information…
The following NEW packages will be installed:
wget
0 upgraded, 1 newly installed, 0 to remove and 20 not upgraded.
Need to get 316 kB of archives.
After this operation, 954 kB of additional disk space will be used.
Get:1 Index of /ubuntu bionic-updates/main amd64 wget amd64 1.19.4-1ubuntu2.2 [316 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 316 kB in 0s (1370 kB/s)
Selecting previously unselected package wget.
(Reading database … 7269 files and directories currently installed.)
Preparing to unpack …/wget_1.19.4-1ubuntu2.2_amd64.deb …
Unpacking wget (1.19.4-1ubuntu2.2) …
Setting up wget (1.19.4-1ubuntu2.2) …
unexpected EOF

balena-engine-eof.log (74.4 KB)

NOTE: I got a ubuntu 16.04.6, using balena-cli 11.9.3 and balena-engine 17.12.0 which works as expected.

I will try downgrade balena-engine to 17.12.0.

I downgraded balena-engine to 17.12.0 producing an image as expected.

The following is working:

balena build --deviceType intel-nuc --arch amd64 --docker ‘/var/run/balena-engine.sock’

Using:

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION=“Ubuntu 18.04.1 LTS”
balena-cli 11.9.7
balena-engine version v17.12.0, build 3fdfd0d

The following is not working, producing ‘unexpected EOF’:

See Balena-cli, balena-engine and SecretRemovalError - #9 by aliasbits with balena-engine-eof.log for balena-engine 18.9.7

balena build --deviceType intel-nuc --arch amd64 --docker ‘/var/run/balena-engine.sock’

Using:

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION=“Ubuntu 18.04.1 LTS”
balena-cli 11.9.7
balena-engine Docker version v18.9.7, build 87ef8e0dca

May be related to https://github.com/moby/moby/pull/37771. Describes some tar split/copy issue in linked https://github.com/moby/moby/issues/37581#issuecomment-417791662

Hi @aliasbits you’re hitting another manifestation of the libc issue with the latest github release. This time however there is no “in-flight” fix for it :confused:

I’m preparing a test release for v18.9.10 that you could try to run if you want?

Just ping, when ready

You can have a look at https://github.com/balena-os/balena-engine/releases/tag/v18.9.10

Docker version v18.9.10, build 7cb464a406-unsupported is working as expected.

Will non-balenaOS use of balena-cli and balena-engine be deprecated at some point?

@pdcastro hint that I am using a less common workflow. From my understanding they should both be usable in a non-balenaOS use case.

I’m pretty sure balena-cli has no restriction on being used on balenaOS as it’s the primary tool used by people to interact with their devices. And balena-engine is basically moby/docker with patches to make it more suitable for the environment we run it in.

The current lag when it comes to engine releases is due to the process still being a manual one (which is something that’s in the pipeline and due to change)
I’m trying to fix these things as time allows, I myself run it on non-balenaOS devices, so please report any strange things that should be working but aren’t here or directly on the issue tracker

Thanks @robertgzr, @pdcastro for the quick response

I will use the 18.9.10 test release until an official release arrives