Deploy failed when trying to deploy on OpenBalena

pgiotis · April 24, 2020, 2:26pm

Hi i am new in open balena.
Have already use Balena cloud with success but now i am trying to play with open balena too.
Although seems that open balena is successfully installed on server and i have already connect a device on it, i have problem with app deployment process.
Trying to execute the follow command in order to deploy a test app on the device
balena deploy testApp --emulated --logs --build

The deployment failed with the error bellow.

/balena-node-hello-world$ balena deploy testApp --emulated --logs --build
[debug] original argv0="node" argv=[/usr/local/bin/node,/usr/local/bin/balena,deploy,testApp,--emulated,--logs,--build] length=7
[Debug]   Parsing input...
[Debug]   Loading project...
[Debug]   Resolving project...
[Info]    No "docker-compose.yml" file found at "/balena-node-hello-world"
[Info]    Creating default composition with source: "/balena-node-hello-world"
[Debug]   Creating project...
[Info]    Building for armv7hf/raspberrypi3
[Build]   Building services...
[Build]   main Preparing...
[Info]    Emulation is enabled
[Debug]   Found build tasks:
[Debug]       main: build [.]
[Debug]   Resolving services with [raspberrypi3|armv7hf]
[Debug]   Found project types:
[Debug]       main: Dockerfile.template
[Debug]   Prepared tasks; building...
[Build]   main Step 1/8 : FROM balenalib/raspberrypi3-node:10-stretch-run
[Build]   main  ---> 5717ce158267
[Build]   main Step 2/8 : COPY [".balena/qemu-execve","/tmp/qemu-execve"]
[Build]   main  ---> Using cache
[Build]   main  ---> ebc6e7b17006
[Build]   main Step 3/8 : WORKDIR /usr/src/app
[Build]   main  ---> Using cache
[Build]   main  ---> cbd20279c04e
[Build]   main Step 4/8 : COPY ["package.json","package.json"]
[Build]   main  ---> Using cache
[Build]   main  ---> 21063578413b
[Build]   Built 1 service in 0:03
[Error]   Deploy failed
The command '/tmp/qemu-execve -execve /bin/sh -c JOBS=MAX npm install --production --unsafe-perm && npm cache verify && rm -rf /tmp/*' returned a non-zero code: 1
Error: The command '/tmp/qemu-execve -execve /bin/sh -c JOBS=MAX npm install --production --unsafe-perm && npm cache verify && rm -rf /tmp/*' returned a non-zero code: 1
    at Stream.<anonymous> (/usr/local/lib/node_modules/balena-cli/node_modules/resin-docker-build/build/builder.js:181:23)
    at Stream.stream.write (/usr/local/lib/node_modules/balena-cli/node_modules/through/index.js:26:11)
    at Stream.ondata (internal/streams/legacy.js:15:31)
    at Stream.emit (events.js:189:13)
    at Stream.EventEmitter.emit (domain.js:441:20)
    at drain (/usr/local/lib/node_modules/balena-cli/node_modules/through/index.js:36:16)
    at Stream.stream.queue.stream.push (/usr/local/lib/node_modules/balena-cli/node_modules/through/index.js:45:5)
    at Parser.parser.onToken (/usr/local/lib/node_modules/balena-cli/node_modules/JSONStream/index.js:132:18)
    at Parser.proto.write (/usr/local/lib/node_modules/balena-cli/node_modules/jsonparse/jsonparse.js:135:34)
    at Stream.<anonymous> (/usr/local/lib/node_modules/balena-cli/node_modules/JSONStream/index.js:23:12)
    at Stream.stream.write (/usr/local/lib/node_modules/balena-cli/node_modules/through/index.js:26:11)
    at IncomingMessage.ondata (_stream_readable.js:689:20)
    at IncomingMessage.emit (events.js:189:13)
    at IncomingMessage.EventEmitter.emit (domain.js:441:20)
    at addChunk (_stream_readable.js:284:12)
    at readableAddChunk (_stream_readable.js:265:11)
    at Bluebird.try.then (/usr/local/lib/node_modules/balena-cli/node_modules/resin-docker-build/build/builder.js:73:20)
    at runCallback (timers.js:705:18)
    at tryOnImmediate (timers.js:676:5)
    at processImmediate (timers.js:658:5)
    at process.topLevelDomainCallback (domain.js:120:23)
    at Builder.createBuildStream (/usr/local/lib/node_modules/balena-cli/node_modules/resin-docker-build/build/builder.js:72:97)
    at Promise (/usr/local/lib/node_modules/balena-cli/node_modules/resin-multibuild/build/build.js:82:17)
    at Object.runBuildTask (/usr/local/lib/node_modules/balena-cli/node_modules/resin-multibuild/build/build.js:59:12)
    at /usr/local/lib/node_modules/balena-cli/node_modules/resin-multibuild/build/index.js:158:34
    at Generator.next (<anonymous>)
    at /usr/local/lib/node_modules/balena-cli/node_modules/resin-multibuild/build/index.js:8:71
    at new Promise (<anonymous>)
    at __awaiter (/usr/local/lib/node_modules/balena-cli/node_modules/resin-multibuild/build/index.js:4:12)
    at performSingleBuild (/usr/local/lib/node_modules/balena-cli/node_modules/resin-multibuild/build/index.js:156:12)
    at Bluebird.map (/usr/local/lib/node_modules/balena-cli/node_modules/resin-multibuild/build/index.js:141:20)
    at runCallback (timers.js:705:18)
    at tryOnImmediate (timers.js:676:5)
    at Object.<anonymous> (/usr/local/lib/node_modules/balena-cli/node_modules/resin-multibuild/build/index.js:140:39)
    at Generator.next (<anonymous>)
    at fulfilled (/usr/local/lib/node_modules/balena-cli/node_modules/resin-multibuild/build/index.js:5:58)
From previous event:
    at runCommand (/usr/local/lib/node_modules/balena-cli/build/app-capitano.js:159:14)
    at Object.exports.run (/usr/local/lib/node_modules/balena-cli/build/app-capitano.js:171:39)
    at routeCliFramework (/usr/local/lib/node_modules/balena-cli/build/preparser.js:39:79)
    at process._tickCallback (internal/process/next_tick.js:68:7)
    at Function.Module.runMain (internal/modules/cjs/loader.js:757:11)
    at startup (internal/bootstrap/node.js:283:19)
    at bootstrapNodeJSCore (internal/bootstrap/node.js:622:3)

Do you have ant idea what is going wrong?
Thanks in advance.

AlidaOdendaal · April 24, 2020, 2:59pm

Hi @pgiotis

Welcome to forums! First, can you please let us know what your local operating system is (where you’re running the balena deploy command)? Also, which version of balena-cli are you using, and how did you install it?

Kind regards
Alida

pgiotis · April 24, 2020, 4:17pm

Hello Alida,
i am running a Kubuntu 18.04 OS with balena-cli version 11.31.15 .
I use npm to install balena-cli with command
sudo npm install balena-cli -g --production --unsafe-perm

Kind regards,
Panagiotis

thgreasi · April 24, 2020, 5:51pm

Hi,
Since you have installed the balena-cli though npm, could you also clarify the node version that you are using so that we can fully document this.

I would first like to rule out whether this is open-balena, or just balena-cli related.
Could you first try using the balena-cli to just build that release locally and let us know whether it works or not?

You can do so by using the balena build command:

balena build --application testApp --emulated

Let me also point you to the respective documentation page:
https://www.balena.io/docs/reference/balena-cli/#build-source

Kind regards,
Thodoris

pgiotis · April 27, 2020, 10:37am

Hi Thodoris,
i am using node v10.15.3.
After running balena build --application testApp --emulated i am getting the same error as with deploy command.

Thanks,
Panagiotis

zvin · April 27, 2020, 11:45am

Hello,
It looks like the issue is with your Dockerfile.
Could you please try creating an application on balenaCloud and push your app here to see if our builders will fail in the same way ?
You could also try splitting the RUN npm install --production --unsafe-perm && npm cache verify && rm -rf /tmp/* line of your Dockerfile into 3 to see which one fails.

pgiotis · April 27, 2020, 11:52am

I have already push the app in Balena cloud and works.

zvin · April 27, 2020, 12:00pm

Can you please provide the output of balena push ?

pgiotis · April 27, 2020, 12:08pm

Balena push on Balena cloud

/balena-node-hello-world$ balena push testApp
[Info]     Starting build for testApp, user panagiotis_giotis
[Info]     Dashboard link: https://dashboard.balena-cloud.com/apps/1650167/devices
[Info]     Building on arm01
[Info]     Pulling previous images for caching purposes...
[Success]  Successfully pulled cache images
[main]     Step 1/7 : FROM balenalib/raspberrypi3-node:10-stretch-run
[main]      ---> 23ef66b82eec
[main]     Step 2/7 : WORKDIR /usr/src/app
[main]     Using cache
[main]      ---> 115d09b3c1a2
[main]     Step 3/7 : COPY package.json package.json
[main]     Using cache
[main]      ---> 8cb7d83cbaeb
[main]     Step 4/7 : RUN JOBS=MAX npm install --production --unsafe-perm && npm cache verify && rm -rf /tmp/*
[main]     Using cache
[main]      ---> 58b256d71866
[main]     Step 5/7 : COPY . ./
[main]     Using cache
[main]      ---> 0507a82addd5
[main]     Step 6/7 : ENV UDEV=1
[main]     Using cache
[main]      ---> effcbe5fdf97
[main]     Step 7/7 : CMD ["npm", "start"]
[main]     Using cache
[main]      ---> 7ccd90f2787e
[main]     Successfully built 7ccd90f2787e
[Info]     Uploading images
[Success]  Successfully uploaded images
[Info]     Built on arm01
[Success]  Release successfully created!
[Info]     Release: 9d6f3287a7dffdd889398047c5f4563a (id: 1358440)
[Info]     ┌─────────┬────────────┬────────────┐
[Info]     │ Service │ Image Size │ Build Time │
[Info]     ├─────────┼────────────┼────────────┤
[Info]     │ main    │ 206.42 MB  │ < 1 second │
[Info]     └─────────┴────────────┴────────────┘
[Info]     Build finished in 18 seconds

Balena build

/balena-node-hello-world$ balena build --application testApp --emulated
[debug] original argv0="node" argv=[/usr/local/bin/node,/usr/local/bin/balena,build,--application,testApp,--emulated] length=6
[Debug]   Parsing input...
[Debug]   Loading project...
[Debug]   Resolving project...
[Info]    No "docker-compose.yml" file found at "/balena-node-hello-world"
[Info]    Creating default composition with source: "/balena-node-hello-world"
[Debug]   Creating project...
[Info]    Building for armv7hf/raspberrypi3
[Info]    Emulation is enabled
[Debug]   Found build tasks:.. |
[Debug]       main: build [.]
[Debug]   Resolving services with [raspberrypi3|armv7hf]
[Debug]   Found project types:
[Debug]       main: Dockerfile.template
[Debug]   Prepared tasks; building...
[Build]   Built 1 service in 0:01
[Build]   main [==========>          ]  50% Step 4/8:  ---> cb894e72007f
[Error]   Build failed
The command '/tmp/qemu-execve -execve /bin/sh -c JOBS=MAX npm install --production --unsafe-perm && npm cache verify && rm -rf /tmp/*' returned a non-zero code: 1
Error: The command '/tmp/qemu-execve -execve /bin/sh -c JOBS=MAX npm install --production --unsafe-perm && npm cache verify && rm -rf /tmp/*' returned a non-zero code: 1
    at Stream.<anonymous> (/usr/local/lib/node_modules/balena-cli/node_modules/resin-docker-build/build/builder.js:181:23)
    at Stream.stream.write (/usr/local/lib/node_modules/balena-cli/node_modules/through/index.js:26:11)
    at Stream.ondata (internal/streams/legacy.js:15:31)
    at Stream.emit (events.js:189:13)
    at Stream.EventEmitter.emit (domain.js:441:20)
    at drain (/usr/local/lib/node_modules/balena-cli/node_modules/through/index.js:36:16)
    at Stream.stream.queue.stream.push (/usr/local/lib/node_modules/balena-cli/node_modules/through/index.js:45:5)
    at Parser.parser.onToken (/usr/local/lib/node_modules/balena-cli/node_modules/JSONStream/index.js:132:18)
    at Parser.proto.write (/usr/local/lib/node_modules/balena-cli/node_modules/jsonparse/jsonparse.js:135:34)
    at Stream.<anonymous> (/usr/local/lib/node_modules/balena-cli/node_modules/JSONStream/index.js:23:12)
    at Stream.stream.write (/usr/local/lib/node_modules/balena-cli/node_modules/through/index.js:26:11)
    at IncomingMessage.ondata (_stream_readable.js:689:20)
    at IncomingMessage.emit (events.js:189:13)
    at IncomingMessage.EventEmitter.emit (domain.js:441:20)
    at addChunk (_stream_readable.js:284:12)
    at readableAddChunk (_stream_readable.js:265:11)
    at Bluebird.try.then (/usr/local/lib/node_modules/balena-cli/node_modules/resin-docker-build/build/builder.js:73:20)
    at runCallback (timers.js:705:18)
    at tryOnImmediate (timers.js:676:5)
    at processImmediate (timers.js:658:5)
    at process.topLevelDomainCallback (domain.js:120:23)
    at Builder.createBuildStream (/usr/local/lib/node_modules/balena-cli/node_modules/resin-docker-build/build/builder.js:72:97)
    at Promise (/usr/local/lib/node_modules/balena-cli/node_modules/resin-multibuild/build/build.js:82:17)
    at Object.runBuildTask (/usr/local/lib/node_modules/balena-cli/node_modules/resin-multibuild/build/build.js:59:12)
    at /usr/local/lib/node_modules/balena-cli/node_modules/resin-multibuild/build/index.js:158:34
    at Generator.next (<anonymous>)
    at /usr/local/lib/node_modules/balena-cli/node_modules/resin-multibuild/build/index.js:8:71
    at new Promise (<anonymous>)
    at __awaiter (/usr/local/lib/node_modules/balena-cli/node_modules/resin-multibuild/build/index.js:4:12)
    at performSingleBuild (/usr/local/lib/node_modules/balena-cli/node_modules/resin-multibuild/build/index.js:156:12)
    at Bluebird.map (/usr/local/lib/node_modules/balena-cli/node_modules/resin-multibuild/build/index.js:141:20)
    at runCallback (timers.js:705:18)
    at tryOnImmediate (timers.js:676:5)
    at Object.<anonymous> (/usr/local/lib/node_modules/balena-cli/node_modules/resin-multibuild/build/index.js:140:39)
    at Generator.next (<anonymous>)
    at fulfilled (/usr/local/lib/node_modules/balena-cli/node_modules/resin-multibuild/build/index.js:5:58)
From previous event:
    at runCommand (/usr/local/lib/node_modules/balena-cli/build/app-capitano.js:159:14)
    at Object.exports.run (/usr/local/lib/node_modules/balena-cli/build/app-capitano.js:171:39)
    at routeCliFramework (/usr/local/lib/node_modules/balena-cli/build/preparser.js:39:79)
    at process._tickCallback (internal/process/next_tick.js:68:7)
    at Function.Module.runMain (internal/modules/cjs/loader.js:757:11)
    at startup (internal/bootstrap/node.js:283:19)
    at bootstrapNodeJSCore (internal/bootstrap/node.js:622:3)

zvin · April 27, 2020, 1:17pm

It looks like qemu is failing to run the npm i command on your computer.
Could you try changing your Dockerfile to see what is failing ?
For example try adding a RUN npm install lodash before the JOBS=MAX npm install --production --unsafe-perm && npm cache verify && rm -rf /tmp/* line in your Dockerfile and check if it fails.

pgiotis · April 28, 2020, 7:26am

Fails again.

Build]   Built 1 service in 0:01
[Build]   main [========>            ]  44% Step 4/9:  ---> cb894e72007f
[Error]   Build failed
The command '/tmp/qemu-execve -execve /bin/sh -c npm install lodash' returned a non-zero code: 1
Error: The command '/tmp/qemu-execve -execve /bin/sh -c npm install lodash' returned a non-zero code: 1
    at Stream.<anonymous> (/usr/local/lib/node_modules/balena-cli/node_modules/resin-docker-build/build/builder.js:181:23)
    at Stream.stream.write (/usr/local/lib/node_modules/balena-cli/node_modules/through/index.js:26:11)
    at Stream.ondata (internal/streams/legacy.js:15:31)
    at Stream.emit (events.js:189:13)

georgiats · April 28, 2020, 2:58pm

Hey Panagiotis, we are currently investigating the issue. We are going to share an update with you soon.

pdcastro · April 28, 2020, 4:22pm

@pgiotis, thank you for reporting the issue and sharing additional information about it. We have narrowed it down to changes in balenalib base images, and we are having an internal discussion on whether the fix belongs to the base images or to the CLI. Meanwhile, you can get around the issue by pinning the base image to a previous version:

The Dockerfile.template file currently contains this FROM line:

FROM balenalib/%%BALENA_MACHINE_NAME%%-node:10-stretch-run

Replace it with:

FROM balenalib/%%BALENA_MACHINE_NAME%%-node:10-stretch-run-20200327

It should then work. When the -20200327 suffix is not included, the latest image version is used, which is currently 20200420. Version 20200417 is also affected by the issue. For your information, a list of the versions is available on Dockerhub:
https://hub.docker.com/r/balenalib/raspberrypi3-node/tags?page=1&name=10-stretch-run

We will get back to you in this thread when we have more information about the issue. Thanks again for reporting it. Meanwhile, let us know if you have any other questions.

pgiotis · April 29, 2020, 8:55am

Hello thanks for the response.
I change the docker template with

FROM balenalib/%%BALENA_MACHINE_NAME%%-node:10-stretch-run-20200327

and the build complete successfully but when i am trying to deploy the build version to openbalena i get the error below

[Error]   Deploy failed
images.push is not iterable (cannot read property Symbol(Symbol.iterator))
TypeError: images.push is not iterable (cannot read property Symbol(Symbol.iterator))
    at authorizePush (/usr/local/lib/node_modules/balena-cli/build/utils/compose.js:397:12)
    at /usr/local/lib/node_modules/balena-cli/build/utils/compose.js:456:40
From previous event:
    at /usr/local/lib/node_modules/balena-cli/build/utils/compose.js:456:18
    at processImmediate (internal/timers.js:456:21)
    at process.topLevelDomainCallback (domain.js:137:15)
From previous event:
    at /usr/local/lib/node_modules/balena-cli/build/utils/compose.js:452:14
From previous event:
    at Object.deployProject (/usr/local/lib/node_modules/balena-cli/build/utils/compose.js:449:10)
    at /usr/local/lib/node_modules/balena-cli/build/actions/deploy.js:104:24
    at processImmediate (internal/timers.js:456:21)
    at process.topLevelDomainCallback (domain.js:137:15)
From previous event:
    at /usr/local/lib/node_modules/balena-cli/build/actions/deploy.js:103:22
From previous event:
    at /usr/local/lib/node_modules/balena-cli/build/actions/deploy.js:86:8
From previous event:
    at deployProject (/usr/local/lib/node_modules/balena-cli/build/actions/deploy.js:52:72)
    at /usr/local/lib/node_modules/balena-cli/build/actions/deploy.js:199:16
    at processImmediate (internal/timers.js:456:21)
    at process.topLevelDomainCallback (domain.js:137:15)
From previous event:
    at /usr/local/lib/node_modules/balena-cli/build/actions/deploy.js:198:22
From previous event:
    at Command.action (/usr/local/lib/node_modules/balena-cli/build/actions/deploy.js:195:8)
    at /usr/local/lib/node_modules/balena-cli/node_modules/capitano/build/command.js:98:37
    at runNextTicks (internal/process/task_queues.js:62:5)
    at processImmediate (internal/timers.js:429:9)
    at process.topLevelDomainCallback (domain.js:137:15)

any idea of what is going wrong?

pgiotis · April 29, 2020, 9:58am

I am downgrade balena cli to v11.31.17 and push the build successful.

pdcastro · April 29, 2020, 10:05am

Thank you for confirming it @pgiotis. It appears that issues with balena deploy were introduced in CLI versions 11.31.18 and 11.31.22, during codebase conversions away from CoffeeScript. One of them was fixed in v11.31.23, the other is still pending. Apologies for the trouble this has caused you. I’ll let you know in this thread when the other issue has been fixed as well.

pdcastro · April 29, 2020, 4:03pm

Panagiotis, the “images.push is not iterable” error you were getting has been fixed in CLI v11.31.24. Hopefully this will allow you to use the latest CLI version with openBalena. Thank you again for reporting the error and helping us resolve it.

Regarding the earlier base image issue, please continue to pin the base image to the 20200327 version, as work on that issue is ongoing. In case you are interested, it seems that the issue was down to the QEMU emulator, and we have a pull request under review: https://github.com/balena-io/qemu/pull/19/files
After we test the QEMU fix and if we confirm that it was indeed the root cause, we will then update the CLI and base images accordingly, and we will let you know in this thread. Thanks again.

pdcastro · April 30, 2020, 11:27am

@pgiotis, CLI release v11.31.26 includes a new version of the QEMU emulator (v4.0.0+balena2) that supports the latests balenalib base images. I have successfully tested the following commands:

$ balena build ~/balena-io-examples/balena-node-hello-world -A armv7hf -d raspberrypi3 --emulated --logs --nocache

$ balena deploy test-rpi --build --emulated --logs --nocache -s ~/balena-io-examples/balena-node-hello-world

Using the project’s default “unpinned” base image:

FROM balenalib/%%BALENA_MACHINE_NAME%%-node:10-stretch-run

By the way, pinning balenalib base images to specific “date” versions is a recommended best practice before going to production. See more at: https://www.balena.io/docs/learn/more/anti-patterns/#not-pinning-base-image-versions

Thank you again for reporting this issue and helping us resolve it.

pgiotis · May 5, 2020, 6:27am

@pdcastro thank you for your help we will be in touch.