Balena deploy does not work suddenly

Hello, balena team.

Balena deploy function was working well before, but now it does not work suddenly.

Here are the error logs.

balena deploy 5e890e9ba48676001409b048-23 --nocache --build
[debug] original argv0=“node” argv=[/usr/local/bin/node,/usr/local/bin/balena,deploy,5e890e9ba48676001409b048-23,–nocache,–build] length=6
[Debug] Parsing input…
[Debug] Loading project…
[Debug] Resolving project…
[Debug] docker-compose.yml file found at “/codebuild/output/src053687981/src”
[Debug] Creating project…
[Info] Building for amd64/intel-nuc
·[?25l·[0J·[2K
[Build] Building services… |
·[2K
[Build] kubeedge Preparing…
·[2A·[0J·[2K
[Build] Building services… /
·[2K
[Build] kubeedge Preparing…
·[2A·[0J·[2K
[Build] Building services… -
·[2K
[Build] kubeedge Preparing…
·[2A·[0J·[2K
[Build] Building services…
·[2K
[Build] kubeedge Preparing…
·[2A·[0J·[2K
[Build] Building services… |
·[2K
[Build] kubeedge Preparing…
·[2A·[0J·[2K
[Build] Building services… /
·[2K
[Build] kubeedge Preparing…
·[2A·[0J·[2K
[Build] Building services… -
·[2K
[Build] kubeedge Preparing…
·[2A·[0J·[2K
[Build] Building services…
·[2K
[Build] kubeedge Preparing…
·[2A·[0J·[2K
[Build] Building services… |
·[2K
[Build] kubeedge Preparing…
·[2A·[0J·[2K
[Build] Building services… /
·[2K
[Build] kubeedge Preparing…
·[2A[Debug] Found build tasks:
[Debug] kubeedge: build [./kubeedge]
[Debug] Resolving services with [intel-nuc|amd64]
[Debug] Found project types:
[Debug] kubeedge: Dockerfile.template
[Debug] Prepared tasks; building…
·[0J·[2K
[Build] Building services… -
·[2K
[Build] kubeedge Preparing…
·[2A·[0J·[2K
[Build] Building services…
·[2K
[Build] kubeedge Preparing…
·[2A·[0J·[2K
[Build] Building services… |
·[2K
[Build] kubeedge Preparing…
·[2A·[0J·[2K
[Build] Building services… /
·[2K
[Build] kubeedge Preparing…
·[2A·[0J·[2K

Build] kubeedge [=================> ] 87% Step 14/16: WORKDIR /usr/src/app
·[2A·[0J·[2K
[Build] Building services… -
·[2K
[Build] kubeedge [=================> ] 87% Step 14/16: WORKDIR /usr/src/app
·[2A·[0J·[2K
[Build] Building services…
·[2K
[Build] kubeedge [=================> ] 87% Step 14/16: —> Running in a230dff6e9bd
·[2A·[0J·[2K
[Build] Building services… |
·[2K
[Build] kubeedge [=================> ] 87% Step 14/16: —> aa361e885d2a
·[2A·[0J·[2K
[Build] Built 1 service in 1:12
·[2K
[Build] kubeedge Image size: 1.41 GB
·[?25h·[2K
[Info] Creating release… |
·[0A·[2K
[Info] Creating release… /
·[0A·[2K
[Info] Creating release… -
·[0A·[2K
[Info] Creating release…
·[0A·[2K
[Info] Creating release…
[Debug] Tagging images…
[Debug] Authorizing push…
[Debug] Untagging images…
·[2K
[Info] Saving release…
[Error] Deploy failed
images.push is not iterable (cannot read property Symbol(Symbol.iterator))
TypeError: images.push is not iterable (cannot read property Symbol(Symbol.iterator))
at authorizePush (/usr/local/lib/node_modules/balena-cli/build/utils/compose.js:397:12)
at /usr/local/lib/node_modules/balena-cli/build/utils/compose.js:458:40
From previous event:
at /usr/local/lib/node_modules/balena-cli/build/utils/compose.js:458:18
at processImmediate (internal/timers.js:456:21)
at process.topLevelDomainCallback (domain.js:137:15)
From previous event:
at /usr/local/lib/node_modules/balena-cli/build/utils/compose.js:454:14
From previous event:
at Object.deployProject (/usr/local/lib/node_modules/balena-cli/build/utils/compose.js:451:10)
at /usr/local/lib/node_modules/balena-cli/build/actions/deploy.js:104:24
at processImmediate (internal/timers.js:456:21)
at process.topLevelDomainCallback (domain.js:137:15)
From previous event:
at /usr/local/lib/node_modules/balena-cli/build/actions/deploy.js:103:22
From previous event:
at /usr/local/lib/node_modules/balena-cli/build/actions/deploy.js:86:8
From previous event:
at deployProject (/usr/local/lib/node_modules/balena-cli/build/actions/deploy.js:52:72)
at /usr/local/lib/node_modules/balena-cli/build/actions/deploy.js:199:16
at processImmediate (internal/timers.js:456:21)
at process.topLevelDomainCallback (domain.js:137:15)
From previous event:
at /usr/local/lib/node_modules/balena-cli/build/actions/deploy.js:198:22
From previous event:
at Command.action (/usr/local/lib/node_modules/balena-cli/build/actions/deploy.js:195:8)
at /usr/local/lib/node_modules/balena-cli/node_modules/capitano/build/command.js:98:37
at runNextTicks (internal/process/task_queues.js:62:5)
at processImmediate (internal/timers.js:429:9)
at process.topLevelDomainCallback (domain.js:137:15)

If you need help, don’t hesitate in contacting our support forums at
https://forums.balena.io

For CLI bug reports or feature requests, have a look at the GitHub issues or
create a new one at: Issues · balena-io/balena-cli · GitHub

Hello, which version of balena-cli are you using ? Have you updated it recently ?

Hi, @zvin
Current balena cli version is 11.31.22.
And I think, I was using 11.30.5 before.

Hi,
Can you also clarify the OS that you are using the balena-cli on and whether it’s an npm installation or a standalone one?
Since I see that this image is quite large, can you also try deploying a different test application and confirm whether that also errors?

Kind regards,
Thodoris

Hi, @thgreasi, @zvin
I installed balena-cli using npm.
I tried with 11.30.5 and now it works again.

Regards
Liu

Hi @l.tiesong,

Thanks for the extra details.
Could you also try a balena deploy with balena-cli v11.31.22 of a test application on a test starter application on balena-cloud?
This could help us rule out whether this might be an issue specific to your open-balena instance.

Kind regards,
Thodoris

I can confirm I’m getting the same issue which started a few hours ago. A balena deploy <app-name> --logs gives me these errors:

[Build]   Built 4 services in 0:05
[Info]    Creating release...
[Info]    Pushing images to registry...
This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason:
Error: tag does not exist: registry2.balena-cloud.com/v2/REDACTED:latest
    at Stream.<anonymous> (/home/linuxbrew/.linuxbrew/Cellar/balena-cli/11.31.22/libexec/lib/node_modules/balena-cli/node_modules/docker-progress/index.js:53:19)
    at Stream.emit (events.js:315:20)
    at Stream.EventEmitter.emit (domain.js:485:12)
    at drain (/home/linuxbrew/.linuxbrew/Cellar/balena-cli/11.31.22/libexec/lib/node_modules/balena-cli/node_modules/through/index.js:36:16)
    at Stream.stream.queue.stream.push (/home/linuxbrew/.linuxbrew/Cellar/balena-cli/11.31.22/libexec/lib/node_modules/balena-cli/node_modules/through/index.js:45:5)
    at Parser.parser.onToken (/home/linuxbrew/.linuxbrew/Cellar/balena-cli/11.31.22/libexec/lib/node_modules/balena-cli/node_modules/JSONStream/index.js:132:18)
    at Parser.proto.write (/home/linuxbrew/.linuxbrew/Cellar/balena-cli/11.31.22/libexec/lib/node_modules/balena-cli/node_modules/jsonparse/jsonparse.js:135:34)
    at Stream.<anonymous> (/home/linuxbrew/.linuxbrew/Cellar/balena-cli/11.31.22/libexec/lib/node_modules/balena-cli/node_modules/JSONStream/index.js:23:12)
    at Stream.stream.write (/home/linuxbrew/.linuxbrew/Cellar/balena-cli/11.31.22/libexec/lib/node_modules/balena-cli/node_modules/through/index.js:26:11)
    at IncomingMessage.ondata (_stream_readable.js:707:22)
    at IncomingMessage.emit (events.js:315:20)
    at IncomingMessage.EventEmitter.emit (domain.js:485:12)
    at addChunk (_stream_readable.js:296:12)
    at readableAddChunk (_stream_readable.js:272:9)
    at IncomingMessage.Readable.push (_stream_readable.js:213:10)
    at HTTPParser.parserOnBody (_http_common.js:130:24)
This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason:
Error: tag does not exist: registry2.balena-cloud.com/v2/REDACTED:latest
    at Stream.<anonymous> (/home/linuxbrew/.linuxbrew/Cellar/balena-cli/11.31.22/libexec/lib/node_modules/balena-cli/node_modules/docker-progress/index.js:53:19)
    at Stream.emit (events.js:315:20)
    at Stream.EventEmitter.emit (domain.js:485:12)
    at drain (/home/linuxbrew/.linuxbrew/Cellar/balena-cli/11.31.22/libexec/lib/node_modules/balena-cli/node_modules/through/index.js:36:16)
    at Stream.stream.queue.stream.push (/home/linuxbrew/.linuxbrew/Cellar/balena-cli/11.31.22/libexec/lib/node_modules/balena-cli/node_modules/through/index.js:45:5)
    at Parser.parser.onToken (/home/linuxbrew/.linuxbrew/Cellar/balena-cli/11.31.22/libexec/lib/node_modules/balena-cli/node_modules/JSONStream/index.js:132:18)
    at Parser.proto.write (/home/linuxbrew/.linuxbrew/Cellar/balena-cli/11.31.22/libexec/lib/node_modules/balena-cli/node_modules/jsonparse/jsonparse.js:135:34)
    at Stream.<anonymous> (/home/linuxbrew/.linuxbrew/Cellar/balena-cli/11.31.22/libexec/lib/node_modules/balena-cli/node_modules/JSONStream/index.js:23:12)
    at Stream.stream.write (/home/linuxbrew/.linuxbrew/Cellar/balena-cli/11.31.22/libexec/lib/node_modules/balena-cli/node_modules/through/index.js:26:11)
    at IncomingMessage.ondata (_stream_readable.js:707:22)
    at IncomingMessage.emit (events.js:315:20)
    at IncomingMessage.EventEmitter.emit (domain.js:485:12)
    at addChunk (_stream_readable.js:296:12)
    at readableAddChunk (_stream_readable.js:272:9)
    at IncomingMessage.Readable.push (_stream_readable.js:213:10)
    at HTTPParser.parserOnBody (_http_common.js:130:24)
This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason:
Error: tag does not exist: registry2.balena-cloud.com/v2/REDACTED:latest
    at Stream.<anonymous> (/home/linuxbrew/.linuxbrew/Cellar/balena-cli/11.31.22/libexec/lib/node_modules/balena-cli/node_modules/docker-progress/index.js:53:19)
    at Stream.emit (events.js:315:20)
    at Stream.EventEmitter.emit (domain.js:485:12)
    at drain (/home/linuxbrew/.linuxbrew/Cellar/balena-cli/11.31.22/libexec/lib/node_modules/balena-cli/node_modules/through/index.js:36:16)
    at Stream.stream.queue.stream.push (/home/linuxbrew/.linuxbrew/Cellar/balena-cli/11.31.22/libexec/lib/node_modules/balena-cli/node_modules/through/index.js:45:5)
    at Parser.parser.onToken (/home/linuxbrew/.linuxbrew/Cellar/balena-cli/11.31.22/libexec/lib/node_modules/balena-cli/node_modules/JSONStream/index.js:132:18)
    at Parser.proto.write (/home/linuxbrew/.linuxbrew/Cellar/balena-cli/11.31.22/libexec/lib/node_modules/balena-cli/node_modules/jsonparse/jsonparse.js:135:34)
    at Stream.<anonymous> (/home/linuxbrew/.linuxbrew/Cellar/balena-cli/11.31.22/libexec/lib/node_modules/balena-cli/node_modules/JSONStream/index.js:23:12)
    at Stream.stream.write (/home/linuxbrew/.linuxbrew/Cellar/balena-cli/11.31.22/libexec/lib/node_modules/balena-cli/node_modules/through/index.js:26:11)
    at IncomingMessage.ondata (_stream_readable.js:707:22)
    at IncomingMessage.emit (events.js:315:20)
    at IncomingMessage.EventEmitter.emit (domain.js:485:12)
    at addChunk (_stream_readable.js:296:12)
    at readableAddChunk (_stream_readable.js:272:9)
    at IncomingMessage.Readable.push (_stream_readable.js:213:10)
    at HTTPParser.parserOnBody (_http_common.js:130:24)
This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason:
Error: tag does not exist: registry2.balena-cloud.com/v2/REDACTED:latest
    at Stream.<anonymous> (/home/linuxbrew/.linuxbrew/Cellar/balena-cli/11.31.22/libexec/lib/node_modules/balena-cli/node_modules/docker-progress/index.js:53:19)
    at Stream.emit (events.js:315:20)
    at Stream.EventEmitter.emit (domain.js:485:12)
    at drain (/home/linuxbrew/.linuxbrew/Cellar/balena-cli/11.31.22/libexec/lib/node_modules/balena-cli/node_modules/through/index.js:36:16)
    at Stream.stream.queue.stream.push (/home/linuxbrew/.linuxbrew/Cellar/balena-cli/11.31.22/libexec/lib/node_modules/balena-cli/node_modules/through/index.js:45:5)
    at Parser.parser.onToken (/home/linuxbrew/.linuxbrew/Cellar/balena-cli/11.31.22/libexec/lib/node_modules/balena-cli/node_modules/JSONStream/index.js:132:18)
    at Parser.proto.write (/home/linuxbrew/.linuxbrew/Cellar/balena-cli/11.31.22/libexec/lib/node_modules/balena-cli/node_modules/jsonparse/jsonparse.js:135:34)
    at Stream.<anonymous> (/home/linuxbrew/.linuxbrew/Cellar/balena-cli/11.31.22/libexec/lib/node_modules/balena-cli/node_modules/JSONStream/index.js:23:12)
    at Stream.stream.write (/home/linuxbrew/.linuxbrew/Cellar/balena-cli/11.31.22/libexec/lib/node_modules/balena-cli/node_modules/through/index.js:26:11)
    at IncomingMessage.ondata (_stream_readable.js:707:22)
    at IncomingMessage.emit (events.js:315:20)
    at IncomingMessage.EventEmitter.emit (domain.js:485:12)
    at addChunk (_stream_readable.js:296:12)
    at readableAddChunk (_stream_readable.js:272:9)
    at IncomingMessage.Readable.push (_stream_readable.js:213:10)
    at HTTPParser.parserOnBody (_http_common.js:130:24)
[Info]    Saving release...
[Success] Deploy succeeded!
[Success] Release: REDACTED

                            \
                             \
                              \\
                               \\
                                >\/7
                            _.-(6'  \
                           (=___._/` \
                                )  \ |
                               /   / |
                              /    > /
                             j    < _\
                         _.-' :      ``.
                         \ r=._\        `.
                        <`\\_  \         .`-.
                         \ r-7  `-. ._  ' .  `\
                          \`,      `-.`7  7)   )
                           \/         \|  \'  / `-._
                                      ||    .'
                                       \\  (
                                        >\  >
                                    ,.-' >.'
                                   <.'_.''
                                     <'

I reverted all of my changes and found that the exact same deploy from this morning that worked fine is now failing.

I just bisected releases of balena-cli. Anything that is v11.31.18 and later is failing. Anything that is v11.31.17 or earlier works fine. It looks like this is the change that caused the break: https://github.com/balena-io/balena-cli/compare/v11.31.17...v11.31.18

This is running on linux using the standalone asset packages.

Hey @braschgroup-ops,

Looks like you’re at the balena-cli v11.31.22 and balena deploy failed, can you please confirm that?
If possible, can you please downgrade to some old versions and see which one is working for you? It will really help us narrow down the problem.

Hey, https://github.com/balena-io/balena-cli/pull/1774 should fix those error messages. As far as I can tell the generated releases from my testing still look correct though so the releases should work

The fix has now been merged and deployed as balena-cli v11.31.23

Hi,
i am getting the same push error and after the update of the balena-cli.

Hi @pgiotis , are you sure you are getting the same error with balena-cli >= v11.31.23 ? Can you send us complete error log?

[Error]   Deploy failed
images.push is not iterable (cannot read property Symbol(Symbol.iterator))
TypeError: images.push is not iterable (cannot read property Symbol(Symbol.iterator))
    at authorizePush (/usr/local/lib/node_modules/balena-cli/build/utils/compose.js:397:12)
    at /usr/local/lib/node_modules/balena-cli/build/utils/compose.js:456:40
From previous event:
    at /usr/local/lib/node_modules/balena-cli/build/utils/compose.js:456:18
    at processImmediate (internal/timers.js:456:21)
    at process.topLevelDomainCallback (domain.js:137:15)
From previous event:
    at /usr/local/lib/node_modules/balena-cli/build/utils/compose.js:452:14
From previous event:
    at Object.deployProject (/usr/local/lib/node_modules/balena-cli/build/utils/compose.js:449:10)
    at /usr/local/lib/node_modules/balena-cli/build/actions/deploy.js:104:24
    at processImmediate (internal/timers.js:456:21)
    at process.topLevelDomainCallback (domain.js:137:15)
From previous event:
    at /usr/local/lib/node_modules/balena-cli/build/actions/deploy.js:103:22
From previous event:
    at /usr/local/lib/node_modules/balena-cli/build/actions/deploy.js:86:8
From previous event:
    at deployProject (/usr/local/lib/node_modules/balena-cli/build/actions/deploy.js:52:72)
    at /usr/local/lib/node_modules/balena-cli/build/actions/deploy.js:199:16
    at processImmediate (internal/timers.js:456:21)
    at process.topLevelDomainCallback (domain.js:137:15)
From previous event:
    at /usr/local/lib/node_modules/balena-cli/build/actions/deploy.js:198:22
From previous event:
    at Command.action (/usr/local/lib/node_modules/balena-cli/build/actions/deploy.js:195:8)
    at /usr/local/lib/node_modules/balena-cli/node_modules/capitano/build/command.js:98:37
    at runNextTicks (internal/process/task_queues.js:62:5)
    at processImmediate (internal/timers.js:429:9)
    at process.topLevelDomainCallback (domain.js:137:15)

i downgrade the cli to 11.31.17 and deploy the build successfully.

  • The “tag does not exist” error was fixed in CLI v11.31.23
  • The “images.push is not iterable” error was fixed in CLI v11.31.24

They were caused by maintenance work on the CLI codebase (a conversion away from CoffeeScript) which unfortunately fell through the testing cracks. The affected CLI releases were 11.31.18 to 11.31.22. Sorry for the trouble this has caused, and thanks to everyone in this thread who reported the errors and helped us get them resolved. :+1: