balena deploy has uncaught errors and creates broken releases

Product support
1

Hi,

We’re building and deploying our images from our CI with balena deploy, and for multiple releases now the deploy has succeeded (exit code 0) with multiple of the following messages:

This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason:
Error: An image does not exist locally with the tag: registry2.balena-cloud.com/v2/[hash removed]
    at Stream.<anonymous> (/snapshot/versioned-source/node_modules/docker-progress/index.js:53:19)
    at Stream.emit (events.js:198:13)
    at Stream.EventEmitter.emit (domain.js:448:20)
    at drain (/snapshot/versioned-source/node_modules/through/index.js:36:16)
    at Stream.stream.queue.stream.push (/snapshot/versioned-source/node_modules/through/index.js:45:5)
    at Parser.exports.parse.parser.onToken (/snapshot/versioned-source/node_modules/JSONStream/index.js:132:18)
    at Parser.proto.write (/snapshot/versioned-source/node_modules/jsonparse/jsonparse.js:135:34)
    at Stream.<anonymous> (/snapshot/versioned-source/node_modules/JSONStream/index.js:23:12)
    at Stream.stream.write (/snapshot/versioned-source/node_modules/through/index.js:26:11)
    at IncomingMessage.ondata (_stream_readable.js:708:20)
    at IncomingMessage.emit (events.js:198:13)
    at IncomingMessage.EventEmitter.emit (domain.js:448:20)
    at addChunk (_stream_readable.js:287:12)
    at readableAddChunk (_stream_readable.js:268:11)
    at IncomingMessage.Readable.push (_stream_readable.js:223:10)
    at HTTPParser.parserOnBody (_http_common.js:122:22)

It creates a new release, however, and when our devices try to pull it, a number of the image pulls fail with 404 errors, which would seem to correspond to the ones that failed in the deploy. There are many very large images being deployed, so maybe the infrastructure can’t handle it, but the errors aren’t even being detected. Luckily we’re still in testing, because it’d be pretty bad if Balena pushed broken releases to the field.

Thanks.

5

Thank you for the report. Could you provide the CLI version you are using so we can look into this please?

6

Can you please also let us know what is the approximate size of the images since you mentioned they are very large?

7

Sorry you’ve encountered this issue. I’ll be looking into this today. Could you please tell me which CLI version you are using? Also, if you have any extra infomation, logs, etc. that you think might help us reproduce and diagnose, it would be much appreciated. Thank you.

13

The CLI version is 11.31.20, and there are 10 images, with 6 around 200-300mb, and 4 around 1 - 1.5 gb.

Besides multiple of the messages I posted above about different containers, the only other logs were the “Saving release…” messages and the “Deploy succeeded!” message.

14

Thank you for the extra information. We are looking into it, and I’ll keep you updated.

16

@mrjogo, an investigation is ongoing, however something that I am aware of is that three issues affecting balena deploy were fixed last week in CLI releases:

  • 11.31.23 Fix unhandled promise rejection when using balena deploy
  • 11.31.24 balena deploy: Fix “TypeError: images.push is not iterable”
  • 11.31.26 balena build/deploy: Update QEMU version to support newer balenalib images
  • CHANGELOG

Two of these issues were introduced in v11.31.18, and I see you mentioned you’re using v11.31.20, therefore the release you are using is affected. Try updating the CLI. Version 11.31.26 is one that I had tested myself (balena deploy), and the current latest is v11.32.14. If for any reason you had to go to an earlier release, v11.31.17 or older may be good options too.

17

It appears that this issue may have been fixed in CLI v11.31.24 (that fix was in response to a very similar issue). Please could you try the latest CLI version, and see if the issue persists?: https://github.com/balena-io/balena-cli/releases/tag/v11.32.14

19

Thanks for looking into it. Was the underlying error fixed (Error: An image does not exist locally with the tag...) or just the fact that the deploy will now fail? Looking at the fix it looks like a retry loop maybe was not being called correctly upon failure?

21

Hello @mrjogo,

The fixes we pointed out on the CLI should fix the problem by handling the underlying issue. Have you tried with the CLI v 11.31.26 or newer?

Please let us know if you are still facing the issue.

Cheers.

23

Hi @ntzovanis,

We switched over to balena push which seems less flaky.

Thanks