Deploying new release gets stuck at delta generation

Product support
, ,
1

Hi,
For the past 6 hours or so, releases have not been going through in our CI. The build goes through but delta generation hangs forever at 71%.
Here are the logs:

Logs 
[Info]                        Generating image deltas from release 8d2086a8fe3fd97b2c4d7f5a7e747ef5 (id: 2642327)
[=>                                                 ] 3%
[=>                                                 ] 3%
[=>                                                 ] 3%
[=>                                                 ] 3%
[...]
[=============>                                     ] 27%
[=============>                                     ] 27%
[=============>                                     ] 27%
[=============>                                     ] 27%
[...]
[===================================>               ] 71%
[===================================>               ] 71%
[===================================>               ] 71%
[===================================>               ] 71%
[===================================>               ] 71%
[===================================>               ] 71%
[===================================>               ] 71%
[===================================>               ] 71%
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...
[Info]                        Still Working...

It’s been saying “Still Working…” for the past hour or so.
status.balena.io says “All systems operational”, but on our end, it looks like the Application Builder shouldn’t be in the green.
Stopping and restarting the release gives the same results.
Please let us know how this can be resolved.
Thank you,
Erwan

3 Likes
2

Update: our CI timed out automatically after 2 hours. Last message was still “Still working…”

3

Experiencing identical issues currently.

4

+1

Exactly the same as OP.

5

Thanks for reaching out, I’m taking a look.

1 Like
6

This is affecting us as well.

7

I guess you disabled delta generation? :eyes:

[Info]                        Generating image deltas from release 8d2086a8fe3fd97b2c4d7f5a7e747ef5 (id: 2642327)
[Warning]                     Failed to generate deltas due to an internal error; will be generated on-demand

Our build just went through!

8

I reverted what I think must be the cause for the regression and deployed in order to restore service as soon as possible and unblock you. I’ll fix whatever issue was introduced when I manage to consistently reproduce it locally and debug, but for now you should be good to go (a small caveat – due to how the builder is deployed, the old buggy containers might linger for a while, so if you think you hit the issue again within the next couple of hours, please cancel the build and retry so you that hit another instance).

2 Likes