Builder stuck on uploading images

Product support
1

First time this happened I killed it after a minute or so. When I re-tried it built my container from scratch, apparently the cached intermediate steps had been lost. But it’s still stuck on “Uploading images”

[main]     Successfully built 9f2dd9d9b907
[Info]     Uploading images
[Info]     Still working...
[Info]     Still working...
[Info]     Still working...
[Info]     Still working...
[Info]     Still working...
[Info]     Still working...
[Info]     Still working...
[Info]     Still working...
3

Hi @whm,

Thanks for your report! We are aware of an issue with one of the ARM builders and are actively investigating it. Can you confirm you were building for an ARM device as well? If you still have the log, towards the top you should see a message like Building on arm01.

4

Hi again @whm,

We have fixed the underlying issue with the build server and builds should be restored. Please let us know if you have any further issues!

Thanks again for the report!

1 Like
5

The image finally uploaded. It only took about 40 minutes :upside_down_face:.

Can you confirm you were building for an ARM device as well? If you still have the log, towards the top you should see a message like Building on arm01 .

Since the image built from scratch the relevant messages are outside my terminal’s scrollback. But I ran another build and it was on arm03. It is also stuck, but I am going to try to restart it. I’ll let you know how that goes.

7

@xginn8 I just started two new builds and they are both stuck in the uploading images phase. So maybe this is a different issue from the one you fixed?

9

@whm we’re investigating the issue. Sorry for the inconvenience. Will get back to you once we found & fix the problem.

10

Hi all,

We arre (at the time of posting) still/also experiencing issues with the builder on ARM03 and ARM01

additionaly i got the following message in the console

[Info]          Building on arm01
[proxy]         [=========================================================] 100%
[Info]          Uploading images
[Info]          Still working...
[Info]          Still working...
[Info]          Still working...
[Info]          Still working...
[Info]          Still working...
[Info]          Still working...
[Info]          Still working...
[Error]         An error occurred: connect ECONNREFUSED 147.75.195.26:2376
[Error]         Not deploying release.
Remote build failed
Additional information may be available by setting a DEBUG=1 environment
variable: "set DEBUG=1" on a Windows command prompt, or "export DEBUG=1"
on Linux or macOS.

If you need help, don't hesitate in contacting our support forums at
https://forums.balena.io
11

@kevdea welcome to the forums!

We’re investigating the issue. I’ll ping you as well once we resolve this issue.

1 Like
12

We’re still working on the problem.

In the meanwhile, you can use emulated builds (not affected by this issue):

In other words, there’s --emulated flag for balena deploy / push . Or you can push to the balena-emulated branch via git push balena master:balena-emulated .

13

I just wanted to let you know that everything is operational as we speak, and you should be able to build on the native ARM builders. Let us know if you still experience any issues.

3 Likes