Many failed builds over the last 24 hours

Product support
1

Hello,

We have experienced many failed builds over the last 24 hours with the remote builders (we have actually not been able to get a build out in the last 24 hours).

Our fleet is on balenalib/iot-gate-imx8-debian and the “build” is basically a no-op (just copying files that we upload from a local cross compile into the docker images). The failure mode is either a complaint about a 408 HTTP error or a silent (successful return code) exiting of the program shortly after uploading the resources.

An example of the silent failure mode:

+ balena push <fleet_name> --release-tag type dev
[Info]                     Starting build for <fleet_name>, user <user_name>
[Info]                     Dashboard link: https://dashboard.balena-cloud.com/apps/<app_id>/devices

The build shows up in our dashboard as “…Running” for several hours (normally they take less than 20 seconds), and then transitions to “Failed” with the logs just showing “Build overview logs still processing or not available”

Any help/information about the status of the ARM builders would be much appreciated.

2

For posterity: we stood up an ARM build cluster a couple hours after this post, completely removing the remote build - just using deploy - which has fixed this issue. Seems like there is something going on with the remote builders.

1 Like