Delta updates failing with 500 error

Product support
1

I seem to have gotten my device into a permanently non-updateable state, where any update from the currently running release shows 500 errors from the delta server.

I’ve also noticed that generating releases gives me a warning that deltas couldn’t be saved and would be generated on-demand (but the release completes successfully). That seems fine, except the on-demand deltas don’t seem to ever be successfully generated during the actual update (even if I leave the device updating for hours).

A 500 seems to indicate an issue on the Balena side - is there any chance someone at Balena could look into it? On my end, apart from blowing the device software away, I’m actually unsure about how to cleanly flash the newest software from scratch, given there’s no way to turn off delta updates. Restarting services doesn’t seem to do it.

Thanks in advance!

3 Likes
2

Seeing the same issues on our side

Failed to download image 'registry2.balena-cloud.com/v2/4fec50fc8c5e65b443437408ba39f934@sha256:8ea489a253220b17ccc5325e89a4bd1ece42766c4af539bd664a533c2f1c8e85' due to 'Got 500 when requesting v3 delta from delta server.'

This has been happening all day today, but https://status.balena.io/ shows no issues.

1 Like
3

Same here. I get the same 500 error but yesterday my delta deploys were working fine.

4

Same, I had no issue before. I get the exact same error message as @g_jessica_austin

For what it’s worth, I did update to my final working release right before today’s VPN outage, but that seems more like a coincidence.

I’m gonna give some older releases a try tomorrow to see if I can generate deltas for a downgrade.

5

Hello @sith, @g_jessica_austin, @panbanda ,
We are having an incident with our Delta generation service. We are working towards fixing it.
Details on the status page here: Balena.io Status - Image deltas are not being generated.

Sorry for the trouble!

Thanks and regards,
Pranav

1 Like
6

Thanks Pranav, appreciate it!

7

@sith @g_jessica_austin @panbanda

Checking in here. Are things back in working order for you all? We’re here to help.

Also flagging for our team that our status page should be much more accurate when these outages happen.

Let us know if you have any other questions.

2 Likes
9

I just verified that delta updates are working for me again. Thanks @andrewnhem - appreciate the support.

11

sorry for the delay in replying, but yes the deltas have been working for us since you reported the fix. thank you!

13

Hello, we are experiencing the same 500 error issues with new image downloading. Since 1h or so. I don’t see any “incidents” on Balena Status, is there a way to contact Balena for Support ?

14

The team is not aware of any ongoing incidents right now but feel free to keep an eye on https://status.balena.io for any news or updates. The level of support depends on the plan you’re on as explained in balena - Get in touch for support