Balena is not able to download delta images for all of the container

Product support
1

Hi,

I am trying to update the release, its update only few of the container not all.
it was working well, but suddenly the problem has come.

Error logs is this

`
Delta still processing remotely. Will retry…
Delta still processing remotely. Will retry…
Delta still processing remotely. Will retry…

Downloading delta for image ‘registry2.balena-cloud.com/v2/390fd9d32acbf4f44044079c10698a8f@sha256:0294501092ccdad83c960b812745f136e4285cf631d63ffe5a08f52e438f8256’
local-dns-lan [INFO] 172.16.80.11:32221 - 23854 “AAAA IN api.balena-cloud.com. udp 38 false 512” NOERROR qr,rd,ra 134 0.003640151s
Downloading delta for image ‘registry2.balena-cloud.com/v2/390fd9d32acbf4f44044079c10698a8f@sha256:0294501092ccdad83c960b812745f136e4285cf631d63ffe5a08f52e438f8256’
Downloading delta for image ‘registry2.balena-cloud.com/v2/26d9ea9618bbcf95a60b768555733811@sha256:a75bbca8f5ed9fc059e7951dbbb3a91721fcbd4ba1e5e54770a7b5af853c9001’
Downloading delta for image 'registry2.balena-cloud.com/v2/a97931df625e63855af9a0989ddad94c@sha256:e2c8d2239baa80d8088ae5cd18d4c1be6e7825b98f9c090ffe257fc41844b51e’`

balena
balena2282×1504 347 KB

Additional details

root@7943ba9:~# curl -I http://api.resin.io
HTTP/1.1 301 Moved Permanently
Server: awselb/2.0
Date: Mon, 13 Nov 2023 03:18:33 GMT
Content-Type: text/html
Content-Length: 134
Connection: keep-alive
Location: https://api.resin.io:443/

root@7943ba9:~# nmcli c show
NAME         UUID                                  TYPE      DEVICE      
northbound   0184a25b-7c07-e1af-1d30-4671f06f7479  ethernet  eth0        
southbound   0184a232-232b-ad0c-a2a5-f9d88457330d  ethernet  eth1        
supervisor0  13d8c9ed-7202-4867-859e-7f0dd5263742  bridge    supervisor0 
cellular     0184a266-e20f-1cf1-4476-b86f095bf760  gsm       --

can you please provide some solution on this or what causing this?

1 Like
2

+1 Experiencing seemingly the same issue.

1 Like
3

+1 Also experiencing this. This is a pretty big deal considering we can’t update/change the release on any of our devices.
Running Supervisor 14.1.0 and balenaOS 2.106.7.

4

@mpous @klutchell sorry to tag you guys but wasn’t sure who to escalate this to. Seems like a pretty big outage (if it is affecting more than just us) as we don’t have another way to update our devices remotely

5

Hey @trey,

We’re looking into this as a priority. We’ll update you as soon as we can.

Phil

6

A build did finally go through on our end FYI. Thanks for the quick response.

1 Like
7

@trey @mugeesh

This should all be sorted now. Plus we’ve added instrumentation to detect any errors with deltas much quicker. And we’ll continue to look into and resolve the root cause. Thanks for your patience.:heart:

Phil

3 Likes
8

I’m hitting this issue presently, is there a new incident or could this also be a local phenomenon? I managed to force the update using the method described here, but this is probably not something I would want to do on production