Getting repeated Failed to download image updating device after pushing balenaSound to application

pigcry · February 25, 2020, 9:37pm

Device is updating after I pushed balenaSound project to the application, now I get the following failures

samothx · February 25, 2020, 10:01pm

Hi @pigcry,
looking at the status page ( https://status.balena.io/ ) we did have an API outage today but the registry did not have any issues.
Otherwise TLS handshake timeout mostly points to bandwidth related problems so it would be good to make sure your device can access the registry with sufficient speed.
Please also provide some info about device type and balena-os version used.
Regards Thomas

pigcry · February 26, 2020, 8:09pm

Still seeing various different errors when trying the download, latest is

Failed to download image ‘registry2.balena-cloud.com/v2/db7a068fbc3fce8146f6b249f556d263@sha256:34145d9efb66eb4f205efa0bd31eb7526d81c5b8c7d17de3f0aeb8f75ca0e180’ due to '(HTTP code 404) no such image - no such image: registry2.balena-cloud.com/v2/db7a068fbc3fce8146f6b249f556d263@sha256:34145d9efb66eb4f205efa0bd31eb7526d81c5b8c7d17de3f0aeb8f75ca0e180: No such image: registry2.balena-cloud.com/v2/db7a068fbc3fce8146f6b249f556d263@sha256:34145d9efb66eb4f205efa0bd31eb7526d81c5b8c7d17de3f0aeb8f75ca0e180

I have flashed a faster class 10 SD card incase it was the write speed and the device has a decent ADSL 8-10Mb down speed. Got closer this time, 13% through the download and 35% through one of the containers but then hit the above errors. Version info is:

HOST OS VERSION - balenaOS 2.46.1+rev1
SUPERVISOR VERSION - 10.6.27

How large are the six balenaSound container downloads? I did a test 12M download in less that 20 seconds as a verification of network speed.

Ereski · February 26, 2020, 9:09pm

Hi. If you enable support access and share the UUID of the device, we can take a look.

Ereski · February 26, 2020, 9:34pm

Could you also share the application UUID?

pigcry · February 27, 2020, 6:30pm

Hi there, access shared uuid is 675e98f2db1149d5009fd4a9efbc4cda

_Page · February 28, 2020, 12:07am

Hey, to update you on this, you can check the image sizes via the “Releases” tab on your application dashboard, but currently it’s 6 images with an average of around 200MB. For the issue itself it looks to me to be that downloading so many large images at once is overloading your raspberrypi as it’s an old model and also doesn’t have much RAM, the faster sd card might help with this as it’ll allow writing to disk faster and may alleviate some of the load.
Outside of that as a workaround you can also try removing a few services from the docker-compose.yml and then pushing, letting those download, then adding some more and letting them download and so on in order to reduce the load, I’ve also raised with the supervisor team an idea of having a max concurrent downloads config var to help with cases like this where a device or network is unable to download so many images in parallel

pigcry · February 28, 2020, 7:45am

Thanks for that, I’ll give that a spin.

pigcry · February 28, 2020, 2:22pm

Thanks that worked. commenting out all but one service at a time in the docker-compose-yml did the trick. Thanks for your help.

alok.span · September 9, 2020, 9:36pm

@pigcry

What is the location where we need to modify docker-compose.yml? I see one at the below location -

(base) C02LX7P3FD57:balena-sound-master alsharm$ pwd
/Users/alsharm/Downloads/Balena/balena-sound-master
(base) C02LX7P3FD57:balena-sound-master alsharm$
(base) C02LX7P3FD57:balena-sound-master alsharm$ ls -l docker-compose.yml
-rwxr-xr-x@ 1 alsharm ABCD Users 1447 Aug 24 12:09 docker-compose.yml

But I think it’s not correct because it’s on my location machine and not on the server.

dash · January 15, 2021, 12:24am

Is there a way to change the behavior of supervisor to only download images sequential to alleviate this in slow network environments?

At the moment it appears to try download them all concurrently.

bblankenship · September 7, 2021, 6:50pm

Can we bump this? I’m experiencing this as well and it prohibits us from updating our devices properly. I understand the “comment out a service at a time” solution, but that’s not viable in a production environment.

ab77 · October 21, 2021, 4:31pm

You can try modifying the balenad service on your device to limit the concurrent downloads (see systemctl cat balena).

Appending --max-concurrent-downloads 1 to BALENAD_EXTRA_ARGS should be enough, followed by a service restart. the root filesystem on the device will require to be remounted read/write in order to achieve this and it may lead to unexpected side effects.

A better solution would be to move the device to a faster network.

olso · April 5, 2022, 9:11pm

I’ve tried this, but it did not have any effect

--max-concurrent-downloads 1 ← this caused service to crash

--max-concurrent-downloads=1 ← this did not

Has there been any development on this?

ramirogm · June 17, 2022, 11:29am

Hi
just checking to see if you have managed to sort this out or you’d like some more help.

Ramiro

Topic		Replies	Views
Never ending "Downloading image" .... Balena-cloud Product support balenasound	17	1197	September 10, 2020
Failed to download image ... due to '(HTTP code 404) no such image Product support	16	1680	August 18, 2020
Issues with Balena Registry Product support	5	553	February 5, 2021
Service images continually fail to download Product support raspberrypi3	6	1101	December 26, 2019
Failed to download image in balenaSound setup balenaSound	2	441	April 8, 2021

Getting repeated Failed to download image updating device after pushing balenaSound to application

Related topics