Some of our Balena Fin-derived devices really struggle to download container images. The failure mode is to repeatedly try to download the image and fail. This behavior continues after a reboot, however internet connectivity is never lost. The error message is as below -
Hey there, can you provide the version of balenaOS you are running on the affected devices? Do the images eventually download, or do they fail forever and get stuck? How large are the images we are talking about?
Sure - the Balena OS version is balenaOS 2.72.0+rev1. The images get about 12 or 13% downloaded then crap out. Even after multiple reboots, they never recover (although the host OS is still OK with network connectivity). The image size is 1.66GB. It seems unlikely to me that any ports are being blocked, as there are 20 other devices on the same router that are not struggling in the same way. Health check on this device shows no problems. Is there a way to manually pull the image, and/or to increase any timeouts on the connection ? - Thanks,
Currently the timeout duration for image pulls is statically set here: balena-supervisor/images.ts at d30116217ae5e28e00e1d88233cfd742ff6db346 · balena-os/balena-supervisor · GitHub, or (2 ^ numImageFailures * 500) ms timeout, so it’s not currently configurable. Based on the error message, this looks like it could be that the engine is having trouble communicating with the balena registry. Could you paste the output of curl -v https://registry2.balena-cloud.com to see if there are networking issues accessing balena’s registry URL? As for your question about manual pull, I’ll ask internally and get back to you.
Hi again, thank you for sharing those details. That tells us this is not a certificate issue. Could try pulling pulling this image (is a public supervisor image) manually to see if you experience the same issue?
Thanks for performing that test. It seems like something is terminating the connection, although is hard to tell a priori if there is a network issue or something else interfering with the download.
What can you tell us about the network these devices are on? Is it on wifi or mobile network?
Do you have the same issue trying to download from other registries? Could you try balena pull ubuntu:latest?
Could you check the device health checks by going to the device page on the dashboard and clicking on the left side menu Diagnostics and then Run checks?
Let us know how those tests work for you. The next step if you are willing would be to enable support access and let us take a look. Thank you