I tried it on AWS with the docker-compose guide in production mode and got a similar results.
Randomly it now starts appearing with in a 1hour interval, any thoughts?
[Logs] [3/12/2020, 9:00:25 PM] Downloading image 'registry.balena/v2/myid@sha'
[Logs] [3/12/2020, 10:13:20 PM] Failed to download image 'registry.balena/v2/myid@sha' due to '(HTTP code 404) no such image - no such image: registry.balena/v2/id@sha: No such image: registry.balena/v2/id@sha '
[Logs] [3/12/2020, 10:13:22 PM] Downloading image 'registry.balena/v2/id@sha'
Since this is a difficult, but important problem i might be able to share access to the device with the balena team, if it might help to resolve the issue.
Thank you for testing this and getting back to us. It would be extremely useful for us to be able to access the device, as this would allow us to inspect its state and try and determine what might be causing the issue.
Since i think you are the resident beekeeper at Balena you have probably my apic.ai mail address. Therefore feel free to contact me and we can sort out how to give you monitored access to the openBalena server and device
I will share our findings here and try to keep the conversation in the forum. I just dont want to share public keys, … in the forum!
#summary Customer on OpenBalena on GCP running supervisor version 10.6.27 sees a HTTP code 404 error when downloading a large update on a slow network. They say the download “breaks every 2 hours. They don’t have any other devices to test if related to this specific supervisor version.
Customer has tried this now on AWS EC2 as well, and the same problems are occuring (but at a 1 hour interval). We’ve asked for access to the device so that we can try and diagnose the situation. Customer has got back privately, and we’re discussing how we might be able to help them here as to get access will involve a secure connection to their VPC. I’m summarising this for the moment but would like this to reopen when required.
Sorry i might have described it a bit off. The 1 hour interval is on GCP present, too.
It`s just with another code version that is updating on another device.
I recurrently saw a failed download attempt every 2 hours. It’s a large update on a very slow connection.
[logs] [3/3/2020, 6:31:06 AM] Failed to download image ‘registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5’ due to '(HTTP code 404) no such image - no such image: registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5: No such image: registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5 ’
[Logs] [3/3/2020, 6:31:09 AM] Downloading image ‘registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5’
[Logs] [3/3/2020, 8:27:16 AM] Failed to download image ‘registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5’ due to '(HTTP code 404) no such image - no such image: registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5: No such image: registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5 ’
Environment, background and context:
I have a really slow internet connections. Only a few kb per sec on a mobile data plan. I checked with the carrier and they did not disconnect the devices every 2 hours. Furthermore on there side the connection seems to be open for multiple days.
When using scp to copy files between the device and another client i observed braking connections due to the slow connection, too.
How did i solve it
I solved it by increasing the timeouts in the haproxy config of open balena.
Since the timeouts are in milliseconds 5 and 50 sec are not a lot. Specially when working with a slow connection. Therefore i increased the timeout to some insane number of 60m. Lastly due to the large timeout i could see a spike in open connections and therefore i set the connection limit manually to maxconn 5000.
Lastly i did a pull request on github in order to add units to the timeout, since this would make it easier to read:
The error message you’re encountering, “Failed to download image ‘xxxxx.com 1’ due to '(HTTP code 404) no such image - no such image,” indicates that the device is unable to find the specified image in the Docker registry. This error typically occurs when the device requests an image that doesn’t exist in the registry at the specified tag or SHA.
Here are some troubleshooting steps you can follow:
Check Image Availability: Confirm that the image with the specified tag or SHA actually exists in your Docker registry. Double-check the image name and tag to ensure they are correct.
Image Pull Policy: Ensure that your application or device is not set to pull images from a different registry or a specific version that doesn’t exist.
Cached Images: Docker may cache images locally. Clear the local Docker image cache on your device and try the deployment again to ensure it attempts to download the image from the registry.
Registry Configuration: Double-check your OpenBalena server configuration to ensure it’s properly configured to store the registry files on S3. Ensure that the registry is correctly configured to pull images from the specified source.
BalenaOS Version: It’s worth checking if there are any known issues or updates related to your BalenaOS version. Upgrading to a newer version may resolve potential issues.
Network Issues: Network issues can also cause problems with image downloads. Ensure that the device has a stable internet connection and can access the Docker registry.
Device Compatibility: Verify that the Raspberry Pi 3 running BalenaOS 2.46.1 is compatible with the images you are trying to download. Ensure that the architecture and platform match.
If redeploying the application resolves the issue temporarily, it could be related to the cache cursed bed images or a temporary network issue. By following the steps above and ensuring your configuration is correct, you can potentially resolve the problem without needing to redeploy the application each time.
