Why is retrieving cache so slow?

My builds say they take 16 seconds to deploy, but if you include retrieving the cache it is much slower than that, what causes cache retrieval to be so slow, and what could be done to improve the time?

It is significant because it affects my development speed. Are there any settings I could modify to make it faster?

EDIT: To be more specific, I’m talking about the time period in which I see this message:
Pulling previous images for caching purposes…

Hey @cnr,

If your image is very big, this will increase the cache retrieval time.
You can decrease the size of your image or disable caching.

Which method are you using to push the applications? As this may change the way you disable caching.

Cheers

Are you using our CLI to push (balena push, balena deploy) or you are using the git push way?

Git push

5 Gb

I don’t want to disable caching because building opencv is part of my dockerfile, so it is definitely faster to run with the cache, it’s just slow to retrieve the cache.

@cnr I believe the speed of pulling the cache is based on the fact that the image registry is on AWS and the build (assuming you are building for Arm based device ) is happening on the packet.net ARM build servers so the bottle neck is most likely the download/network speed of packet.net. You could try force an emulated build using git push balena master:balena-emulated which would mean your actual build would happen on AWS in a qemu emulated build rather than on the ARM builder, this would most likely pull the images much much faster because its on AWS purely, but your build for openCV will be probably 10x slower :confused:

Hey @shaunmulligan,

Could you tell me if build images are cached on AWS for local builds as well? e.g. balena push <my device IP>

I’m experiencing intermittent network issues during local builds and was wondering if there was any background network traffic due to caching?

Thanks,
Elliot

Hi Elliot, when running in Local Mode, the containers are built directly on the device, so any caching is from layers already on the device, but no there is no caching coming from AWS. Of course, network connectivity reaching outside of your network is still possible, as an example the need to download a base image from Dockerhub, a private registry, etc. But, the build itself should not be interfacing with AWS, no. Hope that helps clarify, and there is some more info here: https://www.balena.io/docs/learn/develop/local-mode/
Thanks!