Pulling cached images on Cloud Builder a little slow

I push to the Balena Cloud builders quite often. They work great, but I have noticed the ‘pulling of previous images’ process is quite slow. Here is the stage where the delays are:

[Info]              Pulling previous images for caching purposes...

My total build time is 1minute 49 seconds thanks to the cached layers:

[Info] Build finished in 1 minute, 49 seconds

Yet when I time the process I see that 67 seconds is the pulling of the cached images (73% of the build time is taken fetching the cache). There are two images built in the process, one 134mb and the other 54mb.

Of course overall I am way better off building from the cache as it speeds up the rest of the process, but considering all it is doing is pulling the previous images (and I assume that stage includes extraction), it seems a little slow and is a pinch point in development. Perhaps there is something to do with where the cached images are stored and fetched from that could be optimised to help us out?

Hi, I’ve pinged CLI team and they should have a response for you soon :slight_smile:

Hi,

How long the cache will take to get pulled is going to depend on a number of factors, including the amount of activity on the builders and what is hitting those registries. Cache pull time will take different amounts of time depending on how many builds are happening at once. It is difficult to tell if this ‘slowness’ is expected or not but we have noted your concern for future reference. Hope this helps!

Makes sense. Strange though that a build process runs quickly relative to the pulling time, you would think that would be equally affected by the demand on the builders. Those image sizes I noted are also downloaded images, the actual size of them when compressed which is presumably what is being pulled would be much smaller. So we are talking about 67 seconds to pull and extract under 100mb. Perhaps there is a lot more going on behind the scenes of that one line message of pulling previous images, but something didn’t seem to add up.

Thanks for taking a look at the post.

You are welcome!