# balena-cli is not using the cache on fresh machine **URL:** \ **Category:** openBalena\ **Tags:** balena-cli\ **Created:** [February 9, 2020, 9:38pm UTC](https://forums.balena.io/t/balena-cli-is-not-using-the-cache-on-fresh-machine/62266 "2020-02-09T21:38:00Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![Langhalsdino](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/langhalsdino/32/3039_2.png) [@Langhalsdino](https://forums.balena.io/u/Langhalsdino)\ **Post date:** [February 9, 2020, 9:38pm UTC](https://forums.balena.io/t/balena-cli-is-not-using-the-cache-on-fresh-machine/62266/1 "2020-02-09T21:38:00Z")
Hi, i want to share builds with other people in my team. Therefore i set up a docker registry that stores the build images. Sadly when other people are pulling the image the cache seems not to be used. Since our builds take ~3h on a fast workstation (CUDA, OpenCV, Tensorflow, …) caching is very usefull. ## How to reproduce the behaviour 1. push a build image to you registry `docker tag myapp_main registry.yourdomain.com/bla` `docker push registry.yourdomain.com/bla:latest` 2. prune all local images `docker system prune --all` 3. pull image `docker pull registry.yourdomain.com/bla:latest` `docker tag registry.yourdomain.com/bla:latest myapp_main:latest` 4. do a new build without changing the dockerfile or its resources -\> Cache is not used Do you have any idea on how to tag the images, in order to get chaching to work? ---
**Author:** ![afitzek](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/afitzek/32/359_2.png) [@afitzek](https://forums.balena.io/u/afitzek)\ **Post date:** [February 10, 2020, 11:35am UTC](https://forums.balena.io/t/balena-cli-is-not-using-the-cache-on-fresh-machine/62266/2 "2020-02-10T11:35:44Z")
Hi, Can you provide us with more details on how you build the image in the first place and when the cache is used? Also is this an emulated build? In which case we start a docker daemon in a qemu to perform cross architecture builds. ---
**Author:** ![Langhalsdino](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/langhalsdino/32/3039_2.png) [@Langhalsdino](https://forums.balena.io/u/Langhalsdino)\ **Post date:** [February 10, 2020, 4:37pm UTC](https://forums.balena.io/t/balena-cli-is-not-using-the-cache-on-fresh-machine/62266/4 "2020-02-10T16:37:07Z")
I build the docker image cross platform. Therefore the qemu platform is used to build it. Build and deploy in one stage: `balena deploy myapp --logs --dockerfile Dockerfile --source . --emulated --build` Is there a way to get the cached images into the `~/.balena/bin` folder since it seems like the qemu builds are stored there or am i completely off? ---
**Author:** ![shaunmulligan](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/shaunmulligan/32/34_2.png) [@shaunmulligan](https://forums.balena.io/u/shaunmulligan)\ **Post date:** [February 12, 2020, 8:29am UTC](https://forums.balena.io/t/balena-cli-is-not-using-the-cache-on-fresh-machine/62266/5 "2020-02-12T08:29:56Z")
Hi Frederic, the built images would be stored in your local docker registry once they are built. But I think most likely the CLI won’t by default use all the on disc images that exist while building. docker used to do this automatically but due to some cache poisoning exploits they disabled automatically using all images and only use images that were build by the engine on your host. So one needs to pass a specially flag that tells the build to use all the available images, or a list of images as cache. I suspect the cli deploy command is not using that flag when building, but it should be possible to add. ---
**Author:** ![Langhalsdino](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/langhalsdino/32/3039_2.png) [@Langhalsdino](https://forums.balena.io/u/Langhalsdino)\ **Post date:** [February 12, 2020, 8:45am UTC](https://forums.balena.io/t/balena-cli-is-not-using-the-cache-on-fresh-machine/62266/8 "2020-02-12T08:45:36Z")
@shaunmulligan Can you point me in a direction on what i should look into in order to modify the balena-cli, to do so? ---
**Author:** ![shaunmulligan](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/shaunmulligan/32/34_2.png) [@shaunmulligan](https://forums.balena.io/u/shaunmulligan)\ **Post date:** [February 12, 2020, 8:55am UTC](https://forums.balena.io/t/balena-cli-is-not-using-the-cache-on-fresh-machine/62266/9 "2020-02-12T08:55:54Z")
I think it would be a matter of adding it to [https://github.com/balena-io/balena-cli/blob/master/lib/actions/build.coffee](https://github.com/balena-io/balena-cli/blob/master/lib/actions/build.coffee) . We would need to try add the `--cache-from` option to the build (more details on that option here: [https://docs.docker.com/engine/reference/commandline/build/#specifying-external-cache-sources](https://docs.docker.com/engine/reference/commandline/build/#specifying-external-cache-sources) and it seems the images would need to be built using the buildkit backend of docker too, but unsure on that). I have asked the CLI maintainers to weigh in on this as well as they would be able to give more specific details on how to add that option. ---
**Author:** ![Langhalsdino](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/langhalsdino/32/3039_2.png) [@Langhalsdino](https://forums.balena.io/u/Langhalsdino)\ **Post date:** [February 12, 2020, 9:12am UTC](https://forums.balena.io/t/balena-cli-is-not-using-the-cache-on-fresh-machine/62266/11 "2020-02-12T09:12:47Z")
The --cache-from option is what i looked into, too. But i just could not figure out, where i could hardcode it into the balena-cli for a quick fix and later do a Pull Request with a better implementation. I am no expert in coffee script and the project seems quiet complex to understand. Therefore i am looking for file you can point me to, that describes how to sett the build options with docerode or similar, so i can work my way from there. ---
**Author:** ![shaunmulligan](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/shaunmulligan/32/34_2.png) [@shaunmulligan](https://forums.balena.io/u/shaunmulligan)\ **Post date:** [February 12, 2020, 9:19am UTC](https://forums.balena.io/t/balena-cli-is-not-using-the-cache-on-fresh-machine/62266/12 "2020-02-12T09:19:00Z")
Yeah, I have been trying to find where the options are set as well, but have had no luck and also have little to zero coffee experience, so think we need to wait until the CLI team come online, they will be able to point us in the right direction super fast. ---
**Author:** ![Langhalsdino](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/langhalsdino/32/3039_2.png) [@Langhalsdino](https://forums.balena.io/u/Langhalsdino)\ **Post date:** [February 12, 2020, 9:19am UTC](https://forums.balena.io/t/balena-cli-is-not-using-the-cache-on-fresh-machine/62266/14 "2020-02-12T09:19:45Z")
Cool, thank you for your help. Looking forward to hear from the CLI team 🙂 ---
**Author:** ![Langhalsdino](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/langhalsdino/32/3039_2.png) [@Langhalsdino](https://forums.balena.io/u/Langhalsdino)\ **Post date:** [February 12, 2020, 9:37am UTC](https://forums.balena.io/t/balena-cli-is-not-using-the-cache-on-fresh-machine/62266/16 "2020-02-12T09:37:24Z")
Just as information for the CLI team. The build argument needs to be parsed through dockerode. Since it isusing the docker api [docker\_build](https://docs.docker.com/engine/api/v1.37/#operation/ImageBuild) The `cachefrom` option is defined as `string: JSON array of images used for build cache resolution.`. ---
**Author:** ![pdcastro](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/pdcastro/32/1264_2.png) [@pdcastro](https://forums.balena.io/u/pdcastro)\ **Post date:** [February 12, 2020, 11:49am UTC](https://forums.balena.io/t/balena-cli-is-not-using-the-cache-on-fresh-machine/62266/18 "2020-02-12T11:49:10Z")
> Therefore i am looking for file you can point me to, that describes how to set the build options with dockerode or similar, so i can work my way from there. @Langhalsdino, I’ve created a draft PR, not yet tested and thus maybe not working, with the likely changes to add a `--cachefrom` option to the balena build and deploy commands: [Add '--cachefrom' option to balena build / deploy commands · Issue #1616 · balena-io/balena-cli · GitHub](https://github.com/balena-io/balena-cli/issues/1616) If you get to test it, your feedback will be very welcome! ---
**Author:** ![Langhalsdino](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/langhalsdino/32/3039_2.png) [@Langhalsdino](https://forums.balena.io/u/Langhalsdino)\ **Post date:** [February 12, 2020, 11:51am UTC](https://forums.balena.io/t/balena-cli-is-not-using-the-cache-on-fresh-machine/62266/19 "2020-02-12T11:51:00Z")
I will try it out in the next hours, since this feature is very important to us 🙂 We spend about 8h a day on building images and this feature can cut it down to ~1h a day 🙂 ---
**Author:** ![Langhalsdino](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/langhalsdino/32/3039_2.png) [@Langhalsdino](https://forums.balena.io/u/Langhalsdino)\ **Post date:** [February 12, 2020, 7:57pm UTC](https://forums.balena.io/t/balena-cli-is-not-using-the-cache-on-fresh-machine/62266/21 "2020-02-12T19:57:28Z")
Yes i tried it and it does not seem to work. My current guess is, that the Dockerfile will get treated similar to how docker compose gets treated, therefore we might need to implement something like [this](https://docs.docker.com/compose/compose-file/compose-file-v2/#cache_from). More of the details can be found [here](https://github.com/balena-io/balena-cli/pull/1617) ---
**Author:** ![Langhalsdino](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/langhalsdino/32/3039_2.png) [@Langhalsdino](https://forums.balena.io/u/Langhalsdino)\ **Post date:** [February 12, 2020, 10:30pm UTC](https://forums.balena.io/t/balena-cli-is-not-using-the-cache-on-fresh-machine/62266/23 "2020-02-12T22:30:33Z")
> [@Langhalsdino](#): > > More of the details can be found [here](https://github.com/balena-io/balena-cli/pull/1617) I do not know why, but it started working now with docker-compose files and regular dockerfiles with no change at all 😃 Thanks for the PR 🙂 Just as a recap for other who might be interested: 1. Clone the repository and build the app ```bash git clone https://github.com/balena-io/balena-cli.git cd balena-cli git checkout 1616-deploy-cachefrom npm install npm rund build:fast export MY_PATH_TO_BALENA=$(pwd) ``` 1. build your app with caching ```bash cd your/balena/project/path docker pull registry.gitlab.com/myapp ${MY_PATH_TO_BALENA}/balena-cli/bin/balena deploy my-app --logs --dockerfile Dockerfile --source . --emulated --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from registry.gitlab.com/myapp -t registry.gitlab.com/myapp ``` 1. push new build to registry ```auto docker push registry.gitlab.com/myapp ``` ---
**Author:** ![samothx](https://avatars.discourse-cdn.com/v4/letter/s/c89c15/32.png) [@samothx](https://forums.balena.io/u/samothx)\ **Post date:** [February 12, 2020, 10:37pm UTC](https://forums.balena.io/t/balena-cli-is-not-using-the-cache-on-fresh-machine/62266/24 "2020-02-12T22:37:50Z")
Thanks for posting your results, I will inform the maintainer that the modification works for you. ---
**Author:** ![pdcastro](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/pdcastro/32/1264_2.png) [@pdcastro](https://forums.balena.io/u/pdcastro)\ **Post date:** [February 14, 2020, 3:35pm UTC](https://forums.balena.io/t/balena-cli-is-not-using-the-cache-on-fresh-machine/62266/26 "2020-02-14T15:35:27Z")
@Langhalsdino, thank you for the feature suggestion and for helping with the testing. The `--cache-from` option for the `balena build` and `balena deploy` commands was released in CLI version 11.26.0. For the benefit of others reading this thread, it implements the same feature documented for the `docker build --cache-from` option, “Specifying external cache sources”: > **[docker build](https://docs.docker.com/engine/reference/commandline/build/)** > > Description Build an image from a Dockerfile Usage docker build \[OPTIONS\] PATH | URL | - Options Name, shorthand Default Description --add-host Add a custom host-to-IP mapping (host:ip) --build-arg Set...