Sporadically I get builds failing when pushing to the Balena Git remote.
[Error] An error occured: (HTTP code 404) no such image - no such image: 3fae64f17662: No such image: 3fae64f17662:latest
The image which cannot be found shows up in the build logs, it is a cached image and was successfully built.
[camera] Step 40/40 : CMD ["/bin/bash","start_cmd.sh"]
[camera] Using cache
[camera] ---> 3fae64f17662
[camera] Successfully built 3fae64f17662
When I retry the build it builds every service from scratch not using any cache. This is quite a time consuming process and means that the devices, even for a small change, download a lot of code.
The application was being built using arm03 and the release id is 858ffb0b784419ae99468ff6e51a080bb13ad6c7
Hi @hpgmiskin, could you please provide more details?
It helps to have further context.
To start with, which specific command do you run for pushing?
Assuming that you use balena-cli, what’s the version you run? How about your development environment? It helps if you could give us OS and version.
We will use such info to reproduce the issue on our end.
Next, could you run your push commands with DEBUG=1 prefix and send us the output?
Finally, if it’s possible, it’d be great to see your docker compose file. We’ve seen a similar issue happening when the base image is not pinned to a release. For example: FROM balenalib/raspberrypi3-debian-node:8-run
instead of: FROM balenalib/raspberrypi3-debian-node:8-run-20200115
We are pushing to a git remote therefore not using the Balena CLI. Would it be better to make use of the CLI? Out of interest what differences are there on the builder side between pushing to a git remove or building using the CLI? Will the DEBUG=1 work with git push?
The issue only arrises when the builder is trying to use a layer which was cached from a previous build. I am happy to try fixing the base image to a particular release, I can see this would improve build times as there would never be the need to rebuild lower levels. Would we be required to manually increment the base image each time there are security updates released?
It could likely be a manifestation balena-cli#1629, however, as pointed out by @pdcastro the issue is likely sourced within the Balena builder or multi-builder. Should I create an account for flowdock in order to take a better look at the discussion?
As for resin-multibuild#70 I do not fully understand the issue, is the suggestion of Docker HUB that the user has created their own base image on Docker HUB and then the builder is not able to find the image? If that is the case then the issue is not relevant as we use a Balena base image.
FROM balenalib/%%BALENA_MACHINE_NAME%%-debian:stretch-build
I hope the above make sense, please correct me if I have made any incorrect assumptions about the git push versus using Balena CLI push. I am happy to try steps of resolution which you recommend.
git push and balena push is quite similar on the builder side but there are some limitations with git push (more details can be found in our docs here https://www.balena.io/docs/learn/deploy/deployment/#limitations) and DEBUG=1 only works with balena push so we recommend you should use balena push.
The pinned base image will never be rebuilt so if you want to have latest fixes/updates then you should change to latest pinned base image when needed but on the other hand, your build will be stable and there won’t be any unwanted breaking changes that could harm your application.
You don’t need a flowdock account for this, just keep an eye on the github issue or we can ping you here when the issue is fixed.
Our balenalib base images are hosted in Dockerhub so looks like you’re having this issue https://github.com/balena-io-modules/resin-multibuild/issues/70, I will ping the relevant colleague to see if there is any updates on it. Please let us know if you have more questions