Hello,
Are there any updates on this?
Thanks
This is still causing us issues. As I have mentioned before I think the problems correlate with the amount of load the build servers are under. Is there an issue tracker where I can see the status of the resolution?
2 Likes
We are continuing to investigate the problem. Can you confirm that, as a workaround, you can successfully push the application using balena push with the --nocache option? i.e. balena push myApp --nocache. We believe that this might make it work
It might sound crazy but I somehow noticed that the problems occurs to me mostly when using Windows.
I am pretty unsure if this can be the case or just random that it happens to me on Windows.
I worked the whole day using Mac without any problems but now at home when pulling all changes and pushing it again with my windows desktop computer I somehow receive the “no such image” error again.
If this cannot be the case and just happens randomly please ignore it!
I am seeing the same issue: An error occured: (HTTP code 404) no such image - no such image
I have tried using the --nocache option and it did not work.
Hi @fcovatti – thank you for the additional info. Are you able to post the full output of your build, including the command that you ran to kick it off?
Thanks in advance,
Hugh
I did update balena-cli version to 12.3.4 and tried again the same command with the --nocache and now it worked. Thanks!
I have updated our build system such that we can push code changes with the --nocache option. However, we are still experiencing the issue with "no image found"!
Here is the output from running the Balena version command:
$ balena version --all
balena-cli version "12.3.6"
Node.js version "12.18.2"
When I ran the balena push command with the --nocache option the command output still mentions pulling previous images for caching purposes, however, it does not use and of the cached images in the build process.
$ balena push ***************** --nocache
...
[Info] Starting build for *****************, user admin10
[Info] Dashboard link: https://dashboard.balena-cloud.com/apps/*******/devices
[Info] Building on arm03
[Info] Pulling previous images for caching purposes...
[Success] Successfully pulled cache images
...
[metrics] Step 1/11 : FROM balenalib/raspberrypi3-debian-python:3.5.7-build
[metrics] ---> e87b1cd2d648
[metrics] Step 2/11 : ENV TELEGRAF_VERSION 1.13.0
[metrics] ---> Running in 8d56cf924b9b
[metrics] Removing intermediate container 8d56cf924b9b
[metrics] ---> fbf863eb2303
...
[metrics] Step 11/11 : CMD ["./start.sh"]
[metrics] ---> Running in 61d28d1ef302
[metrics] Removing intermediate container 61d28d1ef302
[metrics] ---> ee6ba1f0ff45
[metrics] Successfully built ee6ba1f0ff45
...
[Info] Uploading images
[Error] An error occured: (HTTP code 404) no such image - no such image: ee6ba1f0ff45: No such image: ee6ba1f0ff45:latest
[Info] Built on arm03
[Error] Not deploying release.
As you can see from the output the missing image is the final image of the metrics service.
Are there any other potential work arounds? Currently we are using balena deploy but this is painfully slow and really hampering our development.
1 Like
I think that’s anecdotal, because as I understood, the job of the CLI is to send the source archive to the servers and then everything happens in the cloud, and what you see are just the logs of the server. Updating the CLI would not have an impact in that case.
I am experiencing the same issue as @hpgmiskin but unlike them, I was not able to make the build work locally so I am stuck with an app that cannot be updated. I won’t pollute the thread here since I have another thread for my issue, but I will be following closely both.
I also dont think the cli has anything to do with it working :). But I was using an older version of the cli (11.8.3), updated (12.3.4) and tried once again to push to collect the logs and post it here. But then it worked.
I have experienced this same issue in the past with the no such image error, tried again a few times to push the image and it worked, with no cli update.
We have set up this page that can be used to track progress on our investigations into the builder failures - you can also subscribe to it to be notified of updates : https://status.balena.io/incidents/hszhtp712z59
I noticed that the incident page suggests there has been a resolution deployed.
We have just ran a build using balena push and it has failed with the no image found error.
Please can you provide an update on the status of the issue?
Hi there, I see that you’ve tried adding --nocache before, and you’ve probably tried it again now, right?
yes, we did a deployment, but we are still monitoring those changes and we are still actively working on this issue.
Thanks for your patience!
Sorry for the delay in response, we have tried building with the --nocache option a number of times and balena push is still failing on occasion with “no image found”. This does seem to fail more frequently during normal working hours.
For example our build will often succeed when building early in the morning. My assumption is that the builder, when under heavier load, will drop stored images. Perhaps even under the hood the images are slightly corrupted due to the number of images stored. When retrieving, if their checksums do not match the image is declared not found.
What is the current status of a fix for this issue? We are looking at rolling our own build server using an ARM EC2 instance on AWS. In early testing we could build from source in less than 7 minutes. Our current work around using qemu emulation takes at least 40 minutes. If there is no fix in the pipeline, creating our own build server will be a worthwhile endeavour.
Hello, I understand your concerns and we are working intensively in resolving this issue.
We have noticed that the chance that the image upload will fail is related to how long the build runs.
Retrying the build is recommended, and if still having no success, disabling data updates may help.
We’ll keep you posted.
Thanks for bearing with us,
Georgia
1 Like
Things are looking to be more reliable at the moment. I have been retrying builds when one fails with “no image found”. In general if they fail first time they will not pass the second time.
Hi there!
Glad that it looks more reliable now. We deployed a potential fix and we are working on stabilizing the builder. The problem seems dependent on the duration of the builds. How long does your build take? is it a multistage build?
We have not had a build failure in a very long time now so would like to thank the Balena team for all the work you have collectively put in to making the builder more reliable. For the time when we were building locally with emulation it really made us appreciate how much Balena does for us and how critical it is to our development.
When running on the arm servers from a fresh build all our services take < 20 minutes to build. Almost all of them use multi stage builds. One of the services installs and builds a number of packages for image processing and takes the lions share of the time to build.
1 Like
Sorry to bother again but yesterday we had a number of failed builds on Balena. These failed due to seeing “no such image” error. This error occurred after a build which took over 1 hour where they normally take at most 20 minutes from scratch. I have checked the logs and it was building with arm01 so should not have taken that long.
1 Like
Hi, I forwarded your information to the builders maintainers. We will get back to you once we have more information on the topic.
Thanks,
Zahari