Hey chrisys - it was indeed an “old” image problem for me (go 1.14). That pilot-error admitted, there are some things for balena to think about here.
My fleet has dependencies (GPIO, GRPC) that are a royal pain in the butt to migrate forward. I need images to stay available basically forever. What’s the downside to Balena in leaving images available? Imho, images should have the same availability guarantee as Fin.
I’m maintaining a public fleet. When you “disappear” an image it breaks that public fleet. This seems like a bad thing for BalenaHub as much as it is a bad thing for me. Perhaps a little consideration for BalenaHub users?
Where do we publish the warning that my build is about to blow up? The only warning I get now is when my nightly build fails.
As this thread makes abundantly clear, we need better error messages when our base image has been “disappeared”. I’m sitting here with a build that worked yesterday and fails today with a completely misleading error message and I didn’t change anything. The only reasonable conclusion is that Balena itself is broken.
Finding base images - I’ve been doing Balena for a long while now and I just recently found the actual base images URL (base-images/balena-base-images at master · balena-io-library/base-images · GitHub). This link needs to be posted all over the place so dopes like me who are lucky enough to guess that they have a disappeared image problem can easily tell if their image has been ‘disappeared’.
“Disappeared image” to me should be a preprocessing check in the “balena push”. < 100 lines of code that produces one of two messages “Image found, proceeding” - “Image NOT found, aborting”.
@chrisys I am currently trying to use balenalib/rpi-raspbian (linux/arm/v6) as the base image, which appears to be maintained.
The target device is an rpi 4, 64 bit.
I noticed this message in the build logs: The requested image's platform (linux/arm/v6) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Later, I get an error: [balena-cam] failed to get destination image "sha256:e1a6df702de55efd1d12527db6bf6b5fe4e8a6276de3404ead94a940c158b3f5": image with reference sha256:e1a6df702de55efd1d12527db6bf6b5fe4e8a6276de3404ead94a940c158b3f5 was found but does not match the specified platform: wanted linux/arm64/v8, actual: linux/arm/v6
However, a 32 bit OS (linux/arm/v6) should be able to run on a 64 bit OS (in this case, arm64/v8)
Hey @rodley I hear all your points, thanks for sharing everything here. There’s no downside to keeping the images available for historical reasons, but it does cause issues such as this when we’re trying to push forward with the development of the builder and engine but are held back by continuing to support very old things. I believe we also have to pay storage costs but would rather continue to provide a resource that helps folks like you to keep building. Just to be clear though, no images have been removed here nor is there any current plan to remove them!
As for finding the base images, we do have plans to improve that. It really needs a search/filter tool. They have been too hard to find for a very long time!
@jpayne0061 thanks for the extra detail here - given the use case you’ve detailed is also affected it does appear that we’ve regressed here. It should be possible to use the 32 bit containers on a 64 bit device. We’ll continue to look into it.
Can you confirm the default device type on your application or fleet?
I think your fleet is set to an aarch64 default device type? So that’s what our builders are using when pushing releases. However as you saw in the the logs, the base image is armv6 so the builder is complaining.
You are correct that this should work, aarch64 is definitely capable of running armv6 images. We are investigating internally to see what can be done to resolve this regression.
In the meantime, if you try pushing the same project to a fleet with an armv6 default device type, does it work?
If you make a new fleet (for testing) and select Raspberry Pi 3 (not the 64-bit version) as the default device type, I suspect that you should be able to push to that fleet as a workaround while we investigate.
I have followed the steps above and successfully created a release image for one of our fleets however when trying to create a release image for another fleet with the same default device type of (aarch64) I am still getting the same error again despite using the newly created base image: failed to get destination image "sha256:9fbae44e7414963313d4a0f2ba517eecfe9b39289e5526afebf6d50e79d43b62": image with reference sha256:9fbae44e7414963313d4a0f2ba517eecfe9b39289e5526afebf6d50e79d43b62 was found but does not match the specified platform: wanted linux/arm64/v8, actual: linux/amd64
Here are the steps I took to create the docker build:
Created a Dockerfile with the line: FROM balenalib/raspberrypi4-64-ubuntu-python:3-bionic-build-20200915
Then ran: docker buildx build . --platform linux/arm64/v8 --tag sophiahaoui/raspberrypi4-64-ubuntu-python:3-bionic-build-20200915
Then pushed the build: docker push sophiahaoui/raspberrypi4-64-ubuntu-python:3-bionic-build-20200915
The docker build is here: Docker and does appear to have the correct architecture linux/arm64
Could there be any other fleet settings that could be causing an issue here?
Thanks,
Sophia
Thank you for the additional investigation, Sophia. We are still investigating on our side and that is a big help!
Just to be clear the error you are reporting happens at the end of a balena push to a fleet, correct?
Does it appear from the logs that all the steps are successful and the error happens at the end of the build, after “Successfully uploaded images” is reported?
Would you mind sharing the logs from your balena push attempt, and maybe the relevant parts of the Dockerfile you are pushing to the fleet?
@sophiahaoui did you then update your original Dockerfile to point to your new image? It sounds like it’s still pulling the old image if it’s saying it’s linux/amd64, as I agree, your new one looks correct and is showing linux/arm64. Alternatively share your Dockerfile as my esteemed colleague suggested above
| Packaging the project source...[Warn]
[Warn] -----------------------------------------------------------------------------------------
[Warn] The following .dockerignore file(s) will not be used:
[Warn] * /Users/sophiahaoui/NewSunRoad/Solsense/balena/solmon/node_modules/sqlite3/.dockerignore
[Warn] When --multi-dockerignore (-m) is used, only .dockerignore files at the
[Warn] root of each service's build context (in a microservices/multicontainer
[Warn] fleet), plus a .dockerignore file at the overall project root, are used.
[Warn] See "balena help push" for more details.
[Warn] -----------------------------------------------------------------------------------------
[Info] Starting build for customers_gen25, user shaoui
[Info] Dashboard link: https://dashboard.balena-cloud.com/apps/1811933/devices
[Info] Building on arm01
[Info] Pulling previous images for caching purposes...
[Success] Successfully pulled cache images
[solmon] Step 1/30 : FROM sophiahaoui/raspberrypi4-64-ubuntu-python:3-bionic-build-20200915 as build
[solmon] ---> 17ea33c69306
[solmon] Step 2/30 : WORKDIR /usr/src/app
[solmon] Using cache
[solmon] ---> 9fbae44e7414
[solmon] Step 3/30 : COPY /lib/requirements3.txt /lib/requirements3.txt
[solmon] failed to get destination image "sha256:9fbae44e7414963313d4a0f2ba517eecfe9b39289e5526afebf6d50e79d43b62": image with reference sha256:9fbae44e7414963313d4a0f2ba517eecfe9b39289e5526afebf6d50e79d43b62 was found but does not match the specified platform: wanted linux/arm64/v8, actual: linux/amd64
[Info] Uploading images
[Success] Successfully uploaded images
[Error] Some services failed to build:
[Error] Service: solmon
[Error] Error: failed to get destination image "sha256:9fbae44e7414963313d4a0f2ba517eecfe9b39289e5526afebf6d50e79d43b62": image with reference sha256:9fbae44e7414963313d4a0f2ba517eecfe9b39289e5526afebf6d50e79d43b62 was found but does not match the specified platform: wanted linux/arm64/v8, actual: linux/amd64
[Info] Built on arm01
[Error] Not deploying release.
Remote build failed
And here are the Dockerfile lines for those first steps that get through:
FROM sophiahaoui/raspberrypi4-64-ubuntu-python:3-bionic-build-20200915 as build
WORKDIR /usr/src/app
COPY /lib/requirements3.txt /lib/requirements3.txt
RUN pip3 install -r /lib/requirements3.txt
Let me know what else would be helpful!
Thanks,
Sophia
Hey @jpayne0061, what were the application errors you were getting at runtime? It’s expected that uname would show the actual architecture of the device, but the armv7 image should have no issues running on aarch64.
Thank you @sophiahaoui, this is very helpful. I’ve been able to reproduce it locally and on our remote builders, and I’m investigating possible solutions. I hope to have more to share soon!
Here’s another failing Dockerfile (stripped down from my original but still failing). I had previously been using golang:1.15, but getting this error made me upgrade. Unfortunately, there’s still an error.
FROM --platform=linux/arm64 balenalib/raspberrypi4-64-golang:1.19 as builder
RUN go install -v github.com/mozz100/tohora@v0.3.2
FROM bh.cr/balenalabs/browser-aarch64
RUN install_packages cec-utils
COPY --from=builder /go/bin/tohora /home/chromium/tohora
Here is how the logs end:
[Info] Uploading images
[Success] Successfully uploaded images
[Error] Some services failed to build:
[Error] Service: kiosk
[Error] Error: failed to get destination image “sha256:3d5a94136dbf7f4a051c26c37fc56d644152b80809daf306cae51c95be20c272”: image with reference sha256:3d5a94136dbf7f4a051c26c37fc56d644152b80809daf306cae51c95be20c272 was found but does not match the specified platform: wanted linux/arm64/v8, actual: linux/amd64
[Info] Built on arm01
[Error] Not deploying release.
Remote build failed
