Balena browser-aarch64 block fails to download… balenaHub - IoT embedded device apps, edge computing resources, and developer tools
Is there something wrong with the container registry?
Could this be the same issue as described here?
I think something is broken on the registry side. Hopefully the Balena team will sort it after the weekend. I’ve had lots of issues with images coming from bh.cr the last few days stumpd me for a bit… I see your running [balena-ads-b]. you can comment out the kiosk and everything else will build fine.
Yeah it’s not me that’s using it but I’m helping maintain balena-ads-b and it’s causing builds to fall.
Basically balena builder is failing to properly pass the ARG in the FROM field to pull the image from bh.cr for the build.
More info in this comment here…
We have a workaround but it seems that the parsing of the Dockerfile is not following the correct logic for how Dockerfile is supposed to work with ARGs allowed before the first FROM and allowed to be referenced therein.
Hey @shawaj , thanks for raising this issue! I was able to reproduce the following error on our cloud builders (per the linked Github issue).
ARG BROWSER_VERSION=2.9.0
FROM bh.cr/balenalabs/browser-%%BALENA_ARCH%%/$BROWSER_VERSION AS base
[main] Step 1/2 : ARG BROWSER_VERSION=2.9.0
[main] Step 2/2 : FROM bh.cr/balenalabs/browser-aarch64/$BROWSER_VERSION AS base
[main] pull access denied for bh.cr/balenalabs/browser-aarch64/2.9.0, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
but this works
ARG BROWSER_VERSION=2.9.0
FROM bh.cr/balenalabs/browser-%%BALENA_ARCH%%/2.9.0 AS base
[main] Step 1/2 : ARG BROWSER_VERSION=2.9.0
[main] Step 2/2 : FROM bh.cr/balenalabs/browser-aarch64/2.9.0 AS base
[main] ---> 78d9a6af0336
[main] Successfully built 78d9a6af0336
I’m going to see if I can sort out when this was introduced and how to resolve it.
1 Like
Much appreciated @klutchell 
Yeah it’s a really weird one because it clearly is parsing the ARG for the error message, but not for the actual pulling of the image!
@shawaj do you have an approximate date when this used to work?
I see that some aspects of automatic authentication do not work when ARGS are included in the image tag, but that shouldn’t apply to public blocks such as this and it seems that bit of behaviour has not changed.
I don’t know if it used to work or not in this particular scenario with Balena. Just that it works in a Dockerfile with docker.
For example it’s even used by the browser block in its own Dockerfile:
In balena-ads-b we only just recently changed to this particular logic to get renovate bot to work in the repo. But were surprised to see it not work.
The example Dockerfile you shared in your last comment is pulling from Dockerhub, and from what I can tell this issue only affects the use of ARGS in the image name when it is coming from bh.cr/. Anything from Dockerhub or other registries should be able to resolve just fine with ARGS in the name.
Can you confirm that is what you are seeing as well?
Oh I haven’t tried it with any other registry as we need to get it from bh.cr I?
Or are you saying the browser-aarch64 package is also available from DockerHub?
Oh I just meant any other image that IS available on Dockerhub for testing. Don’t worry if you haven’t tried. I tried using ARGS with images from Dockerhub and Github Container Registry and had no issues, so it only seems to impact bh.cr.
1 Like
Strange! Is that expected behaviour for bh.cr?
@shawaj ARGS in the image name should be correctly supported by the builder now. Can you give it another try?
3 Likes
All working now, thanks @klutchell - fantastic support from balena as always
P.S. If you have a minute to look at this as well it would be much appreciated… balena wifi-connect not working correctly
1 Like
@klutchell this seems to be broken again now…
Error: Some services failed to build:
Error: Service: kiosk
Error: Error: pull access denied for bh.cr/balenalabs/browser-aarch64/2.9.0, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
From run here - Merge pull request #138 from Teko012/add/osupdate · ketilmo/balena-ads-b@385e91f · GitHub
weirdly - I ran it from github two more times and on it was “third time lucky” - so it seems to maybe be a bit inconsistent/temperamental.
So a bit strange.
Interesting…
My best guess at this point is that something got cached when the first pull failed due to network issues.
Error: Error: Get "https://registry-proxy.balena-cloud.com/v2/balenalabs/browser-aarch64/2.9.0/manifests/latest": Get "https://api.balena-cloud.com/auth/v1/token?scope=repository%3Abalenalabs%2Fbrowser-aarch64%2F2.9.0%3Apull&service=registry2.balena-cloud.com": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
You can see the request to get a token from our API timed out for whatever reason. And since it’s the API that caches the access requests, it might have worked after a 15 minute timeout for cache expiry, or even just hitting a different API pod by chance.
1 Like
Ah ok, that makes sense.
Will keep an eye on it and report back if we see it again. Maybe we were just unlucky that time!