OpenBalena push to registry fails

openBalena
1

I just did a fresh deployment with the latest docker builds and start getting a docker error while trying to push to the registry.

Pushing images to registry...
Retrying "registry.balena.domain.com/v2/c09053d4afa5eaf42f2ad58ac015d292:latest" after 2.00s (1 of 3) due to: Error: unauthorized: authentication required
Retrying "registry.balena.domain.com/v2/c09053d4afa5eaf42f2ad58ac015d292:latest" after 2.80s (2 of 3) due to: Error: unauthorized: authentication required
Retrying "registry.balena.domain.com/v2/c09053d4afa5eaf42f2ad58ac015d292:latest" after 3.92s (3 of 3) due to: Error: unauthorized: authentication required

Since everything seem to work with the v0.48.0 release of balena registry, i try to get my head around why this Error starts turning up.

I encountered it on MacOS X, Ubuntu and Debian with balena-cli versions 11.23 and 11.24.

2

This seems to be a bug with the current open-balean-registry version.
I downgraded to the following setup and it works like a charm.

I set the following versions in the config/activate file.

export OPENBALENA_API_VERSION_TAG=0.49.9
export OPENBALENA_REGISTRY_VERSION_TAG=2.13.1
export OPENBALENA_VPN_VERSION_TAG=9.10.0
export OPENBALENA_DB_VERSION_TAG=3.0.1
export OPENBALENA_S3_VERSION_TAG=2.9.0
5

Hi @Langhalsdino

We just want to confirm which version of open-balena-registry you’re using when encountering this error, is it v2.13.2? When you mention using v0.48.0 of the registry, are you referring to OPENBALENA_API_VERSION_TAG, which was set to 0.49.9 in the configuration that did work?

Kind regards
Alida

7

Sorry it was a little bit late when writing the post. I got confused with balena registry and balena api version.

I used the default installation with :latest tag at 1st Feb. 2020 and pushing to the registry failed.

I checked our currently running productive instance for its versions and it api version was v0.48.0 and registry version v2.12.0

Since upgrading to newer versions is mostly a good idea, i tried a couple of random configuration that are >v0.48.0 >2.12.0 and got the above mentioned configuration to work.

It is mostly up to date, therefore i feel confident upgrading our current productive instance to the above mentioned container versions.

I still could not figure out why the latest build produced the above mentioned error.

8

I am not sure either; I haven’t seen that error before in this use-case.

If you get it again, or find a way to reproduce it reliably then please open an issue here: https://github.com/balena-io/open-balena/issues

10

I have currently a thight schedule, but try to test it this week with a new url and a new server and open an issue accordingly.

I just wanted to put it here, in case some one else is encountering the same problem. Since it took me a while to figure out, that it was about the balena deployment and not anything else.

11

I just wanted to share that I had the same issue as this thread indicated with Error: unauthorized: authentication required reported during image push phase of command
balena deploy myApp --logs --source . --emulated

having just followed https://www.balena.io/open/docs/getting-started/ as of March 20 it seems the wrong version of registry is included. I saw that balena-registry version was 2.11.1 in default installation per the settings found in compose/versions of open-balena.

The fix for me was to wipe everything out, update the below file as such, and rebuild everything.

~/open-balena$ git diff compose/versions
diff --git a/compose/versions b/compose/versions
index fe971ac…8a6b2b7 100644
— a/compose/versions
+++ b/compose/versions
@@ -1,6 +1,12 @@
export OPENBALENA_API_VERSION_TAG=v0.19.5
export OPENBALENA_DB_VERSION_TAG=v2.0.3
export OPENBALENA_MDNS_PUBLISHER_VERSION_TAG=v1.6.2
-export OPENBALENA_REGISTRY_VERSION_TAG=v2.11.1
+export OPENBALENA_REGISTRY_VERSION_TAG=v2.13.5
export OPENBALENA_S3_VERSION_TAG=v2.8.5
export OPENBALENA_VPN_VERSION_TAG=v8.10.0

12

https://github.com/balena-io/open-balena/blob/master/compose/versions#L4 is the line i had to update.

15

Hello Balena team, @richbayliss,

We have Jenkins job to build and deploy. I was successful last time roughly 2 weeks back but now it fails with below error. Any idea what might be the reason?

Logs on Jenkins server –
registry.balena. mydomain.com/v2/c66bb67e52e2628dffc32f7b532da4d7:latest" after 2.80s (2 of 3) due to: Error: EOF

Registry logs from open balena server-
Jan 14 12:30:21 ed22856e0eb6 registry[742]: time=“2021-01-14T12:30:21.680112459Z” level=error msg=“response completed with error” auth.user.name=admin err.code=unknown err.detail=“s3aws: XMinioServerNotInitialized: Server not initialized, please try again.
Jan 14 12:30:21 ed22856e0eb6 registry[742]: status code: 503, request id: 165A191B46B5292F, host id: " err.message=“unknown error” go.version=go1.11.2 http.request.host=registry.balena.mydomain.com http.request.id=ef9c52ca-b75c-4926-bf8a-72c852c75694 http.request.method=POST http.request.remoteaddr=18.192.183.153 http.request.uri=”/v2/v2/2b75911a822d120c11a4c5b150fc0185/blobs/uploads/" http.request.useragent=“docker/19.03.4 go/go1.12.10 git-commit/9013bf583a kernel/4.15.0-1058-aws os/linux arch/amd64” http.response.contenttype=“application/json; charset=utf-8” http.response.duration=6m0.013898027s http.response.status=500 http.response.written=123 vars.name=“v2/2b75911a822d120c11a4c5b150fc0185”
Jan 14 12:30:21 ed22856e0eb6 registry[742]: 172.18.0.9 - - [14/Jan/2021:12:24:21 +0000] "POST

16

Hello Balena team,
Can some one support here please?
Regards

17

Hello balena team,
Above error was because of insufficient disk space.
It is fixed now with increase in space.
Any pointers for best practices after we deploy step? Any explicit step needed for disk cleanup?