registry unauthorized after certificate update

[matteopeluso]

Hi!

I am experiencing the this error and I cannot reach the registry anymore… I am a openbalena v2.0.0 user and I recently update the root/vpn certificates through re-running the quickstart script on my server

I am using the balena.-cli version:

balena --version
12.2.2

I get this error while trying to deploy an application:

DEBUG=1 balena deploy myApp

I got this error:

[debug] original argv0="/usr/local/lib/balena-cli/bin/node" argv=[/usr/local/lib/balena-cli/bin/node,/usr/local/lib/balena-cli/bin/run,deploy,myApp] length=4
[Debug]   Parsing input...
(node:97932) ExperimentalWarning: The fs.promises API is experimental
[Debug]   Loading project...
[Debug]   Resolving project...
[Debug]   docker-compose.yml file found at "/Users/matteo/Desktop/iottacle/balenaApp/myApp"
[Debug]   Creating project...
[Info]    Everything is up to date (use --build to force a rebuild)
[Info]    Creating release...
[Debug]   Tagging images...
[Debug]   Authorizing push...
[Info]    Pushing images to registry...
Retrying "registry.mydomain.com/v2/token:latest" after 2.00s (1 of 3) due to: Error: unauthorized: authentication required
Retrying "registry.mydomain.com/v2/token:latest" after 2.00s (1 of 3) due to: Error: unauthorized: authentication required
Retrying "registry.mydomain.com/v2/token:latest" after 2.00s (1 of 3) due to: Error: unauthorized: authentication required
Retrying "registry.mydomain.com/v2/token:latest" after 2.80s (2 of 3) due to: Error: unauthorized: authentication required
Retrying "registry.mydomain.com/v2/token:latest" after 2.80s (2 of 3) due to: Error: unauthorized: authentication required
Retrying "registry.mydomain.com/v2/token:latest" after 2.80s (2 of 3) due to: Error: unauthorized: authentication required
Retrying "registry.mydomain.com/v2/token:latest" after 3.92s (3 of 3) due to: Error: unauthorized: authentication required
Retrying "registry.mydomain.com/v2/token:latest" after 3.92s (3 of 3) due to: Error: unauthorized: authentication required
Retrying "registry.mydomain.com/v2/token:latest" after 3.92s (3 of 3) due to: Error: unauthorized: authentication required
[Debug]   Saving image registry.mydomain.com/v2/token
[Debug]   Untagging images...
[Info]    Saving release...
[Error]   Deploy failed
unauthorized: authentication required

Error: unauthorized: authentication required
    at Stream.<anonymous> (/usr/local/lib/balena-cli/node_modules/docker-progress/index.js:53:19)
    at Stream.emit (events.js:198:13)
    at Stream.EventEmitter.emit (domain.js:448:20)
    at drain (/usr/local/lib/balena-cli/node_modules/through/index.js:36:16)
    at Stream.stream.queue.stream.push (/usr/local/lib/balena-cli/node_modules/through/index.js:45:5)
    at Parser.parser.onToken (/usr/local/lib/balena-cli/node_modules/JSONStream/index.js:132:18)
    at Parser.proto.write (/usr/local/lib/balena-cli/node_modules/jsonparse/jsonparse.js:135:34)
    at Stream.<anonymous> (/usr/local/lib/balena-cli/node_modules/JSONStream/index.js:23:12)
    at Stream.stream.write (/usr/local/lib/balena-cli/node_modules/through/index.js:26:11)
    at IncomingMessage.ondata (_stream_readable.js:693:20)
    at IncomingMessage.emit (events.js:198:13)
    at IncomingMessage.EventEmitter.emit (domain.js:448:20)
    at addChunk (_stream_readable.js:288:12)
    at readableAddChunk (_stream_readable.js:269:11)
    at IncomingMessage.Readable.push (_stream_readable.js:224:10)
    at HTTPParser.parserOnBody (_http_common.js:122:22)
    at awaitRegistryStream (/usr/local/lib/balena-cli/node_modules/docker-progress/index.js:43:12)
    at /usr/local/lib/balena-cli/node_modules/docker-progress/index.js:416:16
    at runCallback (timers.js:705:18)
    at tryOnImmediate (timers.js:676:5)
    at processImmediate (timers.js:658:5)
    at process.topLevelDomainCallback (domain.js:126:23)
    at DockerProgress.exports.DockerProgress.DockerProgress.push (/usr/local/lib/balena-cli/node_modules/docker-progress/index.js:415:56)
    at Bluebird.join.retry (/usr/local/lib/balena-cli/build/utils/compose.js:499:197)
    at retry (/usr/local/lib/balena-cli/build/utils/helpers.js:111:31)
    at Bluebird.delay.then (/usr/local/lib/balena-cli/build/utils/helpers.js:116:53)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10)
From previous event:
    at runCommand (/usr/local/lib/balena-cli/build/app-capitano.js:57:20)
    at Object.run (/usr/local/lib/balena-cli/build/app-capitano.js:67:42)
    at routeCliFramework (/usr/local/lib/balena-cli/build/preparser.js:44:79)
    at process._tickCallback (internal/process/next_tick.js:68:7)
    at Function.Module.runMain (internal/modules/cjs/loader.js:832:11)
    at startup (internal/bootstrap/node.js:283:19)
    at bootstrapNodeJSCore (internal/bootstrap/node.js:622:3)

I went on the server on the registry container and I found this error:

Oct 26 15:02:51 edbb2ad0bb03 registry[1013]: 172.22.0.9 - - [26/Oct/2021:15:02:51 +0000] "POST /v2/v2/token/blobs/uploads/ HTTP/1.1" 401 272 "" "docker/20.10.8 go/go1.16.6 git-commit/75249d8 kernel/5.10.47-linux
Oct 26 15:02:51 edbb2ad0bb03 registry[1013]: time="2021-10-26T15:02:51.610354925Z" level=warning msg="error authorizing context: invalid token" go.version=go1.11.2 http.request.host=registry.mydomain.com http.request.id=2e57bd7b-5dbb-40
Oct 26 15:02:51 edbb2ad0bb03 registry[1013]: 172.22.0.9 - - [26/Oct/2021:15:02:51 +0000] "POST /v2/v2/token/blobs/uploads/ HTTP/1.1" 401 272 "" "docker/20.10.8 go/go1.16.6 git-commit/75249d8 kernel/5.10.47-linux

I am noticing the same error also on a local device:

[826]: [event]   Event: Docker image download {"image":{"name":"registry.mydomain.com/v2/token@sha256:d1ed6efeed649f2956e2c3a93108ecf2147e200988671e718dfb704ab6298808","app>
	resin-supervisor[1791]: [event]   Event: Docker image download {"image":{"name":"registry.mydomain.com/v2/token@sha256:d1ed6efeed649f2956e2c3a93108ecf2147e200988671e718dfb704ab6298808">
	balenad[826]: time="2021-10-27T07:55:55.529681875Z" level=info msg="Attempting next endpoint for pull after error: errors:\nunauthorized: authentication required\nunauthorized: authentication required\n"
	balenad[826]: time="2021-10-27T07:55:55.531437742Z" level=info msg="Ignoring extra error returned from registry: unauthorized: authentication required"
	balenad[826]: time="2021-10-27T07:55:55.532203046Z" level=error msg="Handler for POST /images/create returned error: unauthorized: authentication required"

I am both able to ping the registry and the api correctly and I am able to use the balena cli commands which interact with the devices.

I’ve tried to “ping” the registry directly with my ca.crt and it doesn’t give me any error

curl -I https://registry.domain.com/ --cacert ~/ca.crt

answer:

HTTP/1.1 200 OK

**Cache-Control** : no-cache

**Date** : Wed, 24 Nov 2021 15:35:28 GMT

I am able to generate a token as:

curl -H "Authorization: Bearer token-auth" "https://api.mydomain.com/auth/v1/token?service=registry.domain.com&scope=repository:v2/token" --cacert ~/ca.crt

but then pulling an image as:
curl -H "Authorization: Bearer <generated-token>" "https://registry.domain.com/v2/v2/token/tags/list" --cacert ~/ca.crt

I get:

{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"v2/token","Action":"pull"}]}]}

Any idea? There’s something I could do to start using the registry again?

Thanks

Matteo

[ab77]

Could you take a look at /etc/docker-registry.yml in registry2 container to see if the API cert is there, matches the cert/key set on the API.

The root CA should be in /etc/ssl/certs/balenaRootCA.pem and the API cert is at /tmp/registry-tokenauth.crt in the registry container. It should verify against this CA cert.

[Winkelmann]

Hi,

There are several topic about this error, none of them solved it for me.

Deploy failed unauthorized: authentication required

I’m trying to deploy my first app to openBalena however I am getting the below error. Adding an application and provisioning of devices etc works OK. balena deploy myapp --logs --source . --emulated [Info] Everything is up to date (use --build to force a rebuild) [Info] Creating release… [Info] Pushing images to registry… Retrying “registry.mydomain.com/v2/d5876ca8376bc70a3fb5470403891a6d:latest” after 2.00s (1 of 3) due to: Error: unauthorized: authentication required Retrying “r…

balena deploy myapp -> Error: unauthorized: authentication required

When trying to do the first deploy to our openbalena instance, I get the following error: $ balena deploy myapp --registry-secrets ../registry-secrets.yml [Info] Everything is up to date (use --build to force a rebuild) [Info] Creating release... [Info] Pushing images to registry... Retrying "registry.balena.${domain}/v2/a0bd40c79a2756d16cd89b7b80959903:latest" after 2.00s (1 of 3) due to: Error: unauthorized: authentication required We require the registry secrets due to private CRs …

Balena deploy fails with error `unauthorized: authentication required` from the registry service

I followed this official tutorial to set up openBalena. The only thing I did differently was to include the -c flag to the ./scripts/quickstart command as it was suggested in the official screencast. Everything worked fine and I am able to add apps and devices to the system (this means that balena apps and balena devices commands work flawlessl. However, when I try to push a new image to the app and execute balena deploy <appName>, I get the following error: Retrying "registry.balena.mydomain…

I have a fresh install of OpenBalena v3.6.0 on DigitalOcean Droplet Ubuntu 22.04, and I use a Let’s Encrypt certificate.

I know I ran the quickstart script (./scripts/quickstart -U "mail" -P "pass" -d domain.com -c) without the -c option once, because I forgot about it. I then ran the quickstart script again, but with the -c, without deleting anything first, and everything worked fine (listing devices, provisionning images, etc).
Only deploying did not work.

What I did to fix it is the following:

• I ran ./scripts/compose down to stop
• I deleted the config folder that OpenBalena generates
• I ran the quickstart script again, making sure to include -c
• I ran ./scripts/compose build. Not sure this is necessary
• I ran ./scripts/compose up -d to restart

I suppose the registry was using the old self-signed cert instead of the Let’s Encrypt cert, which caused the issue in my case.
Hope this helps someone !

Tim