registry unauthorized after certificate update

Hi!

I am experiencing the this error and I cannot reach the registry anymore… I am a openbalena v2.0.0 user and I recently update the root/vpn certificates through re-running the quickstart script on my server

I am using the balena.-cli version:

balena --version
12.2.2

I get this error while trying to deploy an application:

DEBUG=1 balena deploy myApp

I got this error:

[debug] original argv0="/usr/local/lib/balena-cli/bin/node" argv=[/usr/local/lib/balena-cli/bin/node,/usr/local/lib/balena-cli/bin/run,deploy,myApp] length=4
[Debug]   Parsing input...
(node:97932) ExperimentalWarning: The fs.promises API is experimental
[Debug]   Loading project...
[Debug]   Resolving project...
[Debug]   docker-compose.yml file found at "/Users/matteo/Desktop/iottacle/balenaApp/myApp"
[Debug]   Creating project...
[Info]    Everything is up to date (use --build to force a rebuild)
[Info]    Creating release...
[Debug]   Tagging images...
[Debug]   Authorizing push...
[Info]    Pushing images to registry...
Retrying "registry.mydomain.com/v2/token:latest" after 2.00s (1 of 3) due to: Error: unauthorized: authentication required
Retrying "registry.mydomain.com/v2/token:latest" after 2.00s (1 of 3) due to: Error: unauthorized: authentication required
Retrying "registry.mydomain.com/v2/token:latest" after 2.00s (1 of 3) due to: Error: unauthorized: authentication required
Retrying "registry.mydomain.com/v2/token:latest" after 2.80s (2 of 3) due to: Error: unauthorized: authentication required
Retrying "registry.mydomain.com/v2/token:latest" after 2.80s (2 of 3) due to: Error: unauthorized: authentication required
Retrying "registry.mydomain.com/v2/token:latest" after 2.80s (2 of 3) due to: Error: unauthorized: authentication required
Retrying "registry.mydomain.com/v2/token:latest" after 3.92s (3 of 3) due to: Error: unauthorized: authentication required
Retrying "registry.mydomain.com/v2/token:latest" after 3.92s (3 of 3) due to: Error: unauthorized: authentication required
Retrying "registry.mydomain.com/v2/token:latest" after 3.92s (3 of 3) due to: Error: unauthorized: authentication required
[Debug]   Saving image registry.mydomain.com/v2/token
[Debug]   Untagging images...
[Info]    Saving release...
[Error]   Deploy failed
unauthorized: authentication required

Error: unauthorized: authentication required
    at Stream.<anonymous> (/usr/local/lib/balena-cli/node_modules/docker-progress/index.js:53:19)
    at Stream.emit (events.js:198:13)
    at Stream.EventEmitter.emit (domain.js:448:20)
    at drain (/usr/local/lib/balena-cli/node_modules/through/index.js:36:16)
    at Stream.stream.queue.stream.push (/usr/local/lib/balena-cli/node_modules/through/index.js:45:5)
    at Parser.parser.onToken (/usr/local/lib/balena-cli/node_modules/JSONStream/index.js:132:18)
    at Parser.proto.write (/usr/local/lib/balena-cli/node_modules/jsonparse/jsonparse.js:135:34)
    at Stream.<anonymous> (/usr/local/lib/balena-cli/node_modules/JSONStream/index.js:23:12)
    at Stream.stream.write (/usr/local/lib/balena-cli/node_modules/through/index.js:26:11)
    at IncomingMessage.ondata (_stream_readable.js:693:20)
    at IncomingMessage.emit (events.js:198:13)
    at IncomingMessage.EventEmitter.emit (domain.js:448:20)
    at addChunk (_stream_readable.js:288:12)
    at readableAddChunk (_stream_readable.js:269:11)
    at IncomingMessage.Readable.push (_stream_readable.js:224:10)
    at HTTPParser.parserOnBody (_http_common.js:122:22)
    at awaitRegistryStream (/usr/local/lib/balena-cli/node_modules/docker-progress/index.js:43:12)
    at /usr/local/lib/balena-cli/node_modules/docker-progress/index.js:416:16
    at runCallback (timers.js:705:18)
    at tryOnImmediate (timers.js:676:5)
    at processImmediate (timers.js:658:5)
    at process.topLevelDomainCallback (domain.js:126:23)
    at DockerProgress.exports.DockerProgress.DockerProgress.push (/usr/local/lib/balena-cli/node_modules/docker-progress/index.js:415:56)
    at Bluebird.join.retry (/usr/local/lib/balena-cli/build/utils/compose.js:499:197)
    at retry (/usr/local/lib/balena-cli/build/utils/helpers.js:111:31)
    at Bluebird.delay.then (/usr/local/lib/balena-cli/build/utils/helpers.js:116:53)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10)
From previous event:
    at runCommand (/usr/local/lib/balena-cli/build/app-capitano.js:57:20)
    at Object.run (/usr/local/lib/balena-cli/build/app-capitano.js:67:42)
    at routeCliFramework (/usr/local/lib/balena-cli/build/preparser.js:44:79)
    at process._tickCallback (internal/process/next_tick.js:68:7)
    at Function.Module.runMain (internal/modules/cjs/loader.js:832:11)
    at startup (internal/bootstrap/node.js:283:19)
    at bootstrapNodeJSCore (internal/bootstrap/node.js:622:3)

I went on the server on the registry container and I found this error:

Oct 26 15:02:51 edbb2ad0bb03 registry[1013]: 172.22.0.9 - - [26/Oct/2021:15:02:51 +0000] "POST /v2/v2/token/blobs/uploads/ HTTP/1.1" 401 272 "" "docker/20.10.8 go/go1.16.6 git-commit/75249d8 kernel/5.10.47-linux
Oct 26 15:02:51 edbb2ad0bb03 registry[1013]: time="2021-10-26T15:02:51.610354925Z" level=warning msg="error authorizing context: invalid token" go.version=go1.11.2 http.request.host=registry.mydomain.com http.request.id=2e57bd7b-5dbb-40
Oct 26 15:02:51 edbb2ad0bb03 registry[1013]: 172.22.0.9 - - [26/Oct/2021:15:02:51 +0000] "POST /v2/v2/token/blobs/uploads/ HTTP/1.1" 401 272 "" "docker/20.10.8 go/go1.16.6 git-commit/75249d8 kernel/5.10.47-linux

I am noticing the same error also on a local device:

[826]: [event]   Event: Docker image download {"image":{"name":"registry.mydomain.com/v2/token@sha256:d1ed6efeed649f2956e2c3a93108ecf2147e200988671e718dfb704ab6298808","app>
	resin-supervisor[1791]: [event]   Event: Docker image download {"image":{"name":"registry.mydomain.com/v2/token@sha256:d1ed6efeed649f2956e2c3a93108ecf2147e200988671e718dfb704ab6298808">
	balenad[826]: time="2021-10-27T07:55:55.529681875Z" level=info msg="Attempting next endpoint for pull after error: errors:\nunauthorized: authentication required\nunauthorized: authentication required\n"
	balenad[826]: time="2021-10-27T07:55:55.531437742Z" level=info msg="Ignoring extra error returned from registry: unauthorized: authentication required"
	balenad[826]: time="2021-10-27T07:55:55.532203046Z" level=error msg="Handler for POST /images/create returned error: unauthorized: authentication required"

I am both able to ping the registry and the api correctly and I am able to use the balena cli commands which interact with the devices.

I’ve tried to “ping” the registry directly with my ca.crt and it doesn’t give me any error

curl -I https://registry.domain.com/ --cacert ~/ca.crt

answer:

HTTP/1.1 200 OK

**Cache-Control** : no-cache

**Date** : Wed, 24 Nov 2021 15:35:28 GMT

I am able to generate a token as:

curl -H "Authorization: Bearer token-auth" "https://api.mydomain.com/auth/v1/token?service=registry.domain.com&scope=repository:v2/token" --cacert ~/ca.crt

but then pulling an image as:
curl -H "Authorization: Bearer <generated-token>" "https://registry.domain.com/v2/v2/token/tags/list" --cacert ~/ca.crt

I get:

{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"v2/token","Action":"pull"}]}]}

Any idea? There’s something I could do to start using the registry again?

Thanks

Matteo