push to registry error using open-balena v2.0

Hello guys,
I installed open-balena v2.0 (latest I guess) and when balena-cli is going to push the image it fails.
Looking to the journal of the openbalena-registry container into the server I found these errors.

It seems errors are coming from S3. Did anyone have same issue and/or have some hint to solve it?
if not should I revert back to openbalena v1.3?

Thanks in advance
Aleph

Dec 19 16:42:28 780aaf0e0642 registry[1013]: time=“2019-12-19T16:42:28.70269508Z” level=warning msg=“error authorizing context: authorization token required” go.version=go1.11.2 http.request.host=registry.cicd.conti.platform.aleph.solutions http.request.id=bad4b9cd-93d6-4d6b-9e32-de1471f6529e http.request.method=GET http.request.remoteaddr=35.237.184.55 http.request.uri="/v2/" http.request.useragent=“docker/19.03.5 go/go1.12.12 git-commit/633a0ea838 kernel/4.19.78-coreos os/linux arch/amd64”
Dec 19 16:42:28 780aaf0e0642 registry[1013]: 172.18.0.9 - - [19/Dec/2019:16:42:28 +0000] “GET /v2/ HTTP/1.1” 401 87 “” “docker/19.03.5 go/go1.12.12 git-commit/633a0ea838 kernel/4.19.78-coreos os/linux arch/amd64”
Dec 19 16:42:28 780aaf0e0642 registry[1013]: time=“2019-12-19T16:42:28.708176146Z” level=warning msg=“error authorizing context: authorization token required” go.version=go1.11.2 http.request.host=registry.cicd.conti.platform.aleph.solutions http.request.id=58fcfddc-c58f-4762-be0b-4f22c689c91d http.request.method=GET http.request.remoteaddr=35.237.184.55 http.request.uri="/v2/" http.request.useragent=“docker/19.03.5 go/go1.12.12 git-commit/633a0ea838 kernel/4.19.78-coreos os/linux arch/amd64”
Dec 19 16:42:28 780aaf0e0642 registry[1013]: 172.18.0.9 - - [19/Dec/2019:16:42:28 +0000] “GET /v2/ HTTP/1.1” 401 87 “” “docker/19.03.5 go/go1.12.12 git-commit/633a0ea838 kernel/4.19.78-coreos os/linux arch/amd64”
Dec 19 16:42:33 780aaf0e0642 registry[1013]: time=“2019-12-19T16:42:33.276086904Z” level=error msg=“response completed with error” auth.user.name=root err.code=unknown err.detail=“s3aws: SerializationError: failed to decode S3 XML error response
Dec 19 16:42:33 780aaf0e0642 registry[1013]: status code: 503, request id: , host id:
Dec 19 16:42:33 780aaf0e0642 registry[1013]: caused by: expected element type but have " err.message=“unknown error” go.version=go1.11.2 http.request.host=registry.cicd.conti.platform.aleph.solutions http.request.id=adb0b393-be54-4277-b1ef-0373fb37df07 http.request.method=HEAD http.request.remoteaddr=35.237.184.55 http.request.uri=”/v2/v2/f79215e5604e5b09d44e5814b1cd54c5/blobs/sha256:aa0d3d946c86fe24dc45e50d4eb09b4f208d7132aea38d27683ebc259eacc595" http.request.useragent=“docker/19.03.5 go/go1.12.12 git-commit/633a0ea838 kernel/4.19.78-coreos os/linux arch/amd64” http.response.contenttype=“application/json; charset=utf-8” http.response.duration=4.138633531s http.response.status=500 http.response.written=123 vars.digest=“sha256:aa0d3d946c86fe24dc45e50d4eb09b4f208d7132aea38d27683ebc259eacc595” vars.name=“v2/f79215e5604e5b09d44e5814b1cd54c5”

Hey,

Can you specify exactly what you tried to do please? It looks like the CLI was not authorized correctly with the registry, but with only these logs it’s fairly hard to say exactly why.

Hello, these are the steps I perfomed:
I had openbalena 1.0 on ubuntu server vm on azure.
I stopped openbalena.
I removed any container, image, volume.
I removed “config” folder inside openbalena.
I updated openbalena by “git pull” and I ran again quickstart.
I started openbalena.
I created one new application.
I registered one new rpi3 device.
I ran balena deploy and I’ve got error when balena-cli tried to push image to registry.
I connected via ssh to the server.
I ran “docker exec -it openbalena_registry_1 journalctl” and I saw errors I put on my previous post.

hi @aleph.solutions the first thing that comes to mind to check is if you updated the CLI as well for the new version? That is the first thing I would try as its possible the client can’t authenticate correctly or is only working partially. Could you give that a try?