OpenBalena API unavailable after pointing to external S3 bucket for registry storage

openBalena
, ,
1

Hi everyone :smiling_face:

I’m working with OpenBalena v3.8.2 running on an EC2 Ubuntu instance, and the only balena-cli version that I got to be working properly with this OpenBalena version is v15.2.0. I always tested OpenBalena with the standard internal registry container and volume (MinioS3), and everything ran nice and smoothly, but after a certain number of deployments it’s starting to fill the VM disk space.

I thought moving the registry to our external S3 bucket would have been a good idea, as shown in these threads:
https://forums.balena.io/t/using-aws-s3-as-the-registry-storage-for-openbalena/265674
https://forums.balena.io/t/configure-to-use-aws-s3-and-rds/32030

Also, I found this blog article:
https://blog.balena.io/how-to-use-external-aws-s3-provider-openbalena/

I have tried both methods, but I find the second one to be a bit cleaner. In any case, none of them works as it should.

In particular, via the “make-env” method (threads), I get a 500 Internal Server error deploying images to the registry, as shown below:

[Info]    Pushing images to registry...
Retrying "registry.***.***.***/v2/e853d196747542ec678f28bf89af0431:latest" after 2.0s (1 of 2) due to: Error: received unexpected HTTP status: 500 Internal Server Error
Retrying "registry.***.***.***/v2/69a5f743d9128834ac86bff0baa66fe0:latest" after 2.0s (1 of 2) due to: Error: received unexpected HTTP status: 500 Internal Server Error
Retrying "registry.***.***.***/v2/e853d196747542ec678f28bf89af0431:latest" after 2.8s (2 of 2) due to: Error: received unexpected HTTP status: 500 Internal Server Error
Retrying "registry.***.***.***/v2/69a5f743d9128834ac86bff0baa66fe0:latest" after 2.8s (2 of 2) due to: Error: received unexpected HTTP status: 500 Internal Server Error
[Debug]   Saving image registry.***.***.***/v2/e853d196747542ec678f28bf89af0431
[Debug]   Untagging images...
[Info]    Saving release...
[Error]   Deploy failed
received unexpected HTTP status: 500 Internal Server Error

Error: received unexpected HTTP status: 500 Internal Server Error
    at Stream.<anonymous> (/usr/local/lib/balena-cli/node_modules/docker-progress/build/index.js:31:27)
    at Stream.emit (events.js:400:28)
    at Stream.emit (domain.js:475:12)
    at drain (/usr/local/lib/balena-cli/node_modules/through/index.js:36:16)
    at Stream.stream.queue.stream.push (/usr/local/lib/balena-cli/node_modules/through/index.js:45:5)
    at Parser.parser.onToken (/usr/local/lib/balena-cli/node_modules/JSONStream/index.js:132:18)
    at Parser.proto.write (/usr/local/lib/balena-cli/node_modules/jsonparse/jsonparse.js:135:34)
    at Stream.<anonymous> (/usr/local/lib/balena-cli/node_modules/JSONStream/index.js:23:12)
    at Stream.stream.write (/usr/local/lib/balena-cli/node_modules/through/index.js:26:11)
    at IncomingMessage.ondata (internal/streams/readable.js:731:22)
    at IncomingMessage.emit (events.js:400:28)
    at IncomingMessage.emit (domain.js:475:12)
    at addChunk (internal/streams/readable.js:293:12)
    at readableAddChunk (internal/streams/readable.js:267:9)
    at IncomingMessage.Readable.push (internal/streams/readable.js:206:10)
    at HTTPParser.parserOnBody (_http_common.js:140:24)

For further help or support, visit:
https://www.balena.io/docs/reference/balena-cli/#support-faq-and-troubleshooting
2

Using the second method (the blog one), editing the config/docker-compose.yml as shown below:

screen1
screen11076×678 130 KB

3

I get:

screen2
screen21418×286 46.6 KB

and my CLI gets completely disconnected from the OpenBalena server.

4

Searching in the cert_provider server logs, I found this:

screen3
screen32114×484 175 KB

I tried not to force-stopping the standard Minio S3 entrypoint (as shown in the blog), preventing open-balena-s3 container to exit, and everything seems to be reachable again, but as soon as I try to deploy something I get stuck with the same 500 Internal Server Error as shown above for the first case.

Also, as stated by a user in one of the above threads, I successfully pushed a complete backup of the whole openbalena registry (/data folder, approx. 7GB) from the OpenBalena server to our external S3 bucket via aws-cli, but it still throws the same error…

What am I doing wrong? I have run out of ideas :disappointed: