Hi, I have configured an OpenBalena server to use AWS S3 as the registry but when I try to deploy an app it gives a certificate error when trying to push to the registry:
Do we need to used a signed cert to be able to use AWS S3 as the registry or is there something else that could be causing this issue?
Hey @g.corrigan, can you please expand on how you configured openBalena? Did you have an openBalena instance running already before you configured the registry to point to AWS S3?
Have you seen this thread where another openBalena user and my teammate explain how to use S3 as registry?
Indeed you need install signed certificates. Please let us know the full steps you took to install the certificates on your system. You may also consider re-installing from scratch by following our Getting Started guide with the configuration sprinkled on this thread I linked – just to make sure that you’re starting from a clean slate.
Cheers…
Hi @gelbal, thanks for the quick response. I have been following the thread that you linked to but i’m using the self singed certs automatically created by the quickstart script.
The server was cloned from our working server so I could test the s3 migration. I performed the following steps:
Can you confirm that using the self signed certs created by quickstart does not work for using AWS S3 for the registry?
When you say I need to use signed certs, does this mean I need to create certs using Letsencrypt or something similar?
Do I then place these in the certs directory and run quickstart again? I also saw the -c option for quickstart but not sure exactly of its use.
Thanks,
Gerard.
Hi,
I think we need to have the self-signed cert added to keychain (if you are on mac) for this to work. Following is the command to try this out:
sudo security add-trusted-cert -d -r trustRoot -k "/Library/Keychains/System.keychain" "%s/root/ca.crt"\n' "${CERTS_DIR}"
Ref: open-balena/quickstart at master · balena-io/open-balena · GitHub
Also, can you share the full log output, will help understand what all steps were completed (to retrace the flow).
Cheers.
Hi @nitish I’m installing it on Ubuntu.
I’m going to do a clean install on a new Ubuntu instance this evening. I’ll capture all the output and post here.
One question though, you indicate in your post to use a self signed cert but @gelbal indicated that a signed cert is required in order to use AWS S3 as the registry storage?
If I add the -c option to the quickstart script on a new instance will it create signed certs automatically?
Tonight I built a new openBalena server, selected the -c option in quickstart to create signed certs and i’m still getting the same issue:
Retrying “registry.vxxxxi.ch/v2/f8af1503134d1a60bee06a7e835df6c5:latest” after 2.0s (1 of 2) due to: Error: Get https://registry.vxxxxi.ch/v2/: x509: certificate signed by unknown authority
See attached full output from all my actions in setting up the new server, connecting to it and attempting to deploy and app.
openbalena_config.log (23.8 KB)
Hi @g.corrigan, first of all, let me clear confusion: Self-signed certificate should be fine. After reading your following comment, I realized that you were asking of self-signed vs trusted CA signed certificate.
Thanks for explaining how you do the setup and also providing detailed logs. What jumps to me from the logs is the device’s balenaOS version: v2.47.1+rev1. I wonder if what you experience is a compatibility issue. Please see:
Can you please try a device with the latest balenaOS and also use the latest balena-cli?
(assuming that you are running the latest openBalena version)
Hi @gelbal i’m using the latest balena-cli but will check on the OS version and build a device with that to see if there is any difference.
Hi @gelbal and @nitish, I flashed an Intel Nuc device with the latest image and i’m still getting the same error x509 cert error.
[Info] No “docker-compose.yml” file found at “/home/gerard/balena-cli”
[Info] Creating default composition with source: “/home/gerard/balena-cli”
[Info] Everything is up to date (use --build to force a rebuild)
[Info] Creating release…
[Info] Pushing images to registry…
Retrying “registry.visoai.ch/v2/6d5e2816b50fd5bde90200600f894d2d:latest” after 2.0s (1 of 2) due to: Error: Get https://registry.vxxxxi.ch/v2/: x509: certificate signed by unknown authority (possibly because of “crypto/rsa: verification error” while trying to verify candidate authority certificate “ca.vxxxxi.ch”)
Retrying “registry.vxxxxi.ch/v2/6d5e2816b50fd5bde90200600f894d2d:latest” after 2.8s (2 of 2) due to: Error: Get https://registry.vxxxxi.ch/v2/: x509: certificate signed by unknown authority (possibly because of “crypto/rsa: verification error” while trying to verify candidate authority certificate “ca.vxxxxi.ch”)
[Info] Saving release…
[Error] Deploy failed
Get https://registry.vxxxxi.ch/v2/: x509: certificate signed by unknown authority (possibly because of “crypto/rsa: verification error” while trying to verify candidate authority certificate “ca.vxxxxi.ch”)
Additional information may be available with the --debug flag.
For further help or support, visit:
Hi, were you able to restart docker after installing the certificates? Rebooting your your new open-balena server should also do the trick. The quickstart script doesn’t restart it by default but it should warn about it at the end.
You will need to restart your Docker daemon after trusting this certificate to allow your workstation to push images to the registry.
@cmfcruz yes, i’ve rebooted both the server machine and the machine with the balena-cli. Doesn;t make any difference.
Hi @gelbal @nitish @cmfcruz I finally got past the certificate issue. The problem was with Windows WSL2, I had to trust the cert in Windows as well as Linux!
Having got past that i’m getting a 500 Internal Server Error when it tries to push to the registry. I have installed the AWS cli on the openBalena sever machine and confirm that I can write and read from the AWS S3 with the same Access and Secret keys that i’ve added to the openBalena config.
I’ve tried tailing the logs from the various docker containers but they dont seem to be outputting anything. I’ve also run tcpdump on the sever. I can see traffic coming from the balena-cli machine and can see packets to and from AWS S3 - s3.eu-central-1.amazonaws.com.https so it looks like it is trying to the use the external bucket but nothing is written to it.
Any thoughts on what may be the issue?
gerard@GC-Office-Desktop:~/balena-cli$ ./balena deploy virt-balena-dev --debug --logs
[debug] new argv=[/home/gerard/balena-cli/balena,/snapshot/versioned-source/bin/balena,deploy,virt-balena-dev,–logs] length=5
[Debug] Parsing input…
[Debug] Loading project…
[Debug] Resolving project…
[Info] No “docker-compose.yml” file found at “/home/gerard/balena-cli”
[Info] Creating default composition with source: “/home/gerard/balena-cli”
[Debug] Creating project…
[Info] Everything is up to date (use --build to force a rebuild)
[Info] Creating release…
[Debug] Tagging images…
[Debug] Authorizing push…
[Info] Pushing images to registry…
Retrying “registry.vxxxxi.ch/v2/9e82d004f054d1255a2a03f2228b6ec7:latest” after 2.0s (1 of 2) due to: Error: received unexpected HTTP status: 500 Internal Server Error
Retrying “registry.vxxxxi.ch/v2/9e82d004f054d1255a2a03f2228b6ec7:latest” after 2.8s (2 of 2) due to: Error: received unexpected HTTP status: 500 Internal Server Error
[Debug] Saving image registry.visoai.ch/v2/9e82d004f054d1255a2a03f2228b6ec7
[Debug] Untagging images…
[Info] Saving release…
[Error] Deploy failed
received unexpected HTTP status: 500 Internal Server Error
Error: received unexpected HTTP status: 500 Internal Server Error
at Stream. (/snapshot/versioned-source/node_modules/docker-progress/index.js:53:19)
at Stream.emit (events.js:315:20)
at Stream.EventEmitter.emit (domain.js:482:12)
at drain (/snapshot/versioned-source/node_modules/through/index.js:36:16)
at Stream. (/snapshot/versioned-source/node_modules/through/index.js:45:5)
at Parser.onToken (/snapshot/versioned-source/node_modules/JSONStream/index.js:132:18)
at Parser.write (/snapshot/versioned-source/node_modules/jsonparse/jsonparse.js:135:34)
at Stream. (/snapshot/versioned-source/node_modules/JSONStream/index.js:23:12)
at Stream.write (/snapshot/versioned-source/node_modules/through/index.js:26:11)
at IncomingMessage.ondata (_stream_readable.js:717:22)
at IncomingMessage.emit (events.js:315:20)
at IncomingMessage.EventEmitter.emit (domain.js:482:12)
at addChunk (_stream_readable.js:295:12)
at readableAddChunk (_stream_readable.js:271:9)
at IncomingMessage.Readable.push (_stream_readable.js:212:10)
at HTTPParser.parserOnBody (_http_common.js:132:24)
From previous event:
at awaitRegistryStream (/snapshot/versioned-source/node_modules/docker-progress/index.js:43:12)
at /snapshot/versioned-source/node_modules/docker-progress/index.js:416:16
at processImmediate (internal/timers.js:456:21)
at process.topLevelDomainCallback (domain.js:137:15)
From previous event:
at DockerProgress.push (/snapshot/versioned-source/node_modules/docker-progress/index.js:415:56)
at func (/snapshot/versioned-source/build/utils/compose.js:230:34)
at retry (/snapshot/versioned-source/build/utils/helpers.js:145:18)
at async Promise.all (index 1)
at async Promise.all (index 0)
For further help or support, visit:
Does anyone have any ideas on the best methods for troubleshooting the the Internal Server Error above? All containers seems to be running infe and i’m see any errors in container logs.
Thanks,
Gerard.
We have the very same issue with our openBalena server.
Can anyone take this to resolve asap?
Cheers!
Can you please get logs from the registry and API services?
Hi, I would like to follow up on this, were you able to solve the issue? Otherwise could you please check the logs from registry and API services as my colleague suggested so that we can debug further? Thanks.
Hi @mtoman @dfunckt aplogies for the late response and thanks for getting back to me. I only got back to this yesterday. I got it sorted, the issue was due to following the instructions in a previous forum post where it listed the changes to be made in the make-env script, it said to change the OPENBALENA_S3_BUCKETS variable but hadn’t mentioned that the OPENBALENA_REGISTRY2_S3_BUCKET needed to be set as well. Once I set that to the correct bucket name it worked.
Just out of interest in case I need it for future reference can you tell me how to enable debug logging for the API and Registry containers? When i run docker logs against those two containers there is always just one line?
balena@ip-10-xx-x-xx:~/open-balena$ docker logs 3f727a57f0ad
Systemd init system enabled.
@g-corrigan you can access runtime logs using docker exec -it <name or ID> journalctl -u open-balena-api -f command. To have systemd inside containers to output logs to stdout, so that these can be parsed with docker logs ..., I believe you need to edit your docker-compose file and add tty: true under each service. One or both of these should work.
Thanks @ab77