x509 certificate issue, building on BalenaOS device, pushing to registry hosted in EC2

Hey folks,

Help Requested

How can I load the x509 ca.crt certificate from my openBalena registry into ssl certs store on a read-only balenaOS?

Context:

I have an openBalena instance running in AWS EC2. We are dependent on secrets during the build process so cannot build devices on EC2 or my local x86 development machine; we need a balenaEngine, which means we need a BalenaOS. So I built us a little raspberry pi build farm (farm of 1 :wink:) running on the lan, which after a moderate yak-shaving exercise does the build itself nicely.

The failing command is:

balena deploy balenatest -h balena.local --logs

Problem

When it gets to the deploy stage:

Error: Get https://registry.<our_private_domain>/v2/: x509: certificate signed by unknown authority

I have that certificate trusted in the keychain on my local development machine. And I have restarted Docker. But, as the build is actually using the balenaEngine on the build farm Pi, I am assuming that the deploy is actually happening there, and not my local machine.

Hence I want to get the certificate into the BalenaOS farm Pi’s certs store.

Things I have tried and checked

Obviously, as the pi is read-only, the standard linux commands for adding certs are telling me to get lost.

I’ve come across:
How to install a certificate in trusted - and am curious as to how this isn’t failing on the read-only fs at the first line.
Balena deploy -> Error: Get https://registry.iot.domain.edu/v2/: x509: certificate signed by unknown authority - but this is for builds on a local machine, and I have done all those steps to no avail.

I’ve scoured the doco and the forums. But I’m hoping I’ve missed a nugget somewhere.

Any ideas or doco I may have missed?

Many thanks in advance
Al

Hi,
This is pretty cool, what you build there, would love to see more of that pi build farm. :slight_smile:

I would build the application that performs the build as a docker application, and granting it access to the balena socket. In which case you can install the CA certificate into the cert store of the docker image itself and you don’t rely on the balenaOS cert store anymore. Additionally your build pipeline can become an application in your balena instances and updated in the same way as other applications.

Does this make sense for your use case?

That’s a great idea; I’m working on it now.

Do you know any good doco to follow to grant access to the balena socket for the docker container? I’m sure I’ll find it if I google hard enough, but a friendly pointer wouldn’t go amiss if you have one.

Cheers
Alastair

Hi, Can you clarify your question please: do you want to access the balena socket from inside the docker container?

Yes, I think so. That’s what I understood from @afitzek’s post at least.

You should be able to use the io.balena.features.balena-socket label in your docker compose file to bind mount the balena container engine socket into the container. You can read more in the docs here https://www.balena.io/docs/reference/supervisor/docker-compose/#labels

Fantastic, thanks :+1:

I’ve come to the conclusion that trying to run the actual cli commands from within a docker container on the farm pis is a rabbit hole;

  • source needs copying which opens a can of worms around dependencies and different project types, extra keys
  • secrets might need to be pushed across for git access etc

Instead of having one generic build farm, I would have to create a container purpose-built for each project I have. That is an untenable solution given the resources I have to put towards this. Thanks for the thought though. (I wonder how BalenaCloud does it…)

My original question therefore stands; how can I load the ca.crt into the balenaOS host, for use by the ‘remote build’ functionality of balena-cli when it tries to deploy the image to a private openBalena instance.

For anyone who comes across this, there is a reference in Using real (not self-signed) certificates around Mar '19. I haven’t actually worked out how to use that information, but it’s there.

for use by the ‘remote build’ functionality of balena-cli

Could you explain what you mean here? Do you mean the balena push {app} command?

Not Balena Cloud, no.

We love Balena Cloud, but need a secure, independent solution that can be ‘islanded’ and function without any connection to the internet. We want the ‘push’ workflow (with the build happening on a different server from the source code), but need to keep the whole process on our own servers.

So, we’re looking to create a build farm of Pis doing to actual build, with the source on another machine in the network, and openBalena on yet another machine also in the network. (Currently we’re running openBalena in EC2 just for development purposes, but ultimately all of this needs to be islanded).

So, something like balena push {app} -R <local_network_build_server> is the goal.

OK - so a couple of things come to mind;

We do offer balenaCloud in an “OnPrem” format, whereby you would run in in-house and have control over the access etc. This is not free like openBalena, but it does offer the same level of features as our public balenaCloud offering. Please contact us if you want to know more, pricing etc.

Regarding the remote build, like you see in balena push ... this is working because balenaCloud has a builder service which is not part of the openBalena stack. This service takes your source, builds the images, and pushes them to the registry for your device to pull down. We don’t prohibit openBalena having this kind of functionality but we don’t provide it either.

How can I load the x509 ca.crt certificate from my openBalena registry into ssl certs store on a read-only balenaOS?

Going back the original question, if the device is joined to the openBalena instance then it will have it anyway. Otherwise the way to override the CA certs on the device is via the config.json file; add a key to the default object for balenaRootCA with a value of your Root CA PEM in base64 encoded format. When the device boots, this value will be decoded and written into the root FS so that the device will trust any certs signed by that CA in the future.

Great, thanks. I’ll give that a go :+1:

Good to know about the onprem offering too. I didn’t see that in any of the docs though… hidden feature?

Not hidden, just not something applicable to the general user. Given your use-case though, it’s likely a good fit :+1: