PSA: open-balena certificates expiration management

Hi there, I’m doing this post to better inform the community and also ask for validation.
We have been using openBalena for a while now (it’s an awesome product btw!) but our concerns recently shifted to SSL certificates expiration. As we know, API and VPN certs are created with a validity of 2 years which is not a lot. We did several attempts to ensure the certificate renewal process would complete without any problems as the whole system depends on it (correct me if I’m wrong).
We did different tests and I thought it may be useful to share them with the community.

Let’s Encrypt automatic certificate renewal
We tried to use -c parameter in the quickstart command during open-balena installation but we had no luck: the cert-provider always reported an error of some kind, never leaving the staging phase. Reading this forum we did some progress changing ACME version pulled from github, commenting two line in the .sh script but after all, we weren’t able to accomplish a self-renewing certificate, even with different “hacks” which didn’t give us much confidence, to begin with.

Manually renewing the certificates
We tried to manually renew the server certificate using the automatically generated CA (which you can find in open-balena/config/certs/root) and we were able to “install” it in open-balena placing the .pem file in open-balena/src/haproxy/open-balena.pem and rebuilding the container. With this procedure, we were able to make the HTTPS API be served with a newly generated certificate with a custom expiration date. This procedure, although kinda hacky, seemed to be working, but we had no idea what should have been done VPN-wise as we saw there is a dedicated folder in open-balena/config/certs/vpn. Since this way still seemed too hacky and we still had doubts about the VPN we tried a different route.

Changing certificates duration upon quickstart
We noticed the ssl-common.sh (which is invoked in the quickstart process) in open-balena source files contains a couple of useful variables:

# global expiry settings
CA_EXPIRY_DAYS=3650
CRT_EXPIRY_DAYS=730

Changing those variables to a greater timespan and re-running the quickstart from scratch (after a cleanup of docker volumes) did produce an open-balena setup with a custom certificates expiration and, honestly, it actually seems a pretty solid solution. AFAIK this is valid only for a fresh start of the system and cannot be done for an existing and running instance. Of course, an 500y lasting certificate may not be great from a security point of view but it’s still better than a certificate expiring in 2 years without a proper way to renew it without using some hack.

Conclusion
We know open-balena is marked as not production-ready, so we completely understand automated LE certificates renewal process may not be fully working flawlessly yet and that’s ok. What we did find quite frustrating was how difficult changing the certificate’s validity seemed to be. Let’s go straight to the point: if anyone uses open-balena as it is without digging deep inside the forum, is basically using a demo version of the software which is set to expire after 2 years (correct me if I’m wrong here). The solution is there: just change the ssl-common.sh variables but it’s actually not very well documented and, since it’s something which must be set upon initial installation, I think certificates management with possible solutions are worth mentioning in the getting-started section.

Please don’t take this post as an attack. We just wanted to add our perspective and highlight a possible concern also giving future users a hint about what can be done. Balena ecosystem is an amazing world we all love, I’m just trying to make life easier for future users!

3 Likes

Hi @AndreaAlgeri

You can renew the certificates easily enough.

This is a procedure I documented for myself a while back. Its included in a post on here somewhere:

  1. Rename the following files in the config/certs/vpn folder to create a backup.
  • issued/vpn.mydomain.com.crt
  • private/vpn.mydomain.com.key
  • reqs/vpn.mydomain.com.req
  • index.txt
  1. Download easy-rsa to a temporary folder and extract it with following command:

$ curl -sL https://github.com/OpenVPN/easy-rsa/releases/download/v3.0.5/EasyRSA-nix-3.0.5.tgz | tar xz --strip-components=1

  1. Create new certificate:

$ ./easyrsa/easyrsa --pki-dir="./vpn" --days=730 build-server-full “vpn.mydomain.com” nopass

  1. Convert certificate and the key-file base64 string:
$ echo "$(cat ./vpn/issued/vpn.mydomain.com.crt)" | base64 --wrap=0 2>/dev/null
$ echo "$(cat ./vpn/private/vpn.mydomain.com.key)" | base64 --wrap=0 2>/dev/null
  1. Replace the values of the keys OPENBALENA_VPN_SERVER_CRT and OPENBALENA_VPN_SERVER_KEY in config/activate with base64 encoded values from step 4.

  2. Restart open balena and recreate services.

$ ./scripts/compose up -d --force-recreate --no-deps

No warranty provided obviously but I have done this a couple of time and not had any impacts on fleet devices.

EDIT: I think this works in pepertuality provided you always generate the new certificates with the same CA. This because the devices have the CA set in their config.json file to trust. You could of course change the CA in the config.json of all fleet devices if you really had to using a script. Theres an example of how to bulk edit config.json of fleet devices on the balena GitHub. Would avoid that unless truly necessary.

Cheers
Dashals

Hi @dash
Thank you for your input, we actually stumbled across your solution while reading the forums. We didn’t have the time to test the solution and I don’t doubt it’s gonna work. I still have some doubts:

  • I’m no certificates expert, but the procedure you describe seems to be working on VPN’s certificate only. Is this correct? What about the API (HTTPs) cert?
  • CA has a default expiration of 10 years. I know this may seem like a lot, but from a project management perspective, it’s a critical constraint. I know you can update CA remotely, but it doesn’t feel like a trivial (and well documented) procedure as you say.

The point of my post, however, was more focused on how these procedures (like the one you described) are not very well documented and an open-balena user must go through the different forum posts in order to understand how to manage certificates correctly (which is something crucial for the system).

1 Like

Hi @AndreaAlgeri

This method will work for any domain certificate required by openBalena - just change the domain as required.

Changing the CA is non-trival but definitely manageable using this tool: GitHub - balena-os/configizer: Safe(r) balenaOS config.json updates remotely to automate updating CA on remote devices.

I have actually submitted an issue to create a community managed documentation repo that we can contribute to. Unfortunately this hasn’t been actioned yet. Once/if this occurs I have a collection of
docs which covers all of the scenarios I have encounter in running it for three years.

Cheers,
Dash