PSA: open-balena certificates expiration management

openBalena
1

Hi there, I’m doing this post to better inform the community and also ask for validation.
We have been using openBalena for a while now (it’s an awesome product btw!) but our concerns recently shifted to SSL certificates expiration. As we know, API and VPN certs are created with a validity of 2 years which is not a lot. We did several attempts to ensure the certificate renewal process would complete without any problems as the whole system depends on it (correct me if I’m wrong).
We did different tests and I thought it may be useful to share them with the community.

Let’s Encrypt automatic certificate renewal
We tried to use -c parameter in the quickstart command during open-balena installation but we had no luck: the cert-provider always reported an error of some kind, never leaving the staging phase. Reading this forum we did some progress changing ACME version pulled from github, commenting two line in the .sh script but after all, we weren’t able to accomplish a self-renewing certificate, even with different “hacks” which didn’t give us much confidence, to begin with.

Manually renewing the certificates
We tried to manually renew the server certificate using the automatically generated CA (which you can find in open-balena/config/certs/root) and we were able to “install” it in open-balena placing the .pem file in open-balena/src/haproxy/open-balena.pem and rebuilding the container. With this procedure, we were able to make the HTTPS API be served with a newly generated certificate with a custom expiration date. This procedure, although kinda hacky, seemed to be working, but we had no idea what should have been done VPN-wise as we saw there is a dedicated folder in open-balena/config/certs/vpn. Since this way still seemed too hacky and we still had doubts about the VPN we tried a different route.

Changing certificates duration upon quickstart
We noticed the ssl-common.sh (which is invoked in the quickstart process) in open-balena source files contains a couple of useful variables:

# global expiry settings
CA_EXPIRY_DAYS=3650
CRT_EXPIRY_DAYS=730

Changing those variables to a greater timespan and re-running the quickstart from scratch (after a cleanup of docker volumes) did produce an open-balena setup with a custom certificates expiration and, honestly, it actually seems a pretty solid solution. AFAIK this is valid only for a fresh start of the system and cannot be done for an existing and running instance. Of course, an 500y lasting certificate may not be great from a security point of view but it’s still better than a certificate expiring in 2 years without a proper way to renew it without using some hack.

Conclusion
We know open-balena is marked as not production-ready, so we completely understand automated LE certificates renewal process may not be fully working flawlessly yet and that’s ok. What we did find quite frustrating was how difficult changing the certificate’s validity seemed to be. Let’s go straight to the point: if anyone uses open-balena as it is without digging deep inside the forum, is basically using a demo version of the software which is set to expire after 2 years (correct me if I’m wrong here). The solution is there: just change the ssl-common.sh variables but it’s actually not very well documented and, since it’s something which must be set upon initial installation, I think certificates management with possible solutions are worth mentioning in the getting-started section.

Please don’t take this post as an attack. We just wanted to add our perspective and highlight a possible concern also giving future users a hint about what can be done. Balena ecosystem is an amazing world we all love, I’m just trying to make life easier for future users!

3 Likes
2

Hi @AndreaAlgeri

You can renew the certificates easily enough.

This is a procedure I documented for myself a while back. Its included in a post on here somewhere:

  1. Rename the following files in the config/certs/vpn folder to create a backup.
  • issued/vpn.mydomain.com.crt
  • private/vpn.mydomain.com.key
  • reqs/vpn.mydomain.com.req
  • index.txt
  1. Download easy-rsa to a temporary folder and extract it with following command:

$ curl -sL https://github.com/OpenVPN/easy-rsa/releases/download/v3.0.5/EasyRSA-nix-3.0.5.tgz | tar xz --strip-components=1

  1. Create new certificate:

$ ./easyrsa/easyrsa --pki-dir=“./vpn” --days=730 build-server-full “vpn.mydomain.com” nopass

  1. Convert certificate and the key-file base64 string:
$ echo "$(cat ./vpn/issued/vpn.mydomain.com.crt)" | base64 --wrap=0 2>/dev/null
$ echo "$(cat ./vpn/private/vpn.mydomain.com.key)" | base64 --wrap=0 2>/dev/null

  1. Replace the values of the keys OPENBALENA_VPN_SERVER_CRT and OPENBALENA_VPN_SERVER_KEY in config/activate with base64 encoded values from step 4.

  2. Restart open balena and recreate services.

$ ./scripts/compose up -d --force-recreate --no-deps

No warranty provided obviously but I have done this a couple of time and not had any impacts on fleet devices.

EDIT: I think this works in pepertuality provided you always generate the new certificates with the same CA. This because the devices have the CA set in their config.json file to trust. You could of course change the CA in the config.json of all fleet devices if you really had to using a script. Theres an example of how to bulk edit config.json of fleet devices on the balena GitHub. Would avoid that unless truly necessary.

Cheers
Dashals

2 Likes
3

Hi @dash
Thank you for your input, we actually stumbled across your solution while reading the forums. We didn’t have the time to test the solution and I don’t doubt it’s gonna work. I still have some doubts:

  • I’m no certificates expert, but the procedure you describe seems to be working on VPN’s certificate only. Is this correct? What about the API (HTTPs) cert?
  • CA has a default expiration of 10 years. I know this may seem like a lot, but from a project management perspective, it’s a critical constraint. I know you can update CA remotely, but it doesn’t feel like a trivial (and well documented) procedure as you say.

The point of my post, however, was more focused on how these procedures (like the one you described) are not very well documented and an open-balena user must go through the different forum posts in order to understand how to manage certificates correctly (which is something crucial for the system).

1 Like
4

Hi @AndreaAlgeri

This method will work for any domain certificate required by openBalena - just change the domain as required.

Changing the CA is non-trival but definitely manageable using this tool: GitHub - balena-os/configizer: Safe(r) balenaOS config.json updates remotely to automate updating CA on remote devices.

I have actually submitted an issue to create a community managed documentation repo that we can contribute to. Unfortunately this hasn’t been actioned yet. Once/if this occurs I have a collection of
docs which covers all of the scenarios I have encounter in running it for three years.

Cheers,
Dash

11

Thanks!

On steps 2-3, the curl may need to be done inside a separately created easyrsa directory, and here is a version of step 3 with the quotation characters fixed:

$ ./easyrsa/easyrsa --pki-dir="./vpn" --days=730 build-server-full "vpn.mydomain.com" nopass

easy-rsa may also complain if index.txt does not exist. Overall, the process works well for vpn. Some adjustment might be needed for api certs, which appear to have a different file structure in my case.

12

@dash’s method adjusted for the root certificate (top priority when the CLI says CERT_HAS_EXPIRED: request to https://api.mydomain.com/login_ failed, reason: certificate has expired or devices stop reporting state information).

  1. Rename the following files in config/certs/root to create a backup (they must not exist when easyrsa runs):

    root/issued/*.mydomain.com.crt (signed certificate)
    root/private/*.mydomain.com.key (private key)
    root/reqs/*.mydomain.com.req (requests)
    root/index.txt (certificate database)

  2. Basically the same thing; executable path might depend on the zip file.

$ mkdir easyrsa;
$ cd easyrsa;
$ curl -sL https://github.com/OpenVPN/easy-rsa/releases/download/v3.0.5/EasyRSA-nix-3.0.5.tgz | tar xz --strip-components=1;
$ cd ..;
  1. $ ./easyrsa/easyrsa --pki-dir="./root" --days=730 build-server-full "*.mydomain.com" nopass
  2. Echo the revised path
$ echo "$(cat ./root/issued/*.mydomain.com.crt)" | base64 --wrap=0 2>/dev/null
$ echo "$(cat ./root/private/*.mydomain.com.key)" | base64 --wrap=0 2>/dev/null
  1. Replace OPENBALENA_ROOT_CRT and OPENBALENA_ROOT_KEY. The start/end of the new strings may look like the old one, but they do differ.
  2. Same container rebuild:
$ ./scripts/compose up -d --force-recreate --no-deps
2 Likes
14

I tried @MBer’s method for renewing root cert but always get
Easy-RSA error: signing failed

with no usefull errors
has anyone successfully change root certificate?

15

For anyone wondering.
I redeployed the balena server and used -c flag on quickstart so that it uses ACME certificates.

Which meant I had to put new images on devices. Luckily we didn’t have many yet.

16

I just walked through my process for another renewal and it worked, but a lot could depend on which folder you are in when running the commands (my post does not cover all folder navigation steps) and various details of your system and the existing certificates.

1 Like
17

This one took me by surprise recently. Thanks to @AndreaAlgeri for their original PSA and also @dash and @MBer for their help/follow-ups!

This was able to get me to the point where I could reach my devices again, but I still ran into two issues:

  1. I was unable to deploy to my devices. I don’t have the error handy, but the registry (that is, registry.mydomain.com) was complaining there was no Subject Alternative Name in the certificate.
  2. Comparing the new certificate to the original, I also found the new certificate was using a smaller key (2048 bit) compared to the original’s 4096 bit.

The solution to both was to update the easyrsa command as follows:

./easyrsa/easyrsa --keysize=4096 --subject-alt-name='DNS:*.mydmain.com' --pki-dir="./root" --days=1001 build-server-full '*.mydmain.com' nopass

I keep detailed project notes for things like this to ensure that future me (he’s likable enough, but has an awful memory) will be able to reproduce/automate any particular solution. For what it’s worth, here is that section from my personal notes covering the step-by-step process to resolve certificate expiration:

OpenBalena Certificate Expiration

There is a CA cert that lives for 10 years, along with a ‘root’ and ‘vpn’ certificates that live for only 2 yeaers. These will need to be renewed. The following is the process for those renewals.

Go to the source

sudo su
cd /home/balena/open-balena

Back it up

cp -a config config.bak

Go to the certs

cd config/certs

Clean out old stuff

rm 'root/issued/*.mydomain.com.crt'
rm 'root/reqs/*.mydomain.com.req'
rm 'root/private/*.mydomain.com.key'
rm 'root/index.txt'
rm 'vpn/issued/vpn.mydomain.com.crt'
rm 'vpn/reqs/vpn.mydomain.com.req'
rm 'vpn/private/vpn.mydomain.com.key'
rm 'vpn/index.txt'

Get Easy-RSA

mkdir easyrsa
cd easyrsa
curl -sL https://github.com/OpenVPN/easy-rsa/releases/download/v3.2.0/EasyRSA-3.2.0.tgz | tar xz --strip-components=1
cd ..

Create new certificate

./easyrsa/easyrsa --keysize=4096 --subject-alt-name='DNS:*.mydmain.com' --pki-dir="./root" --days=730 build-server-full '*.mydmain.com' nopass
  > "Confirm OVER-WRITE existing inline file ?"
    > y
  > "Confirm request details:"
    > yes
  > "Enter pass phrase for ./root/private/ca.key:"
    > <password>

./easyrsa/easyrsa --keysize=4096 --subject-alt-name='DNS:vpn.mydmain.com' --pki-dir="./vpn" --days=730 build-server-full 'vpn.mydmain.com' nopass
  > "Confirm OVER-WRITE existing inline file ?"
    > y
  > "Confirm request details:"
    > yes
  > "Enter pass phrase for ./vpn/private/ca.key:"
    > <password>

Optionally view each key

openssl x509 -noout -text -in 'root/issued/*.mydomain.com.crt'
openssl x509 -noout -text -in 'vpn/issued/vpn.mydomain.com.crt'

> Check each for a 4096 bit public key
> Check each for an X509v3 extension "X509v3 Subject Alternative Name"

Convert certificate to a key-file base64 string

echo "$(cat './root/issued/*.mydomain.com.crt')" | base64 --wrap=0 2>/dev/null && echo
echo "$(cat './root/private/*.mydomain.com.key')" | base64 --wrap=0 2>/dev/null && echo
echo "$(cat './vpn/issued/vpn.mydomain.com.crt')" | base64 --wrap=0 2>/dev/null && echo
echo "$(cat './vpn/private/vpn.mydomain.com.key')" | base64 --wrap=0 2>/dev/null && echo

Move to the OpenBelana directory

cd /home/balena/open-balena

Update the certificates

edit config/activate
   > Replace the following values using the echod base64 strings from above (in matching order)
     > OPENBALENA_ROOT_CRT
     > OPENBALENA_ROOT_KEY
     > VPN_SERVER_CRT
     > VPN_SERVER_KEY

Restart OpenBalena

./scripts/compose up -d --force-recreate --no-deps

Quick verification of the new certificate expiration. Dump the certificate used by haproxy and scan the first dozen lines of output for the NOT BEFORE and NOT AFTER lines:

docker exec -it openbalena_haproxy_1 cat /etc/ssl/private/open-balena.pem
1 Like