Certificates of openBalena

bversluijs · November 12, 2020, 10:02am

Hi all,

I’ve been busy with openBalena for a while and this time I have a question about the certificates of openBalena.
When creating an openBalena instance using ./scripts/quickstart, some certificates are being generated. These are used in the following environment variables (afaik):

OPENBALENA_ROOT_CA
OPENBALENA_ROOT_CRT
OPENBALENA_ROOT_KEY

OPENBALENA_VPN_CA
OPENBALENA_VPN_CA_CHAIN
OPENBALENA_VPN_SERVER_CRT
OPENBALENA_VPN_SERVER_KEY
OPENBALENA_VPN_SERVER_DH

I have limited knowledge about certificates and I’d like to know more about the openBalena certificates. I’ve only worked with SSL certificates for HTTPS connections. Now, I know that openBalena uses HTTPS, and I use Let’s Encrypt for the HTTPS endpoints instead of the generated certificates, so I don’t get any errors while trying to connect. However, the VPN endpoint and VPN certificates don’t use Let’s Encrypt, because these are other certificates.

As far as I know, certificates have an expiry date. Let’s Encrypt uses 90 days by default and other SSL providers use 1 year. But what’s the expiry date on the openBalena certificates, like the VPN? And if they expire, what happens next? And is there a way to create new certificates for devices or something?

I’m asking this because I don’t want to know about the expiry after they’ve expired .

Thanks in advance!

sradevski · November 12, 2020, 10:21am

Hey Bart, let me double-check with the team just so I don’t give you inaccurate information and we’ll get back to you.

richbayliss · November 12, 2020, 11:10am

Hi Bart,

So the defaults for the certificate generation are found here: https://github.com/balena-io/open-balena/blob/709d00b898dc7503baf1275c58ae4d0e95544d66/scripts/ssl-common.sh#L23-L25; The CA is 10 years and the individual certificates are 2 years.

The VPN configuration is sent to the device from the API via the /os/v1/config endpoint (here is the balenaCloud one https://api.balena-cloud.com/os/v1/config) and the value in that JSON are coming from the environment variables that the API instance is running with. As long as the CA certificate which signed the VPN’s server certificate (set via an env var on the VPN service) matches the one delivered by the API then it should be good.

To answer your question about renewals, right now openBalena doesn’t renew the VPN or API certs, but the Let’s Encrypt certs DO get renewed via the CertProvider service.

bversluijs · November 12, 2020, 3:15pm

Hi Rich,

Thanks for your answer. So in other words, the VPN certificate does expire, but it’s not really a problem because they’ll match and thus trust each other?

And openBalena doesn’t autorenew it’s certificates (except the Let’s Encrypt certs), but it is possible to renew them manually and inform the devices about it?

Thanks!

georgiats · November 13, 2020, 3:33pm

Hey Bart,
We generate a CA and certs for the VPN & VPN Server, and these are set in environment vars for the API and VPN services. The CA should last a long time, and the server cert is signed by that CA cert so if it expired then you would make a new one and update the ENV vars.

Hope this answers your question.
Please let me know if you need any further help,
Georgia

Steve · November 18, 2020, 2:27am

On a similar topic of certs, I downloaded the generated ca.crt to my device, then set the NODE_EXTRA_CA_CERTS env variable.

I can use balena login fine, but when I try a balena deploy myapp, it ends up failing with:

[Error]   Deploy failed
Get "https://registry.balena.${domain}/v2/": x509: certificate signed by unknown authority

From looking at the output of curl, things seem to be correct:

* Server certificate:
*  subject: CN=*.balena.${domain}
*  start date: Nov 13 02:15:59 2020 GMT
*  expire date: Nov 13 02:15:59 2022 GMT
*  issuer: CN=ca.balena.${domain}

All the *.balena.${domain} names a CNAMEs back to balena.${domain}

Is there something else required with certs?

edit:

$ balena --version
12.27.4

edit 2:
Looks like I had to copy across the ca.crt into /etc/pki/tls/certs/ and then restart docker via systemctl restart docker.

Now I get a different problem:

[Info]    Everything is up to date (use --build to force a rebuild)
[Info]    Creating release...
[Info]    Pushing images to registry...
Retrying "registry.balena.${domain}/v2/a0bd40c79a2756d16cd89b7b80959903:latest" after 2.00s (1 of 3) due to: Error: unauthorized: authentication required

I might open a new thread as to not pollute this one furter.

bversluijs · January 18, 2022, 7:48pm

Although this is an old topic, I still have some questions.
We’re about 2 years later since I’ve set up our openBalena instance. So the certificates are about to expire.

If I understand correctly, I can renew the certificates of the VPN and place them in the environment variables. Or to put it in code terms:

mv "${VPN_PKI}/index.txt" "${VPN_PKI}/index.txt.old"
mv $VPN_CRT "$VPN_CRT.old"
mv $VPN_KEY "$VPN_KEY.old"
mv $VPN_REQS "$VPN_REQS.old"

touch "${VPN_PKI}/index.txt"

"$easyrsa_bin" --pki-dir="${VPN_PKI}" --days="${CRT_EXPIRY_DAYS}" build-server-full "vpn.${CN}" nopass

Now, I have to update:

OPENBALENA_VPN_SERVER_CRT
OPENBALENA_VPN_SERVER_KEY

With the base64 values of the issued crt and private key.

So far so good! And if not, please tell me

Now, the root CA and CRT (OPENBALENA_ROOT_CA, OPENBALENA_ROOT_CRT, OPENBALENA_ROOT_KEY). Are these used for internal communication between all services, or am I missing something? And I’d probably need to renew these as well?

But now, the bigger question. How do I renew the CA of the VPN? It’s valid for 10 years at start, fast forward 2 years later and they’re valid for 8 years.

Just generating a new CA wouldn’t work, because all devices don’t know the new CA and fail to connect with VPN. As @richbayliss mentioned, the device uses the /os/v1/config endpoint to get to know the newest VPN CA. But does the supervisor regularly check this endpoint, or is this a one-time thing? And if so, what happens after those 10 years, are all devices bricked?

Anyhow, thanks in advance with getting to understand openBalena and their certificates!

ab77 · April 18, 2022, 4:30pm

But does the supervisor regularly check this endpoint, or is this a one-time thing? And if so, what happens after those 10 years, are all devices bricked?

Currently os- config service runs at boot and on a systemd 24h timer to check for updated CA certificate. There is a PR against meta-balena to make this check every time OpenVPN service is restarted.

But now, the bigger question. How do I renew the CA of the VPN? It’s valid for 10 years at start, fast forward 2 years later and they’re valid for 8 years.

The ideal process would be to create a root CA (~20yr) and a server CA (~10yr) singed by the root CA. These two form the CA bundle. The server CA is the one that issues the actual server certificate(s), which could have a ~5yr validity.

CA certificates should re renewed at around half of their validity period as a matter of good practice. So if and when that happens, and all the PKI assets are refreshed and VPN service restarted, the clients will see the connections drop and attempt to reconnect. If the client is running the Os where the VPN service checks for new PKI assets every-time it attempts to connect, the transition should be fairly seamless, but if not, it should recover within 24hrs.

drcnyc · April 9, 2023, 3:03am

I’d like to revisit this topic as we are nearing the expiry of our 2-year server certificate and would like to ensure we are clear on what is needed to renew it. If I’m understanding the chain above, we simply need to generate a new server certificate / key combination using the same ca (which now has about 8 years left), and once we have this we can simply update the following environment variables in the open-balena-vpn container:

VPN_OPENVPN_SERVER_CRT
VPN_OPENVPN_SERVER_KEY

with the newly generated certificate / key combination - which after updating (upon container restart) will generate new server.crt and server.key files via the confd templates.

I also want to confirm that nothing should need to be changed on each device - they will all now just authenticate using the new server certificates.

And lastly, some of the environment variable names mentioned above don’t seem to match what’s actually used in open-balena-vpn which is also causing some confusion.

Any guidance on all of this would be greatly appreciated!

ab77 · April 10, 2023, 3:11pm

Unfortunately it’s a bit clunky right now. There was an attempt to automate, but it has been blocked internally: openBalena on balenaOS by ab77 · Pull Request #141 · balena-io/open-balena · GitHub

In the meantime, take a look at upsert_ca_root, upsert_vpn_key, upsert_vpn_cert, upsert_vpn_dhparams and upsert_vpn_ca functions in open-balena-base/configure-balena.sh at master · balena-io-modules/open-balena-base · GitHub

These are the env vars required for VPN + API services in order for certificate to remain valid and for API to serve the correct updated content to devices.

Cheers!