VPN Certs seems to be expired

Hello there,

we use open-balena now for some time. Thank you for that great product. It helps us a to manage our small fleet. But the last days I have the problem, that our some of our devices are marked as offline and are not reachable per ssh. I think, the problem is, that the devices can’t connect with the vpn server. I got this log from openvpn on a fresh device:

Feb 25 10:36:12 0142229 openvpn[1571]: Thu Feb 25 10:36:12 2021 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Feb 25 10:36:12 0142229 openvpn[1571]: Thu Feb 25 10:36:12 2021 TCP/UDP: Preserving recently used remote address: [AF_INET]*.*.*.*:443
Feb 25 10:36:12 0142229 openvpn[1571]: Thu Feb 25 10:36:12 2021 Socket Buffers: R=[131072->131072] S=[16384->16384]
Feb 25 10:36:12 0142229 openvpn[1571]: Thu Feb 25 10:36:12 2021 Attempting to establish TCP connection with [AF_INET]*.*.*.*:443 [nonblock]
Feb 25 10:36:13 0142229 openvpn[1571]: Thu Feb 25 10:36:13 2021 TCP connection established with [AF_INET]*.*.*.*:443
Feb 25 10:36:13 0142229 openvpn[1571]: Thu Feb 25 10:36:13 2021 TCP_CLIENT link local: (not bound)
Feb 25 10:36:13 0142229 openvpn[1571]: Thu Feb 25 10:36:13 2021 TCP_CLIENT link remote: [AF_INET]*.*.*.*:443
Feb 25 10:36:13 0142229 openvpn[1571]: Thu Feb 25 10:36:13 2021 TLS: Initial packet from [AF_INET]*.*.*.*:443, sid=1dae39af 877e29d6
Feb 25 10:36:13 0142229 openvpn[1571]: Thu Feb 25 10:36:13 2021 VERIFY OK: depth=2, CN=ca.balena.<server_domain>
Feb 25 10:36:13 0142229 openvpn[1571]: Thu Feb 25 10:36:13 2021 VERIFY OK: depth=1, CN=vpn-ca.balena.<server_domain>
Feb 25 10:36:13 0142229 openvpn[1571]: Thu Feb 25 10:36:13 2021 VERIFY ERROR: depth=0, error=certificate has expired: CN=vpn.balena.<server_domain>
Feb 25 10:36:13 0142229 openvpn[1571]: Thu Feb 25 10:36:13 2021 OpenSSL: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
Feb 25 10:36:13 0142229 openvpn[1571]: Thu Feb 25 10:36:13 2021 TLS_ERROR: BIO read tls_read_plaintext error
Feb 25 10:36:13 0142229 openvpn[1571]: Thu Feb 25 10:36:13 2021 TLS Error: TLS object -> incoming plaintext read error
Feb 25 10:36:13 0142229 openvpn[1571]: Thu Feb 25 10:36:13 2021 TLS Error: TLS handshake failed
Feb 25 10:36:13 0142229 openvpn[1571]: Thu Feb 25 10:36:13 2021 Fatal TLS error (check_tls_errors_co), restarting
Feb 25 10:36:13 0142229 openvpn[1571]: Thu Feb 25 10:36:13 2021 SIGUSR1[soft,tls-error] received, process restarting
Feb 25 10:36:13 0142229 openvpn[1571]: Thu Feb 25 10:36:13 2021 Restart pause, 120 second(s)

It seems, that the certificate of the vpn server is expired, what is possible. We use that server now for over two years. But if I check the cert with openssl with this command I get the answer, that the cert is still valid until April 2022.

echo | openssl s_client -showcerts -servername vpn.balena.<server_domain> -connect vpn.balena.<server_domain>:443 2>/dev/null | openssl x509 -inform pem -noout -text

But I have to admit, that I confused. I think that openssl delivered me the cert, that we have signed with the root ca of our company. But is that the same cert, that openvpn expect? If not, can I use that cert for both, api and vpn?

Hi there, it could be that the time difference between your VPN server and the device(s) has drifted by a large enough margin, for OpenVPN not to accept it. Can you please check/sync the time across all of your component and see if that resolves the issue. You should also check expiration on all of the certificates, not just the VPN server certificate, but also the corresponding CA cert.

Hallo @ab77 ,
thank you for the respond. I would like to try to follow your suggestions. I checked the time on the different devices and containers with the simple command date. This is the result.

balenaOS - Wed Mar 3 07:14:06 UTC 2021
supervisor - Wed Mar 3 07:14:48 UTC 2021

open-balena host os - Wed Mar 3 08:19:30 CET 2021
vpn container - Wed Mar 3 07:20:19 UTC 2021

So beside the timezone differenc everything seems fine.
But I digged deeper and found under config/certs/vpn/issued/vpn.<server_domain>.crt the vpn cert, that was created with the quickstart script. And it seems, that this certificate expired on the 6. february. So my question is can I renew that cert and if yes what changes I have to do on the clients as well?

Other maybe a little naive question. Is it possible to use the same CA and CERT for vpn that I use for the Root? So that value, that is used for OPENBALENA_ROOT_CA can I use that for OPENBALENA_VPN_CA and OPENBALENA_ROOT_CRT for OPENBALENA_VPN_CRT and so on?

Hi there Karl, thinking about this, perhaps the easiest method is to actually re-run the quickstart script somewhere else, and let it generate some new certs. Then, take their base64 encoded Env variables and replace your existing ones with the new. There is also some info in this thread: Certificates of openBalena

As for sharing the Certs, I am not 100% certain if that will cause any issues as I have not tried it myself. If you have a test environment setup you could certainly give it a try and see what happens!

Hi @dtischler,
thank you for the input. That put me on the right track. I think, to renew all certs with the quickstart script possible create a new root ca. But I only need a new server cert. But maybe your are right and this way works as well.

I choose a different way and manually create a new server certificat with following steps.

  1. Renamed the following old files in the config/certs/vpn folder.
  • issued/vpn.balena.iet.mw.tu-dresden.crt
  • private/vpn.balena.iet.mw.tu-dresden.key
  • reqs/vpn.balena.iet.mw.tu-dresden.req
  • index.txt
  1. Download easy-rsa to a temporary folder and extract it with following command:

curl -sL https://github.com/OpenVPN/easy-rsa/releases/download/v3.0.5/EasyRSA-nix-3.0.5.tgz | tar xz --strip-components=1

  1. Recreate the servcer certificat with key with this command:

./easyrsa/easyrsa --pki-dir="./vpn" --days=730 build-server-full "vpn.<server-domain>" nopass

  1. Convert with this command the cert- and the key-file to an base64 string and replace the values of the keys OPENBALENA_VPN_SERVER_CRT and OPENBALENA_VPN_SERVER_KEY.
  • echo "$(cat ./vpn/issued/vpn.<server-domain>.crt)" | base64 --wrap=0 2>/dev/null
  • echo "$(cat ./vpn/private/vpn.<server-domain>.key)" | base64 --wrap=0 2>/dev/null
  1. Recreate the VPN Container.

./scripts/compose up -d --force-recreate --no-deps

This steps worked for me and the client connect after a short time and marked as online in the device list.

5 Likes

@wolf_karl Thanks so much for sharing. I just hit this same issue and was just trying to work out how to renew safely myself. Saved me hours of work no doubt.

I can’t thanks you enougth.
I had my api. certificats that expired, and managed to have the server up again using your commands ( just not in the “vpn” directory, but in the “root” one)
Two things to add: easyrsa need the index.txt, so just add this command after renaming it:

touch index.txt

And the OPENBALENA_VPN_SERVER_CRT and OPENBALENA_VPN_SERVER_KEY are found in the config/activate file

Thanks again

1 Like