Openbalena K8S / Helm Chart - Certificate Error

openBalena
1

First off, major kudos to @bartversluijs for developing the helm chart for openbalena. We’ve been corresponding via email and have decided to take support matters to the forum to benefit everyone else who might be using it.

I am using cert-manager / letsencrypt, and I’m running into an issue where the VPN is not accepting connections due to self signed certs (see error from the device openvpn logs below). This keeps repeating and the device fails to connect / report online. I’ve followed the helm instructions to restart the haproxy ingress container but no luck - same error occurs. It seems it’s not taking the letsencrypt certs.

I know the rest of the app is taking them as I’m able to curl the https://api…/ping endpoint with no problem, certs are good there. So for some reason VPN is not picking them up.

I saw on another thread relating to this same issue, the advice was to delete the config folder with each build and run the openbalena quickstart script using -c, which I tried and it didn’t work for me, with or without the haproxy ingress restart.

Any ideas?

Mar 03 19:43:38 bbb15d7 openvpn[67074]: Thu Mar  3 19:43:38 2022 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Mar 03 19:43:38 bbb15d7 openvpn[67074]: Thu Mar  3 19:43:38 2022 TCP/UDP: Preserving recently used remote address: [AF_INET]xxx:443
Mar 03 19:43:38 bbb15d7 openvpn[67074]: Thu Mar  3 19:43:38 2022 Socket Buffers: R=[131072->131072] S=[16384->16384]
Mar 03 19:43:38 bbb15d7 openvpn[67074]: Thu Mar  3 19:43:38 2022 Attempting to establish TCP connection with [AF_INET]xxx:443 [nonblock]
Mar 03 19:43:39 bbb15d7 openvpn[67074]: Thu Mar  3 19:43:39 2022 TCP connection established with [AF_INET]xxx:443
Mar 03 19:43:39 bbb15d7 openvpn[67074]: Thu Mar  3 19:43:39 2022 TCP_CLIENT link local: (not bound)
Mar 03 19:43:39 bbb15d7 openvpn[67074]: Thu Mar  3 19:43:39 2022 TCP_CLIENT link remote: [AF_INET]xxx:443
Mar 03 19:43:39 bbb15d7 openvpn[67074]: Thu Mar  3 19:43:39 2022 TLS: Initial packet from [AF_INET]xxx:443, sid=a98d146f c43f4a89
Mar 03 19:43:39 bbb15d7 openvpn[67074]: Thu Mar  3 19:43:39 2022 VERIFY ERROR: depth=1, error=self signed certificate in certificate chain: CN=vpn-ca.xxx
Mar 03 19:43:39 bbb15d7 openvpn[67074]: Thu Mar  3 19:43:39 2022 OpenSSL: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
Mar 03 19:43:39 bbb15d7 openvpn[67074]: Thu Mar  3 19:43:39 2022 TLS_ERROR: BIO read tls_read_plaintext error
Mar 03 19:43:39 bbb15d7 openvpn[67074]: Thu Mar  3 19:43:39 2022 TLS Error: TLS object -> incoming plaintext read error
Mar 03 19:43:39 bbb15d7 openvpn[67074]: Thu Mar  3 19:43:39 2022 TLS Error: TLS handshake failed
Mar 03 19:43:39 bbb15d7 openvpn[67074]: Thu Mar  3 19:43:39 2022 Fatal TLS error (check_tls_errors_co), restarting
Mar 03 19:43:39 bbb15d7 openvpn[67074]: Thu Mar  3 19:43:39 2022 SIGUSR1[soft,tls-error] received, process restarting
Mar 03 19:43:39 bbb15d7 openvpn[67074]: Thu Mar  3 19:43:39 2022 Restart pause, 10 second(s)
2

Just want to post an answer to my own question, in case anyone else runs into this same issue. Looks like it had to do with migrating devices to the new K8S openbalena server. I had manually moved the first device over when I encountered this error. While I had updated all of the values in /mnt/boot/config.json to match the new server, I neglected to put the new server’s ca.crt in /etc/openvpn. Once I did that, everything worked just fine. For future devices I’m going to try the balena leave / join functionality of balena-cli, hopefully that goes smoother.

3

Hi David,

I see that you’ve figured it out with the certificates.
The VPN certificate is a self-signed certificate, the Kubernetes Helm chart is configurable to use a certificate manager for the endpoints (so api, tunnel, s3, registry etc). The VPN itself however is created by the quickstart command. This is the default way and isn’t any different than the openBalena repository.

However, it is still quite confusing how the certificates work in openBalena and what certificate is responsible for which service. And what is even more important, is how to renew those certificates in the future, before all VPN connections are lost. I hope there will be support for this in the near-future.

Hope this helps!

4

Hello,

just to let you know. I’ve created this issue on openBalena GH repo: Update openVPN certificates and deploy to devices · Issue #142 · balena-io/open-balena · GitHub
We are aware of the issue and am now tracking it.

Best Regards
Harald

1 Like