Certificates error in device

openBalena
1

Hello,

I’m trying to get openBalena up on a Raspberry Pi 4 and an Azure VM. I am currently following the getting started guide.

I’m running into an issue with the certificate. I have gotten to the " Deploy our first application" section and have ran: ‘balena apps’ and ‘balena os confiugure --app myApp’ without problems. The device loads but gives the “check your dashboard” message with the four flashes from the led indicating a connection problem.
https://www.balena.io/docs/faq/troubleshooting/troubleshooting/

From my local terminal:
Running ‘curl <api.ADDRESS> -v’ returns: * SSL certificate verify ok.

From the device through the ‘balena ssh’ connection:
Running ‘curl <api.ADDRESS> -v’ returns:
Trying <IP_ADDRESS>…

  • TCP_NODELAY set
  • Connected to <api.ADDRESS> (<IP_ADDRESS>) port 443 (#0)
  • found 128 certificates in /etc/ssl/certs/ca-certificates.crt
  • ALPN, offering http/1.1
  • SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
  • server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
  • Closing connection 0
    curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
    More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

It seems that the certificate that I have copied from the server to my local system isn’t isn’t getting to the balena os image. Any help is appreciated, thank you for your time.

4

Hi, could you please check that the time and date on the device are configured appropriately? This can cause problems with certificate validation.

7

Hi Alex,

The date appears to be correct.
root@17ffd81:~# date
Mon Jul 27 17:40:05 UTC 2020

Here is a snip from a log as well.

root@17ffd81:~# journalctl -fn100
– Logs begin at Mon 2020-05-11 13:18:42 UTC. –
Jul 27 17:17:55 17ffd81 openvpn[4511]: Mon Jul 27 17:17:55 2020 [vpn.] Peer Connection Initiated with [AF_INET]:443
Jul 27 17:17:56 17ffd81 openvpn[4511]: Mon Jul 27 17:17:56 2020 SENT CONTROL [vpn.]: ‘PUSH_REQUEST’ (status=1)
Jul 27 17:17:56 17ffd81 openvpn[4511]: Mon Jul 27 17:17:56 2020 AUTH: Received control message: AUTH_FAILED
Jul 27 17:17:56 17ffd81 openvpn[4511]: Mon Jul 27 17:17:56 2020 SIGTERM[soft,auth-failure] received, process exiting
Jul 27 17:17:57 17ffd81 cbb8d40a9162[1457]: [event] Event: Device bootstrap {}
Jul 27 17:17:57 17ffd81 resin-supervisor[3721]: [event] Event: Device bootstrap {}
Jul 27 17:17:57 17ffd81 cbb8d40a9162[1457]: [info] New device detected. Provisioning…
Jul 27 17:17:57 17ffd81 resin-supervisor[3721]: [info] New device detected. Provisioning…
Jul 27 17:17:57 17ffd81 cbb8d40a9162[1457]: [event] Event: Device bootstrap failed, retrying {“delay”:30000,“error”:{“message”:""}}
Jul 27 17:17:57 17ffd81 resin-supervisor[3721]: [event] Event: Device bootstrap failed, retrying {“delay”:30000,“error”:{“message”:""}}
Jul 27 17:18:06 17ffd81 prepare-openvpn[4515]: prepare-openvpn: [INFO] Balena.io VPN authentication.
Jul 27 17:18:06 17ffd81 openvpn[4519]: Mon Jul 27 17:18:06 2020 WARNING: file ‘/var/volatile/vpn-auth’ is group or others accessible
Jul 27 17:18:06 17ffd81 openvpn[4519]: Mon Jul 27 17:18:06 2020 OpenVPN 2.4.7 aarch64-poky-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Jun 2 2020
Jul 27 17:18:06 17ffd81 openvpn[4519]: Mon Jul 27 17:18:06 2020 library versions: OpenSSL 1.1.1b 26 Feb 2019, LZO 2.10
Jul 27 17:18:06 17ffd81 openvpn[4519]: Mon Jul 27 17:18:06 2020 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Jul 27 17:18:06 17ffd81 openvpn[4519]: Mon Jul 27 17:18:06 2020 TCP/UDP: Preserving recently used remote address: [AF_INET]:443
Jul 27 17:18:06 17ffd81 openvpn[4519]: Mon Jul 27 17:18:06 2020 Socket Buffers: R=[87380->87380] S=[16384->16384]
Jul 27 17:18:06 17ffd81 openvpn[4519]: Mon Jul 27 17:18:06 2020 Attempting to establish TCP connection with [AF_INET]:443 [nonblock]
Jul 27 17:18:07 17ffd81 openvpn[4519]: Mon Jul 27 17:18:07 2020 TCP connection established with [AF_INET]:443
Jul 27 17:18:07 17ffd81 openvpn[4519]: Mon Jul 27 17:18:07 2020 TCP_CLIENT link local: (not bound)
Jul 27 17:18:07 17ffd81 openvpn[4519]: Mon Jul 27 17:18:07 2020 TCP_CLIENT link remote: [AF_INET]:443
Jul 27 17:18:07 17ffd81 openvpn[4519]: Mon Jul 27 17:18:07 2020 NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
Jul 27 17:18:07 17ffd81 openvpn[4519]: Mon Jul 27 17:18:07 2020 TLS: Initial packet from [AF_INET]:443, sid=3f51cbc5 3f40874f
Jul 27 17:18:07 17ffd81 openvpn[4519]: Mon Jul 27 17:18:07 2020 WARNING: this configuration may cache passwords in memory – use the auth-nocache option to prevent this
Jul 27 17:18:07 17ffd81 openvpn[4519]: Mon Jul 27 17:18:07 2020 VERIFY OK: depth=1, CN=vpn-ca.
Jul 27 17:18:07 17ffd81 openvpn[4519]: Mon Jul 27 17:18:07 2020 VERIFY KU OK
Jul 27 17:18:07 17ffd81 openvpn[4519]: Mon Jul 27 17:18:07 2020 Validating certificate extended key usage
Jul 27 17:18:07 17ffd81 openvpn[4519]: Mon Jul 27 17:18:07 2020 ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
Jul 27 17:18:07 17ffd81 openvpn[4519]: Mon Jul 27 17:18:07 2020 VERIFY EKU OK
Jul 27 17:18:07 17ffd81 openvpn[4519]: Mon Jul 27 17:18:07 2020 VERIFY OK: depth=0, CN=vpn.
Jul 27 17:18:07 17ffd81 openvpn[4519]: Mon Jul 27 17:18:07 2020 Control Channel: TLSv1.2, cipher TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 4096 bit RSA
Jul 27 17:18:07 17ffd81 openvpn[4519]: Mon Jul 27 17:18:07 2020 [vpn.] Peer Connection Initiated with [AF_INET]:443
Jul 27 17:18:09 17ffd81 openvpn[4519]: Mon Jul 27 17:18:09 2020 SENT CONTROL [vpn.]: ‘PUSH_REQUEST’ (status=1)
Jul 27 17:18:09 17ffd81 openvpn[4519]: Mon Jul 27 17:18:09 2020 AUTH: Received control message: AUTH_FAILED
Jul 27 17:18:09 17ffd81 openvpn[4519]: Mon Jul 27 17:18:09 2020 SIGTERM[soft,auth-failure] received, process exiting
Jul 27 17:18:19 17ffd81 prepare-openvpn[4548]: prepare-openvpn: [INFO] Balena.io VPN authentication.
Jul 27 17:18:19 17ffd81 openvpn[4561]: Mon Jul 27 17:18:19 2020 WARNING: file ‘/var/volatile/vpn-auth’ is group or others accessible
Jul 27 17:18:19 17ffd81 openvpn[4561]: Mon Jul 27 17:18:19 2020 OpenVPN 2.4.7 aarch64-poky-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Jun 2 2020
Jul 27 17:18:19 17ffd81 openvpn[4561]: Mon Jul 27 17:18:19 2020 library versions: OpenSSL 1.1.1b 26 Feb 2019, LZO 2.10
Jul 27 17:18:19 17ffd81 openvpn[4561]: Mon Jul 27 17:18:19 2020 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Jul 27 17:18:19 17ffd81 openvpn[4561]: Mon Jul 27 17:18:19 2020 TCP/UDP: Preserving recently used remote address: [AF_INET]:443
Jul 27 17:18:19 17ffd81 openvpn[4561]: Mon Jul 27 17:18:19 2020 Socket Buffers: R=[87380->87380] S=[16384->16384]
Jul 27 17:18:19 17ffd81 openvpn[4561]: Mon Jul 27 17:18:19 2020 Attempting to establish TCP connection with [AF_INET]:443 [nonblock]
Jul 27 17:18:19 17ffd81 balenad[1457]: time=“2020-07-27T17:18:19.558958788Z” level=info msg=“shim balena-engine-containerd-shim started” address=/containerd-shim/e75a809f03aa79c37e18cd58831c15b566a6ad4acd4717217e3db54913435708.sock debug=false pid=4564
Jul 27 17:18:20 17ffd81 openvpn[4561]: Mon Jul 27 17:18:20 2020 TCP connection established with [AF_INET]:443
Jul 27 17:18:20 17ffd81 openvpn[4561]: Mon Jul 27 17:18:20 2020 TCP_CLIENT link local: (not bound)
Jul 27 17:18:20 17ffd81 openvpn[4561]: Mon Jul 27 17:18:20 2020 TCP_CLIENT link remote: [AF_INET]:443
Jul 27 17:18:20 17ffd81 openvpn[4561]: Mon Jul 27 17:18:20 2020 NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
Jul 27 17:18:20 17ffd81 openvpn[4561]: Mon Jul 27 17:18:20 2020 TLS: Initial packet from [AF_INET]:443, sid=a118cdd8 6aa748b8
Jul 27 17:18:20 17ffd81 openvpn[4561]: Mon Jul 27 17:18:20 2020 WARNING: this configuration may cache passwords in memory – use the auth-nocache option to prevent this
Jul 27 17:18:20 17ffd81 openvpn[4561]: Mon Jul 27 17:18:20 2020 VERIFY OK: depth=1, CN=vpn-ca.
Jul 27 17:18:20 17ffd81 openvpn[4561]: Mon Jul 27 17:18:20 2020 VERIFY KU OK
Jul 27 17:18:20 17ffd81 openvpn[4561]: Mon Jul 27 17:18:20 2020 Validating certificate extended key usage
Jul 27 17:18:20 17ffd81 openvpn[4561]: Mon Jul 27 17:18:20 2020 ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
Jul 27 17:18:20 17ffd81 openvpn[4561]: Mon Jul 27 17:18:20 2020 VERIFY EKU OK
Jul 27 17:18:20 17ffd81 openvpn[4561]: Mon Jul 27 17:18:20 2020 VERIFY OK: depth=0, CN=vpn.
Jul 27 17:18:20 17ffd81 openvpn[4561]: Mon Jul 27 17:18:20 2020 Control Channel: TLSv1.2, cipher TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 4096 bit RSA
Jul 27 17:18:20 17ffd81 openvpn[4561]: Mon Jul 27 17:18:20 2020 [vpn.] Peer Connection Initiated with [AF_INET]:443
Jul 27 17:18:20 17ffd81 balenad[1457]: time=“2020-07-27T17:18:20.719848954Z” level=info msg=“shim reaped” id=56e9395b3cda030a466a161bc981a97cdb2114dc45b10792109f137fd1ee6511
Jul 27 17:18:20 17ffd81 balenad[1457]: time=“2020-07-27T17:18:20.722734880Z” level=info msg=“ignoring event” module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jul 27 17:18:21 17ffd81 openvpn[4561]: Mon Jul 27 17:18:21 2020 SENT CONTROL [vpn.]: ‘PUSH_REQUEST’ (status=1)
Jul 27 17:18:21 17ffd81 openvpn[4561]: Mon Jul 27 17:18:21 2020 AUTH: Received control message: AUTH_FAILED
Jul 27 17:18:21 17ffd81 openvpn[4561]: Mon Jul 27 17:18:21 2020 SIGTERM[soft,auth-failure] received, process exiting

10

Hi,
looking at openvpn logs it does not look like the CA certificate is the issue - it verifies it OK and then the AUTH_FAILED is quite unclean. Could you have a look at the openvpn server log which should be more verbose about that?
As for your initial curl test - this is the expected behavior. The self-signed certificate is not added to the system-wide bundle at this moment, it is only used by the particular services that need it for internal communication. Adding it to the system-wide CA bundle is something we are working on: https://github.com/balena-os/meta-balena/issues/1398

12

I’m going with getting it up with the let’s encrypt cert. If I circle back to this I will take a look at what you mentioned. Thank you for your response.