Certificates error in device

Hello,

I’m trying to get openBalena up on a Raspberry Pi 4 and an Azure VM. I am currently following the getting started guide.

I’m running into an issue with the certificate. I have gotten to the " Deploy our first application" section and have ran: ‘balena apps’ and ‘balena os confiugure --app myApp’ without problems. The device loads but gives the “check your dashboard” message with the four flashes from the led indicating a connection problem.
https://www.balena.io/docs/faq/troubleshooting/troubleshooting/

From my local terminal:
Running ‘curl <api.ADDRESS> -v’ returns: * SSL certificate verify ok.

From the device through the ‘balena ssh’ connection:
Running ‘curl <api.ADDRESS> -v’ returns:
Trying <IP_ADDRESS>…

  • TCP_NODELAY set
  • Connected to <api.ADDRESS> (<IP_ADDRESS>) port 443 (#0)
  • found 128 certificates in /etc/ssl/certs/ca-certificates.crt
  • ALPN, offering http/1.1
  • SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
  • server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
  • Closing connection 0
    curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
    More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

It seems that the certificate that I have copied from the server to my local system isn’t isn’t getting to the balena os image. Any help is appreciated, thank you for your time.

Hi, could you please check that the time and date on the device are configured appropriately? This can cause problems with certificate validation.

Hi Alex,

The date appears to be correct.
root@17ffd81:~# date
Mon Jul 27 17:40:05 UTC 2020

Here is a snip from a log as well.

root@17ffd81:~# journalctl -fn100
– Logs begin at Mon 2020-05-11 13:18:42 UTC. –
Jul 27 17:17:55 17ffd81 openvpn[4511]: Mon Jul 27 17:17:55 2020 [vpn.] Peer Connection Initiated with [AF_INET]:443
Jul 27 17:17:56 17ffd81 openvpn[4511]: Mon Jul 27 17:17:56 2020 SENT CONTROL [vpn.]: ‘PUSH_REQUEST’ (status=1)
Jul 27 17:17:56 17ffd81 openvpn[4511]: Mon Jul 27 17:17:56 2020 AUTH: Received control message: AUTH_FAILED
Jul 27 17:17:56 17ffd81 openvpn[4511]: Mon Jul 27 17:17:56 2020 SIGTERM[soft,auth-failure] received, process exiting
Jul 27 17:17:57 17ffd81 cbb8d40a9162[1457]: [event] Event: Device bootstrap {}
Jul 27 17:17:57 17ffd81 resin-supervisor[3721]: [event] Event: Device bootstrap {}
Jul 27 17:17:57 17ffd81 cbb8d40a9162[1457]: [info] New device detected. Provisioning…
Jul 27 17:17:57 17ffd81 resin-supervisor[3721]: [info] New device detected. Provisioning…
Jul 27 17:17:57 17ffd81 cbb8d40a9162[1457]: [event] Event: Device bootstrap failed, retrying {“delay”:30000,“error”:{“message”:""}}
Jul 27 17:17:57 17ffd81 resin-supervisor[3721]: [event] Event: Device bootstrap failed, retrying {“delay”:30000,“error”:{“message”:""}}
Jul 27 17:18:06 17ffd81 prepare-openvpn[4515]: prepare-openvpn: [INFO] Balena.io VPN authentication.
Jul 27 17:18:06 17ffd81 openvpn[4519]: Mon Jul 27 17:18:06 2020 WARNING: file ‘/var/volatile/vpn-auth’ is group or others accessible
Jul 27 17:18:06 17ffd81 openvpn[4519]: Mon Jul 27 17:18:06 2020 OpenVPN 2.4.7 aarch64-poky-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Jun 2 2020
Jul 27 17:18:06 17ffd81 openvpn[4519]: Mon Jul 27 17:18:06 2020 library versions: OpenSSL 1.1.1b 26 Feb 2019, LZO 2.10
Jul 27 17:18:06 17ffd81 openvpn[4519]: Mon Jul 27 17:18:06 2020 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Jul 27 17:18:06 17ffd81 openvpn[4519]: Mon Jul 27 17:18:06 2020 TCP/UDP: Preserving recently used remote address: [AF_INET]:443
Jul 27 17:18:06 17ffd81 openvpn[4519]: Mon Jul 27 17:18:06 2020 Socket Buffers: R=[87380->87380] S=[16384->16384]
Jul 27 17:18:06 17ffd81 openvpn[4519]: Mon Jul 27 17:18:06 2020 Attempting to establish TCP connection with [AF_INET]:443 [nonblock]
Jul 27 17:18:07 17ffd81 openvpn[4519]: Mon Jul 27 17:18:07 2020 TCP connection established with [AF_INET]:443
Jul 27 17:18:07 17ffd81 openvpn[4519]: Mon Jul 27 17:18:07 2020 TCP_CLIENT link local: (not bound)
Jul 27 17:18:07 17ffd81 openvpn[4519]: Mon Jul 27 17:18:07 2020 TCP_CLIENT link remote: [AF_INET]:443
Jul 27 17:18:07 17ffd81 openvpn[4519]: Mon Jul 27 17:18:07 2020 NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
Jul 27 17:18:07 17ffd81 openvpn[4519]: Mon Jul 27 17:18:07 2020 TLS: Initial packet from [AF_INET]:443, sid=3f51cbc5 3f40874f
Jul 27 17:18:07 17ffd81 openvpn[4519]: Mon Jul 27 17:18:07 2020 WARNING: this configuration may cache passwords in memory – use the auth-nocache option to prevent this
Jul 27 17:18:07 17ffd81 openvpn[4519]: Mon Jul 27 17:18:07 2020 VERIFY OK: depth=1, CN=vpn-ca.
Jul 27 17:18:07 17ffd81 openvpn[4519]: Mon Jul 27 17:18:07 2020 VERIFY KU OK
Jul 27 17:18:07 17ffd81 openvpn[4519]: Mon Jul 27 17:18:07 2020 Validating certificate extended key usage
Jul 27 17:18:07 17ffd81 openvpn[4519]: Mon Jul 27 17:18:07 2020 ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
Jul 27 17:18:07 17ffd81 openvpn[4519]: Mon Jul 27 17:18:07 2020 VERIFY EKU OK
Jul 27 17:18:07 17ffd81 openvpn[4519]: Mon Jul 27 17:18:07 2020 VERIFY OK: depth=0, CN=vpn.
Jul 27 17:18:07 17ffd81 openvpn[4519]: Mon Jul 27 17:18:07 2020 Control Channel: TLSv1.2, cipher TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 4096 bit RSA
Jul 27 17:18:07 17ffd81 openvpn[4519]: Mon Jul 27 17:18:07 2020 [vpn.] Peer Connection Initiated with [AF_INET]:443
Jul 27 17:18:09 17ffd81 openvpn[4519]: Mon Jul 27 17:18:09 2020 SENT CONTROL [vpn.]: ‘PUSH_REQUEST’ (status=1)
Jul 27 17:18:09 17ffd81 openvpn[4519]: Mon Jul 27 17:18:09 2020 AUTH: Received control message: AUTH_FAILED
Jul 27 17:18:09 17ffd81 openvpn[4519]: Mon Jul 27 17:18:09 2020 SIGTERM[soft,auth-failure] received, process exiting
Jul 27 17:18:19 17ffd81 prepare-openvpn[4548]: prepare-openvpn: [INFO] Balena.io VPN authentication.
Jul 27 17:18:19 17ffd81 openvpn[4561]: Mon Jul 27 17:18:19 2020 WARNING: file ‘/var/volatile/vpn-auth’ is group or others accessible
Jul 27 17:18:19 17ffd81 openvpn[4561]: Mon Jul 27 17:18:19 2020 OpenVPN 2.4.7 aarch64-poky-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Jun 2 2020
Jul 27 17:18:19 17ffd81 openvpn[4561]: Mon Jul 27 17:18:19 2020 library versions: OpenSSL 1.1.1b 26 Feb 2019, LZO 2.10
Jul 27 17:18:19 17ffd81 openvpn[4561]: Mon Jul 27 17:18:19 2020 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Jul 27 17:18:19 17ffd81 openvpn[4561]: Mon Jul 27 17:18:19 2020 TCP/UDP: Preserving recently used remote address: [AF_INET]:443
Jul 27 17:18:19 17ffd81 openvpn[4561]: Mon Jul 27 17:18:19 2020 Socket Buffers: R=[87380->87380] S=[16384->16384]
Jul 27 17:18:19 17ffd81 openvpn[4561]: Mon Jul 27 17:18:19 2020 Attempting to establish TCP connection with [AF_INET]:443 [nonblock]
Jul 27 17:18:19 17ffd81 balenad[1457]: time=“2020-07-27T17:18:19.558958788Z” level=info msg=“shim balena-engine-containerd-shim started” address=/containerd-shim/e75a809f03aa79c37e18cd58831c15b566a6ad4acd4717217e3db54913435708.sock debug=false pid=4564
Jul 27 17:18:20 17ffd81 openvpn[4561]: Mon Jul 27 17:18:20 2020 TCP connection established with [AF_INET]:443
Jul 27 17:18:20 17ffd81 openvpn[4561]: Mon Jul 27 17:18:20 2020 TCP_CLIENT link local: (not bound)
Jul 27 17:18:20 17ffd81 openvpn[4561]: Mon Jul 27 17:18:20 2020 TCP_CLIENT link remote: [AF_INET]:443
Jul 27 17:18:20 17ffd81 openvpn[4561]: Mon Jul 27 17:18:20 2020 NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
Jul 27 17:18:20 17ffd81 openvpn[4561]: Mon Jul 27 17:18:20 2020 TLS: Initial packet from [AF_INET]:443, sid=a118cdd8 6aa748b8
Jul 27 17:18:20 17ffd81 openvpn[4561]: Mon Jul 27 17:18:20 2020 WARNING: this configuration may cache passwords in memory – use the auth-nocache option to prevent this
Jul 27 17:18:20 17ffd81 openvpn[4561]: Mon Jul 27 17:18:20 2020 VERIFY OK: depth=1, CN=vpn-ca.
Jul 27 17:18:20 17ffd81 openvpn[4561]: Mon Jul 27 17:18:20 2020 VERIFY KU OK
Jul 27 17:18:20 17ffd81 openvpn[4561]: Mon Jul 27 17:18:20 2020 Validating certificate extended key usage
Jul 27 17:18:20 17ffd81 openvpn[4561]: Mon Jul 27 17:18:20 2020 ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
Jul 27 17:18:20 17ffd81 openvpn[4561]: Mon Jul 27 17:18:20 2020 VERIFY EKU OK
Jul 27 17:18:20 17ffd81 openvpn[4561]: Mon Jul 27 17:18:20 2020 VERIFY OK: depth=0, CN=vpn.
Jul 27 17:18:20 17ffd81 openvpn[4561]: Mon Jul 27 17:18:20 2020 Control Channel: TLSv1.2, cipher TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 4096 bit RSA
Jul 27 17:18:20 17ffd81 openvpn[4561]: Mon Jul 27 17:18:20 2020 [vpn.] Peer Connection Initiated with [AF_INET]:443
Jul 27 17:18:20 17ffd81 balenad[1457]: time=“2020-07-27T17:18:20.719848954Z” level=info msg=“shim reaped” id=56e9395b3cda030a466a161bc981a97cdb2114dc45b10792109f137fd1ee6511
Jul 27 17:18:20 17ffd81 balenad[1457]: time=“2020-07-27T17:18:20.722734880Z” level=info msg=“ignoring event” module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jul 27 17:18:21 17ffd81 openvpn[4561]: Mon Jul 27 17:18:21 2020 SENT CONTROL [vpn.]: ‘PUSH_REQUEST’ (status=1)
Jul 27 17:18:21 17ffd81 openvpn[4561]: Mon Jul 27 17:18:21 2020 AUTH: Received control message: AUTH_FAILED
Jul 27 17:18:21 17ffd81 openvpn[4561]: Mon Jul 27 17:18:21 2020 SIGTERM[soft,auth-failure] received, process exiting

Hi,
looking at openvpn logs it does not look like the CA certificate is the issue - it verifies it OK and then the AUTH_FAILED is quite unclean. Could you have a look at the openvpn server log which should be more verbose about that?
As for your initial curl test - this is the expected behavior. The self-signed certificate is not added to the system-wide bundle at this moment, it is only used by the particular services that need it for internal communication. Adding it to the system-wide CA bundle is something we are working on: https://github.com/balena-os/meta-balena/issues/1398

I’m going with getting it up with the let’s encrypt cert. If I circle back to this I will take a look at what you mentioned. Thank you for your response.