balenaOS self-signed certificates error

openBalena
, ,
1

Hi,
I setup yesterday an openBalena v3.1.1 instance with a self-signed certificate using the Getting started instructions.
One particularity is that there is no customizable DNS server available on the network, so I added an Unbound instance to the server running openBalena with the proper records. This means in order to resolve the *.openbalena.internal domain, each device has to use the IP address of the openBalena server in its DNS configuration.
I installed the self-signed certificate on my computer. So far so good, balena login and balena devices works without issue.
I generated the following config.json:

$ balena config generate --app myApp --version 2.65.0 --generate-device-api-key --deviceType raspberrypi4-64 --network wifi --wifiSsid [REDACTED] --wifiKey [REDACTED] --output config.json
? Check for updates every X minutes 10
applicationId:         1
deviceType:            raspberrypi4-64
userId:                2
appUpdatePollInterval: 600000
listenPort:            48484
vpnPort:               443
apiEndpoint:           https://api.openbalena.internal
vpnEndpoint:           vpn.openbalena.internal
registryEndpoint:      registry.openbalena.internal
deltaEndpoint:         https://delta.openbalena.internal
mixpanelToken:         __unused__
wifiSsid:              [REDACTED]
wifiKey:               [REDACTED]
balenaRootCA:          [REDACTED]
apiKey:                [REDACTED]
$ cat config.json
{
  "applicationId": 1,
  "deviceType": "raspberrypi4-64",
  "userId": 2,
  "appUpdatePollInterval": 600000,
  "listenPort": 48484,
  "vpnPort": 443,
  "apiEndpoint": "https://api.openbalena.internal",
  "vpnEndpoint": "vpn.openbalena.internal",
  "registryEndpoint": "registry.openbalena.internal",
  "deltaEndpoint": "https://delta.openbalena.internal",
  "mixpanelToken": "__unused__",
  "wifiSsid": "[REDACTED]",
  "wifiKey": "[REDACTED]",
  "balenaRootCA": "[REDACTED]",
  "apiKey": "[REDACTED]",
  "dnsServers": "192.168.11.178"
}

I then downloaded balena-cloud-raspberrypi4-64-2.65.0+rev1-dev-v12.2.11.img, flashed it on an SD card, then replaced the stock config.json with the one generated previously.
Unfortunately, the device never comes online and the status LED blinks 4 times, meaning no connectivity.
Using balena scan, I found the device IP address, then connected using ssh root@[REDACTED] -p 22222
Once logged in, I checked the logs:

Full logs 
$ ssh root@[REDACTED] -p 22222
root@35e2016:~# journalctl -u resin-supervisor
-- Logs begin at Thu 2021-01-21 14:12:15 UTC, end at Thu 2021-01-21 15:05:33 UTC. --
Jan 21 14:12:24 35e2016 resin-supervisor[1790]: resin_supervisor
Jan 21 14:12:24 35e2016 resin-supervisor[1822]: active
Jan 21 14:12:26 35e2016 resin-supervisor[1824]: Container config has not changed
Jan 21 14:12:29 35e2016 resin-supervisor[1824]: [info]    Supervisor v12.2.11 starting up...
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [info]    Setting host to discoverable
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [warn]    Invalid firewall mode: . Reverting to state: off
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [warn]    Invalid firewall mode: . Reverting to state: off
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [info]    Setting host to discoverable
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [warn]    Invalid firewall mode: . Reverting to state: off
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [info]    �🔥 Applying firewall mode: off
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [debug]   Starting logging infrastructure
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [debug]   Performing database cleanup for container log timestamps
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [info]    Starting firewall
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [info]    Previous engine snapshot was not stored. Skipping cleanup.
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [debug]   Handling of local mode switch is completed
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: (node:1) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buf>Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [info]    API Binder bound to: https://api.openbalena.internal/v6/
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [success] �🔥 Firewall mode applied
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [debug]   Starting api binder
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [event]   Event: Supervisor start {}
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [debug]   Connectivity check enabled: true
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [debug]   Starting periodic check for IP addresses
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [info]    Reporting initial state, supervisor version and API info
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [info]    Attempting to load any preloaded applications
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [error]   LogBackend: unexpected error: Error: self signed certificate in certificate chain
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [error]         at TLSSocket.onConnectSecure (_tls_wrap.js:1474:34)
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [error]       at TLSSocket.emit (events.js:310:20)
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [error]       at TLSSocket._finishInit (_tls_wrap.js:917:8)
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [error]       at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:687:12)
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [info]    VPN connection is not active.
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [info]    Waiting for connectivity...
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [info]    Starting API server
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [info]    Supervisor API successfully started on port 48484
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [info]    Applying target state
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [debug]   Ensuring device is provisioned
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [event]   Event: Device bootstrap {}
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [info]    New device detected. Provisioning...
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [debug]   Finished applying target state
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [success] Device state apply success
Jan 21 15:03:35 35e2016 resin-supervisor[1824]: [event]   Event: Device bootstrap {}
Jan 21 15:03:35 35e2016 resin-supervisor[1824]: [info]    New device detected. Provisioning...
Jan 21 15:03:35 35e2016 resin-supervisor[1824]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Jan 21 15:04:05 35e2016 resin-supervisor[1824]: [event]   Event: Device bootstrap {}
Jan 21 15:04:05 35e2016 resin-supervisor[1824]: [info]    New device detected. Provisioning...
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [warn]    Invalid firewall mode: . Reverting to state: off
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [info]    �🔥 Applying firewall mode: off
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [debug]   Starting logging infrastructure
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [debug]   Performing database cleanup for container log timestamps
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [info]    Starting firewall
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [info]    Previous engine snapshot was not stored. Skipping cleanup.
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [debug]   Handling of local mode switch is completed
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: (node:1) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buf>Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [info]    API Binder bound to: https://api.openbalena.internal/v6/
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [success] �🔥 Firewall mode applied
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [debug]   Starting api binder
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [event]   Event: Supervisor start {}
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [debug]   Connectivity check enabled: true
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [debug]   Starting periodic check for IP addresses
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [info]    Reporting initial state, supervisor version and API info
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [info]    Attempting to load any preloaded applications
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [error]   LogBackend: unexpected error: Error: self signed certificate in certificate chain
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [error]         at TLSSocket.onConnectSecure (_tls_wrap.js:1474:34)
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [error]       at TLSSocket.emit (events.js:310:20)
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [error]       at TLSSocket._finishInit (_tls_wrap.js:917:8)
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [error]       at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:687:12)
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [info]    VPN connection is not active.
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [info]    Waiting for connectivity...
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [info]    Starting API server
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [info]    Supervisor API successfully started on port 48484
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [info]    Applying target state
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [debug]   Ensuring device is provisioned
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [event]   Event: Device bootstrap {}
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [info]    New device detected. Provisioning...
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [debug]   Finished applying target state
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [success] Device state apply success
Jan 21 15:03:35 35e2016 resin-supervisor[1824]: [event]   Event: Device bootstrap {}
Jan 21 15:03:35 35e2016 resin-supervisor[1824]: [info]    New device detected. Provisioning...
Jan 21 15:03:35 35e2016 resin-supervisor[1824]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Jan 21 15:04:05 35e2016 resin-supervisor[1824]: [event]   Event: Device bootstrap {}
Jan 21 15:04:05 35e2016 resin-supervisor[1824]: [info]    New device detected. Provisioning...
Jan 21 15:04:35 35e2016 resin-supervisor[1824]: [event]   Event: Device bootstrap {}
Jan 21 15:04:35 35e2016 resin-supervisor[1824]: [info]    New device detected. Provisioning...
Jan 21 15:04:35 35e2016 resin-supervisor[1824]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Jan 21 15:05:05 35e2016 resin-supervisor[1824]: [event]   Event: Device bootstrap {}
Jan 21 15:05:05 35e2016 resin-supervisor[1824]: [info]    New device detected. Provisioning...

What stands out to me is the following:

Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [error]   LogBackend: unexpected error: Error: self signed certificate in certificate chain
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [error]         at TLSSocket.onConnectSecure (_tls_wrap.js:1474:34)
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [error]       at TLSSocket.emit (events.js:310:20)
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [error]       at TLSSocket._finishInit (_tls_wrap.js:917:8)
Jan 21 14:12:30 35e2016 resin-supervisor[1824]: [error]       at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:687:12)

According to this post, the cert should be populated in /etc/ssl/certs/balenaRootCA.pem. I opened the file and can confirmed the content are correct (identical to the ca.crt cert on openBalena).

I don’t know where else to look. It seems the certificated is correctly installed but isn’t picked up properly. I wonder if the custom DNS could be at fault, but the logs show the domains resolve.

Any help is appreciated.
Thanks in advance,
Erwan

4

Hey @edorgeville, indeed that error you highlighted seems like indicating the root issue here.

Let me echo some of the points made in that post you linked. You might have already tried these but it’d good to double check. Have you followed this section of the getting started guide?

The important bit is export NODE_EXTRA_CA_CERTS='/path/to/ca.crt'

Next question about this sentence:

then replaced the stock config.json with the one generated previously.

How did you do this step?
We advise the following balena CLI command to write config.json: balena CLI Documentation - Balena Documentation

Finally, what’s your balena CLI version?
We encountered incompatibility issues in the past.

6

Hi @gelbal,
Thanks for answering.

Have you followed this section of the getting started guide?

I did.

The important bit is export NODE_EXTRA_CA_CERTS='/path/to/ca.crt'

Yes, the balena CLI works on my computer.

We advise the following balena CLI command to write config.json : balena CLI Documentation - Balena Documentation

I created config.json using the balena config generate command, wrote balenaOS to the SD card using balenaEtcher, then copied config.json to the SD card manually using the file explorer. This is the same result as pre-configuring the img file.

Finally, what’s your balena CLI version?
We encountered incompatibility issues in the past.

The error happens on balenaOS on the Raspberry Pi 4, not on my computer. Still, here is the CLI version on my computer:

$ balena --version
12.38.3

Thanks again for your time,
Erwan

9

Hi there, I think a good starting point to troubleshoot this would be to verify client=>API communication off-device with the self-signed CA certificate. So uou should be able to test it from your local workstation as follows:

$ curl --cacert /path/to/ca.crt \
  https://api.openbalena.internal/config

This should return a JSON string and you should not need to specify --insecure flag to cURL.

If that works, we can try the same command on-device. The CA cert should be unpacked into /etc/ssl…

11

Hi @ab77, thanks for your answer.

Off-device:

~/certs/openbalena.internal $ curl --cacert ca.crt https://api.openbalena.internal/config
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>Cannot GET /config</pre>
</body>
</html>

~/certs/openbalena.internal $ curl --cacert ca.crt https://api.openbalena.internal/ping
OK

No HTTPS issue, only the /config enpoint doesn’t seem to answer to GET requests.

On-device:

root@35e2016:~# curl --cacert /etc/ssl/certs/balenaRootCA.pem https://api.openbalena.internal/config
curl: (6) Could not resolve host: api.openbalena.internal

It seems the domain won’t resolve. Running dig doesn’t use the proper DNS server:

root@35e2016:~# balena run --rm -it alpine sh
/ # apk add bind-tools
[...]
OK: 19 MiB in 31 packages
/ # dig api.openbalena.internal

; <<>> DiG 9.16.11 <<>> api.openbalena.internal
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 40531
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;api.openbalena.internal.       IN      A

;; Query time: 0 msec
;; SERVER: 10.114.102.1#53(10.114.102.1)
;; WHEN: Thu Jan 21 19:10:18 UTC 2021
;; MSG SIZE  rcvd: 52

Forcing a DNS server works:

/ # dig @192.168.11.178 api.openbalena.internal

; <<>> DiG 9.16.11 <<>> @192.168.11.178 api.openbalena.internal
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58922
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;api.openbalena.internal.       IN      A

;; ANSWER SECTION:
api.openbalena.internal. 3600   IN      A       192.168.11.178

;; Query time: 12 msec
;; SERVER: 192.168.11.178#53(192.168.11.178)
;; WHEN: Thu Jan 21 19:10:58 UTC 2021
;; MSG SIZE  rcvd: 68

Is it possible the DNS server from config.json isn’t used?

PS: I edited my original post to include the IP address of the internal DNS server in the config.json part.

12

OK, I am not sure about the /config endpoint, but that test passed anyway, no cert errors as expected.

Can you see what’s in /etc/resolv.dnsmasq? This is the resolver config for dnsmasq running on host @ nameserver 127.0.0.2. So next test would be to dig @127.0.0.2 api.openbalena.internal.

I have a feeling, spinning up a container manually on the hostOS will probably place it on the network bridge not configured to use the correct resolver…

13

alternatively, just on the host, all of the following should work nslookup api.openbalena.internal or nslookup api.openbalena.internal 127.0.0.2 or nslookup api.openbalena.internal 192.168.11.178.

15

All the following were ran directly on host:

Can you see what’s in /etc/resolv.dnsmasq ?

Seems like dnsmasq uses the DNS server provided by DHCP:

root@35e2016:~# cat /etc/resolv.dnsmasq
nameserver 192.168.10.1

alternatively, just on the host, all of the following should work nslookup api.openbalena.internal or nslookup api.openbalena.internal 127.0.0.2 or nslookup api.openbalena.internal 192.168.11.178 .

root@35e2016:~# nslookup api.openbalena.internal
Server:    127.0.0.2
Address 1: 127.0.0.2 35e2016

nslookup: can't resolve 'api.openbalena.internal'
root@35e2016:~# nslookup api.openbalena.internal 127.0.0.2
Server:    127.0.0.2
Address 1: 127.0.0.2 35e2016

nslookup: can't resolve 'api.openbalena.internal'
root@35e2016:~# nslookup api.openbalena.internal 192.168.11.178
Server:    192.168.11.178
Address 1: 192.168.11.178

Name:      api.openbalena.internal
Address 1: 192.168.11.178
16

Hello! I think I’m a little confused about what your network setup looks like, so perhaps we can walk through what IPs you’re expecting to see in various places - feel free to use placeholders if you prefer to keep this information private.

You have a device running both openBalena and Unbound. The Unbound instance is acting as a DNS server for the openbalena.internal domain, and configured with A records for api among other records. Your workstation is able to resolve these DNS records. Is the Unbound instance acting as both a DNS server and a DNS resolver for all the involved devices (workstation, RPi, OB server)? Does this have the IP of 192.168.10.1, the IP address reported on the RPi in the dnsmasq resolver settings?

Why I’m stuck on troubleshooting at this point is because clearly at some point, the RPi did resolve a hostname for OB, as it connected to something to get the TLS error. But from your nslookup, it looks like it’s no longer able to resolve. And the 192.168.11.178, is that IP related to Unbound or OpenBalena?

18

Hi @notnamed,
You are right that my network setup isn’t obvious from my posts :^)

Devices on the network such as my workstation and Raspberry Pi get their network configuration from the default DHCP server at 192.168.10.1. The DHCP provided IP address for DNS is 192.168.10.1 as well.
In order to use custom domains on this network, I needed a DNS server that can serve custom DNS records. Unfortunately the DNS server that is on this network doesn’t have such capabilities.
So I installed Unbound on the openBalena server, which has a fixed IP address 192.168.11.178.
On my workstation, I set my DNS configuration manually to 192.168.11.178, which allows me to resolve openbalena.internal to the same IP address, 192.168.11.178.

Correct, as long as all those devices are manually configurated to use the 192.168.1.178 as their DNS server.

The IP 192.168.10.1 is provided by DHCP should be replaced by the IP 192.168.11.178 provided in config.json

Both, they’re the same machine.

I am confused as well, I did a fresh balenaOS install and got different results this time:

root@15d918a:~# journalctl -u resin-supervisor
-- Logs begin at Fri 2020-12-18 10:11:17 UTC, end at Fri 2021-01-22 14:02:21 UTC. --
Dec 18 10:11:25 15d918a resin-supervisor[2635]: Error response from daemon: No such container: resin_supervisor
Dec 18 10:11:25 15d918a resin-supervisor[2749]: active
Dec 18 10:11:26 15d918a resin-supervisor[2755]: Error: No such object: resin_supervisor
Dec 18 10:11:27 15d918a resin-supervisor[2755]: Error: No such container: resin_supervisor
Dec 18 10:11:31 15d918a resin-supervisor[2755]: [info]    Supervisor v12.2.11 starting up...
Dec 18 10:11:32 15d918a resin-supervisor[2755]: [info]    Setting host to discoverable
Dec 18 10:11:32 15d918a resin-supervisor[2755]: [warn]    Invalid firewall mode: . Reverting to state: off
Dec 18 10:11:32 15d918a resin-supervisor[2755]: [info]    🔥 Applying firewall mode: off
Dec 18 10:11:32 15d918a resin-supervisor[2755]: [debug]   Starting logging infrastructure
Dec 18 10:11:32 15d918a resin-supervisor[2755]: [debug]   Performing database cleanup for container log timestamps
Dec 18 10:11:32 15d918a resin-supervisor[2755]: [info]    Starting firewall
Dec 18 10:11:32 15d918a resin-supervisor[2755]: [info]    Previous engine snapshot was not stored. Skipping cleanup.
Dec 18 10:11:32 15d918a resin-supervisor[2755]: [debug]   Handling of local mode switch is completed
Dec 18 10:11:32 15d918a resin-supervisor[2755]: (node:1) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
Dec 18 10:11:32 15d918a resin-supervisor[2755]: [info]    API Binder bound to: https://api.openbalena.internal/v6/
Dec 18 10:11:32 15d918a resin-supervisor[2755]: [success] 🔥 Firewall mode applied
Dec 18 10:11:32 15d918a resin-supervisor[2755]: [debug]   Starting api binder
Dec 18 10:11:32 15d918a resin-supervisor[2755]: [event]   Event: Supervisor start {}
Dec 18 10:11:32 15d918a resin-supervisor[2755]: [debug]   Connectivity check enabled: true
Dec 18 10:11:32 15d918a resin-supervisor[2755]: [debug]   Starting periodic check for IP addresses
Dec 18 10:11:32 15d918a resin-supervisor[2755]: [info]    VPN connection is not active.
Dec 18 10:11:32 15d918a resin-supervisor[2755]: [info]    Waiting for connectivity...
Dec 18 10:11:32 15d918a resin-supervisor[2755]: [error]   LogBackend: unexpected error: Error: certificate is not yet valid
Dec 18 10:11:32 15d918a resin-supervisor[2755]: [error]         at TLSSocket.onConnectSecure (_tls_wrap.js:1474:34)
Dec 18 10:11:32 15d918a resin-supervisor[2755]: [error]       at TLSSocket.emit (events.js:310:20)
Dec 18 10:11:32 15d918a resin-supervisor[2755]: [error]       at TLSSocket._finishInit (_tls_wrap.js:917:8)
Dec 18 10:11:32 15d918a resin-supervisor[2755]: [error]       at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:687:12)
Dec 18 10:11:32 15d918a resin-supervisor[2755]: [info]    Reporting initial state, supervisor version and API info
Dec 18 10:11:32 15d918a resin-supervisor[2755]: [info]    Attempting to load any preloaded applications
Dec 18 10:11:32 15d918a resin-supervisor[2755]: [info]    Starting API server
Dec 18 10:11:32 15d918a resin-supervisor[2755]: [info]    Supervisor API successfully started on port 48484
Dec 18 10:11:32 15d918a resin-supervisor[2755]: [info]    Applying target state
Dec 18 10:11:32 15d918a resin-supervisor[2755]: [debug]   Ensuring device is provisioned
Dec 18 10:11:32 15d918a resin-supervisor[2755]: [event]   Event: Device bootstrap {}
Dec 18 10:11:32 15d918a resin-supervisor[2755]: [info]    New device detected. Provisioning...
Dec 18 10:11:32 15d918a resin-supervisor[2755]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Dec 18 10:11:33 15d918a resin-supervisor[2755]: [debug]   Creating supervisor0 network
Jan 22 14:02:01 15d918a resin-supervisor[2755]: [event]   Event: Device bootstrap {}
Jan 22 14:02:01 15d918a resin-supervisor[2755]: [info]    New device detected. Provisioning...
Jan 22 14:02:01 15d918a resin-supervisor[2755]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
root@15d918a:~# cat /etc/resolv.dnsmasq
nameserver 192.168.10.1

Notice how the date jumps from Dec 18 to Jan 22, after getting a certificate is not yet valid error. I suppose this is NTP kicking in a bit late.
Also, /etc/resolv.dnsmasq contains the wrong IP address (192.168.10.1). Is it possible that it did contain 192.168.11.178 at some point but was overwritten?

EDIT: Here is a longer log with a fresh install running for longer:

Console log 
$ ssh root@192.168.13.149 -p 22222 "journalctl -u resin-supervisor"
-- Logs begin at Fri 2020-12-18 10:11:17 UTC, end at Fri 2021-01-22 14:28:07 UTC. --
Dec 18 10:11:25 772d428 resin-supervisor[2703]: Error response from daemon: No such container: resin_supervisor
Dec 18 10:11:25 772d428 resin-supervisor[2802]: active
Dec 18 10:11:26 772d428 resin-supervisor[2810]: Error: No such object: resin_supervisor
Dec 18 10:11:26 772d428 resin-supervisor[2810]: Error: No such container: resin_supervisor
Dec 18 10:11:31 772d428 resin-supervisor[2810]: [info]    Supervisor v12.2.11 starting up...
Dec 18 10:11:32 772d428 resin-supervisor[2810]: [info]    Setting host to discoverable
Dec 18 10:11:32 772d428 resin-supervisor[2810]: [warn]    Invalid firewall mode: . Reverting to state: off
Dec 18 10:11:32 772d428 resin-supervisor[2810]: [info]    🔥 Applying firewall mode: off
Dec 18 10:11:32 772d428 resin-supervisor[2810]: [debug]   Starting logging infrastructure
Dec 18 10:11:32 772d428 resin-supervisor[2810]: [debug]   Performing database cleanup for container log timestamps
Dec 18 10:11:32 772d428 resin-supervisor[2810]: [info]    Starting firewall
Dec 18 10:11:32 772d428 resin-supervisor[2810]: [info]    Previous engine snapshot was not stored. Skipping cleanup.
Dec 18 10:11:32 772d428 resin-supervisor[2810]: [debug]   Handling of local mode switch is completed
Dec 18 10:11:32 772d428 resin-supervisor[2810]: (node:1) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
Dec 18 10:11:32 772d428 resin-supervisor[2810]: [info]    API Binder bound to: https://api.openbalena.internal/v6/
Dec 18 10:11:32 772d428 resin-supervisor[2810]: [success] 🔥 Firewall mode applied
Dec 18 10:11:32 772d428 resin-supervisor[2810]: [debug]   Starting api binder
Dec 18 10:11:32 772d428 resin-supervisor[2810]: [event]   Event: Supervisor start {}
Dec 18 10:11:32 772d428 resin-supervisor[2810]: [debug]   Connectivity check enabled: true
Dec 18 10:11:32 772d428 resin-supervisor[2810]: [debug]   Starting periodic check for IP addresses
Dec 18 10:11:32 772d428 resin-supervisor[2810]: [error]   LogBackend: unexpected error: Error: certificate is not yet valid
Dec 18 10:11:32 772d428 resin-supervisor[2810]: [error]         at TLSSocket.onConnectSecure (_tls_wrap.js:1474:34)
Dec 18 10:11:32 772d428 resin-supervisor[2810]: [error]       at TLSSocket.emit (events.js:310:20)
Dec 18 10:11:32 772d428 resin-supervisor[2810]: [error]       at TLSSocket._finishInit (_tls_wrap.js:917:8)
Dec 18 10:11:32 772d428 resin-supervisor[2810]: [error]       at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:687:12)
Dec 18 10:11:32 772d428 resin-supervisor[2810]: [info]    VPN connection is not active.
Dec 18 10:11:32 772d428 resin-supervisor[2810]: [info]    Waiting for connectivity...
Dec 18 10:11:32 772d428 resin-supervisor[2810]: [info]    Reporting initial state, supervisor version and API info
Dec 18 10:11:32 772d428 resin-supervisor[2810]: [info]    Attempting to load any preloaded applications
Dec 18 10:11:32 772d428 resin-supervisor[2810]: [info]    Starting API server
Dec 18 10:11:32 772d428 resin-supervisor[2810]: [info]    Supervisor API successfully started on port 48484
Dec 18 10:11:32 772d428 resin-supervisor[2810]: [info]    Applying target state
Dec 18 10:11:32 772d428 resin-supervisor[2810]: [debug]   Ensuring device is provisioned
Dec 18 10:11:32 772d428 resin-supervisor[2810]: [event]   Event: Device bootstrap {}
Dec 18 10:11:32 772d428 resin-supervisor[2810]: [info]    New device detected. Provisioning...
Dec 18 10:11:32 772d428 resin-supervisor[2810]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Dec 18 10:11:32 772d428 resin-supervisor[2810]: [debug]   Creating supervisor0 network
Jan 22 14:15:07 772d428 resin-supervisor[3355]: resin_supervisor
Jan 22 14:15:07 772d428 systemd[1]: resin-supervisor.service: Main process exited, code=killed, status=15/TERM
Jan 22 14:15:08 772d428 resin-supervisor[3450]: resin_supervisor
Jan 22 14:15:08 772d428 resin-supervisor[3459]: active
Jan 22 14:15:09 772d428 resin-supervisor[3460]: Container config has not changed
Jan 22 14:15:11 772d428 resin-supervisor[3460]: [info]    Supervisor v12.2.11 starting up...
Jan 22 14:15:11 772d428 resin-supervisor[3460]: [info]    Setting host to discoverable
Jan 22 14:15:12 772d428 resin-supervisor[3460]: [warn]    Invalid firewall mode: . Reverting to state: off
Jan 22 14:15:12 772d428 resin-supervisor[3460]: [info]    🔥 Applying firewall mode: off
Jan 22 14:15:12 772d428 resin-supervisor[3460]: [debug]   Starting logging infrastructure
Jan 22 14:15:12 772d428 resin-supervisor[3460]: [debug]   Performing database cleanup for container log timestamps
Jan 22 14:15:12 772d428 resin-supervisor[3460]: [info]    Starting firewall
Jan 22 14:15:12 772d428 resin-supervisor[3460]: [success] 🔥 Firewall mode applied
Jan 22 14:15:12 772d428 resin-supervisor[3460]: [debug]   Starting api binder
Jan 22 14:15:12 772d428 resin-supervisor[3460]: [info]    Previous engine snapshot was not stored. Skipping cleanup.
Jan 22 14:15:12 772d428 resin-supervisor[3460]: [debug]   Handling of local mode switch is completed
Jan 22 14:15:12 772d428 resin-supervisor[3460]: [info]    API Binder bound to: https://api.openbalena.internal/v6/
Jan 22 14:15:12 772d428 resin-supervisor[3460]: [event]   Event: Supervisor start {}
Jan 22 14:15:12 772d428 resin-supervisor[3460]: (node:1) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
Jan 22 14:15:12 772d428 resin-supervisor[3460]: [debug]   Connectivity check enabled: true
Jan 22 14:15:12 772d428 resin-supervisor[3460]: [debug]   Starting periodic check for IP addresses
Jan 22 14:15:12 772d428 resin-supervisor[3460]: [info]    Reporting initial state, supervisor version and API info
Jan 22 14:15:12 772d428 resin-supervisor[3460]: [info]    Attempting to load any preloaded applications
Jan 22 14:15:12 772d428 resin-supervisor[3460]: [debug]   VPN status path exists.
Jan 22 14:15:12 772d428 resin-supervisor[3460]: [info]    VPN connection is not active.
Jan 22 14:15:12 772d428 resin-supervisor[3460]: [error]   LogBackend: unexpected error: Error: self signed certificate in certificate chain
Jan 22 14:15:12 772d428 resin-supervisor[3460]: [error]         at TLSSocket.onConnectSecure (_tls_wrap.js:1474:34)
Jan 22 14:15:12 772d428 resin-supervisor[3460]: [error]       at TLSSocket.emit (events.js:310:20)
Jan 22 14:15:12 772d428 resin-supervisor[3460]: [error]       at TLSSocket._finishInit (_tls_wrap.js:917:8)
Jan 22 14:15:12 772d428 resin-supervisor[3460]: [error]       at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:687:12)
Jan 22 14:15:12 772d428 resin-supervisor[3460]: [info]    Waiting for connectivity...
Jan 22 14:15:12 772d428 resin-supervisor[3460]: [info]    Starting API server
Jan 22 14:15:12 772d428 resin-supervisor[3460]: [info]    Supervisor API successfully started on port 48484
Jan 22 14:15:12 772d428 resin-supervisor[3460]: [info]    Applying target state
Jan 22 14:15:12 772d428 resin-supervisor[3460]: [debug]   Ensuring device is provisioned
Jan 22 14:15:12 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap {}
Jan 22 14:15:12 772d428 resin-supervisor[3460]: [info]    New device detected. Provisioning...
Jan 22 14:15:12 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Jan 22 14:15:12 772d428 resin-supervisor[3460]: [debug]   Finished applying target state
Jan 22 14:15:12 772d428 resin-supervisor[3460]: [success] Device state apply success
Jan 22 14:15:42 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap {}
Jan 22 14:15:42 772d428 resin-supervisor[3460]: [info]    New device detected. Provisioning...
Jan 22 14:15:42 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Jan 22 14:16:12 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap {}
Jan 22 14:16:12 772d428 resin-supervisor[3460]: [info]    New device detected. Provisioning...
Jan 22 14:16:12 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Jan 22 14:16:42 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap {}
Jan 22 14:16:42 772d428 resin-supervisor[3460]: [info]    New device detected. Provisioning...
Jan 22 14:16:42 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Jan 22 14:17:12 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap {}
Jan 22 14:17:12 772d428 resin-supervisor[3460]: [info]    New device detected. Provisioning...
Jan 22 14:17:12 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Jan 22 14:17:42 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap {}
Jan 22 14:17:42 772d428 resin-supervisor[3460]: [info]    New device detected. Provisioning...
Jan 22 14:17:42 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Jan 22 14:18:12 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap {}
Jan 22 14:18:12 772d428 resin-supervisor[3460]: [info]    New device detected. Provisioning...
Jan 22 14:18:12 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Jan 22 14:18:42 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap {}
Jan 22 14:18:42 772d428 resin-supervisor[3460]: [info]    New device detected. Provisioning...
Jan 22 14:18:42 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Jan 22 14:19:12 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap {}
Jan 22 14:19:12 772d428 resin-supervisor[3460]: [info]    New device detected. Provisioning...
Jan 22 14:19:12 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Jan 22 14:19:42 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap {}
Jan 22 14:19:42 772d428 resin-supervisor[3460]: [info]    New device detected. Provisioning...
Jan 22 14:19:42 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Jan 22 14:20:10 772d428 resin-supervisor[3460]: [api]     GET /v1/healthy 200 - 18.167 ms
Jan 22 14:20:12 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap {}
Jan 22 14:20:12 772d428 resin-supervisor[3460]: [info]    New device detected. Provisioning...
Jan 22 14:20:12 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Jan 22 14:20:42 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap {}
Jan 22 14:20:42 772d428 resin-supervisor[3460]: [info]    New device detected. Provisioning...
Jan 22 14:20:42 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Jan 22 14:21:12 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap {}
Jan 22 14:21:13 772d428 resin-supervisor[3460]: [info]    New device detected. Provisioning...
Jan 22 14:21:13 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Jan 22 14:21:43 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap {}
Jan 22 14:21:43 772d428 resin-supervisor[3460]: [info]    New device detected. Provisioning...
Jan 22 14:21:43 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Jan 22 14:22:13 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap {}
Jan 22 14:22:13 772d428 resin-supervisor[3460]: [info]    New device detected. Provisioning...
Jan 22 14:22:13 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Jan 22 14:22:43 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap {}
Jan 22 14:22:43 772d428 resin-supervisor[3460]: [info]    New device detected. Provisioning...
Jan 22 14:22:43 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Jan 22 14:23:13 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap {}
Jan 22 14:23:13 772d428 resin-supervisor[3460]: [info]    New device detected. Provisioning...
Jan 22 14:23:13 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Jan 22 14:23:43 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap {}
Jan 22 14:23:43 772d428 resin-supervisor[3460]: [info]    New device detected. Provisioning...
Jan 22 14:23:43 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Jan 22 14:24:13 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap {}
Jan 22 14:24:13 772d428 resin-supervisor[3460]: [info]    New device detected. Provisioning...
Jan 22 14:24:13 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Jan 22 14:24:43 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap {}
Jan 22 14:24:43 772d428 resin-supervisor[3460]: [info]    New device detected. Provisioning...
Jan 22 14:24:43 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Jan 22 14:25:11 772d428 resin-supervisor[3460]: [api]     GET /v1/healthy 200 - 3.468 ms
Jan 22 14:25:11 772d428 resin-supervisor[3460]: [debug]   Attempting container log timestamp flush...
Jan 22 14:25:11 772d428 resin-supervisor[3460]: [debug]   Container log timestamp flush complete
Jan 22 14:25:13 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap {}
Jan 22 14:25:13 772d428 resin-supervisor[3460]: [info]    New device detected. Provisioning...
Jan 22 14:25:13 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Jan 22 14:25:43 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap {}
Jan 22 14:25:43 772d428 resin-supervisor[3460]: [info]    New device detected. Provisioning...
Jan 22 14:25:43 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Jan 22 14:26:13 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap {}
Jan 22 14:26:13 772d428 resin-supervisor[3460]: [info]    New device detected. Provisioning...
Jan 22 14:26:13 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Jan 22 14:26:43 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap {}
Jan 22 14:26:43 772d428 resin-supervisor[3460]: [info]    New device detected. Provisioning...
Jan 22 14:26:43 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Jan 22 14:27:13 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap {}
Jan 22 14:27:13 772d428 resin-supervisor[3460]: [info]    New device detected. Provisioning...
Jan 22 14:27:14 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Jan 22 14:27:44 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap {}
Jan 22 14:27:44 772d428 resin-supervisor[3460]: [info]    New device detected. Provisioning...
Jan 22 14:27:44 772d428 resin-supervisor[3460]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}

$ ssh root@192.168.13.149 -p 22222 "nslookup api.openbalena.internal"
Server:    127.0.0.2
Address 1: 127.0.0.2 772d428

Name:      api.openbalena.internal
Address 1: 192.168.11.178

$

The supervisor says certificate is not yet valid, crashes, then self signed certificate in certificate chain

23

@edorgeville, thanks for sharing the logs and pointing out all the suspicious bits. :slight_smile:

The IP 192.168.10.1 is provided by DHCP should be replaced by the IP 192.168.11.178 provided in config.json

There may be a twist here, because a DNS server specified in config.json is not necessarily the DNS server that will be used by the device. The documentation of the dnsServers field of config.json reads:

Configuration - Balena Documentation
When dnsServers is defined and not “null”, the listed servers will be added to the list of servers obtained via DHCP or statically configured via a NetworkManager connection profile.

Therefore, it cannot be assumed that the device will use the DNS server specified in config.json in preference over the DNS server provided by DHCP - it’s just one more server among others. To prevent usage of the DNS server(s) provided by the DHCP server, you would have to either reconfigure the DHCP server (for example, configuration conditional on MAC addresses), or setup a separate subnet with your own DHCP and DNS servers, for example using the popular dnsmasq server.

However, I am still not clear why this DNS server issue would result in Error: self signed certificate in certificate chain. It may be a case of multiple issues happening at the same time, and the DNS issue may be separate to the self-signed certificate issue (I am not sure).

the date jumps from Dec 18 to Jan 22, after getting a certificate is not yet valid error. I suppose this is NTP kicking in a bit late.

Yes, this could be a (3rd) separate issue as well. The fact that the date jumps, however, indicates that the time is eventually sorted, so again I don’t think it is an explanation for the later Error: self signed certificate in certificate chain.

I don’t know where else to look. It seems the certificated is correctly installed but isn’t picked up properly. I wonder if the custom DNS could be at fault, […]

Indeed, it would seem a case of the root certificate (ca.crt) not correctly installed, but you have already checked it. I would suggest fixing the DNS issue first (as suggested above), even if it is not clear that it would also fix the self-signed certificate error. At least we would be able to definitely rule it out, and then go from there.

25

I was using the same host image file “balena-cloud-raspberrypi4-64-2.65.0+rev1-dev-v12.2.11.img”, and seeing the same error “self signed certificate in certificate chain” from Balena supervisor log after following all the steps in https://www.balena.io/open/docs/getting-started/.

I guess it is due to the CA certificate not properly propagated to the supervisor container, because no environment variable set by BALENA_ROOT_CA or NODE_EXTRA_CA_CERTS inside the container.

After I manually copy the CA certificate to the supervisor container file “/usr/local/share/ca-certificates/balenaRootCA.crt”, and run update-ca-certificates in container. Also set NODE_EXTRA_CA_CERTS=/usr/local/share/ca-certificates/balenaRootCA.crt in container file /usr/src/app/entry.sh, then the device can be seen by cli with “balena devices”.

I think the supervisor container was not create properly in “balena-cloud-raspberrypi4-64-2.65.0+rev1-dev-v12.2.11.img”.

27

Hi, just to make sure the guide was followed correctly, did you export the NODE_EXTRA_CA_CERTS on your development machine as described in the guide and been able to login with balena login? A freshly downloaded image will not contain the self-generated certificate of the open server you configured.
Thanks,
Zahari

29

Yes, all the steps were followed, otherwise I guess I would not be able to login or create app. I also checked that the only file modified by command “balena os configure” is config.json in the first partition, with content like below:

{
“applicationId”: 1,
“deviceType”: “raspberrypi4-64”,
“userId”: 2,
“appUpdatePollInterval”: 600000,
“listenPort”: 48484,
“vpnPort”: 443,
“apiEndpoint”: “https://api.{{domain}}”,
“vpnEndpoint”: “vpn.{{domain}}”,
“registryEndpoint”: “registry.{{domain}}”,
“deltaEndpoint”: “https://delta.{{domain}}”,
“mixpanelToken”: “__unused__”,
“balenaRootCA”: “{{base64}}”,
“apiKey”: “1ZwF7DtX3IeOnJgtbpVWBWunJGjAdCgI”
}

So once host image file was modified correctly with this json file and flashed to SD, then what happened in device side would be out of my development machine’s control.

30

Hi, I found the following recent ticket that seems to be addressing the issue you are talking about: start-resin-supervisor: populate NODE_EXTRA_CA_CERTS when custom CA i… · balena-os/meta-balena@4cf8239 · GitHub
I do not think such a recent OS version is available for production download, but you may try it out on our staging environment: https://dashboard.balena-staging.com/. Alternatively you may try building an image yourself (GitHub - balena-os/balena-raspberrypi: Balena support for RaspberryPI boards).
If you download an image from the staging environment it will have a config.json prepopulated, so you still need to configure balena os configure on top of that image to change it to point to your open balena server.
Please let us know whether this solves the issue for you.
Thanks,
Zahari

32

I got confirmation from our team that this was a regression between around 2.54 and 2.66, so the current production version is indeed affected by it.

1 Like
33

I confirm that the regression has been fixed in version balenaOS 2.67.3+rev4 (available on Balena Staging env)
Thanks a lot guys for your impressive work