problems with self-signed certificate

Hi, everyone!
We have some difficulties connecting the jetsons to the balena due to the self-signed certificate:

curl https://api.balena...ru
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: curl - SSL CA Certificates

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Can we replace the self-signed certificate painlessly? Or are we just doing something wrong?

When trying to send data to the balena, jetson throws the following errors:

Hello just to clarify the self signed is on your server??? If so you could use https://letsencrypt.org/ to have a valid valid certificate on your server.

Yes. We use this certificate:

config/certs/root/ca.crt

I understand correctly that we can change it to a wild card from the https://letsencrypt.org/ , or from somewhere else?

Hi @kvshumsk, judging from the screenshot you experience the issue on the device, right? Not on the server?

So you could add the certificate to the config.json of the device. We actually have an open PR that’s about to be merged that documents this:

The relevant repo: https://github.com/balena-os/meta-balena#configjson

And we advise balena users to use configizer to edit config.json safely:

Please use caution if you are editing config.json of a running device. A simple mistake could make the device inaccessible. That’s why we advise using configizer.

Let us know how it goes.

The fact curl fails suggests you haven’t installed the cert in your system. The fact the device fails to communicate with the backend suggests it hasn’t been installed correctly on the device. Can you please ensure you’ve followed the Getting Started guide correctly? Was this certificate generated using the open-balena quickstart script? How did you provision the device?

Hello,

I have exactly the same problem with a fresh install of open-balena

Here is what I have found

On Balena OS:

root@4da4116:~# whoami
root

root@4da4116:~# ls /etc/ssl/certs/balenaRootCA.pem
/etc/ssl/certs/balenaRootCA.pem

root@4da4116:~# curl https://api.sanitized.domain/ping
OK

Everything is okay, connection is possible and cert is setup

In the Supervisor container:
balena exec -it f9aa8b9b65a2 /bin/sh

/usr/src/app # whoami
root

/usr/src/app # ls /usr/local/share/ca-certificates/balenaRootCA.crt
ls: /usr/local/share/ca-certificates/balenaRootCA.crt: No such file or directory

/usr/src/app # ls /etc/ssl/certs/balenaRootCA.pem
ls: /etc/ssl/certs/balenaRootCA.pem: No such file or directory

/usr/src/app # curl https://api.sanitized.domain/ping
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned 

The environment of Supervisor

/usr/src/app # env
BOOT_MOUNTPOINT=/mnt/boot
CONFIG_MOUNT_POINT=/boot/config.json
DEFAULT_MIXPANEL_TOKEN=bananasbananas
DELTA_ENDPOINT=https://delta.sanitized.domain
DOCKER_ROOT=/mnt/root/var/lib/docker
DOCKER_SOCKET=/var/run/balena-engine.sock
HOME=/root
HOSTNAME=4da4116
LED_FILE=/sys/class/leds/led0/brightness
LISTEN_PORT=48484
MIXPANEL_TOKEN=__unused__
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/usr/src/app
SHLVL=1
SUPERVISOR_CONTAINER_ID=f9aa8b9b65a2d03ea00eb997f41c1bb4c099ee313353224f5f7ba02a74ce8b73
SUPERVISOR_IMAGE=balena/aarch64-supervisor:v12.2.11
TERM=xterm

And to finish, it seems that the entry.sh should do the following

if [ -n "${BALENA_ROOT_CA}" ]; then
        if [ ! -e '/etc/ssl/certs/balenaRootCA.pem' ]; then
                echo "${BALENA_ROOT_CA}" > /etc/ssl/certs/balenaRootCA.pem

                # Include the balenaRootCA in the system store for services like Docker
                mkdir -p /usr/local/share/ca-certificates
                echo "${BALENA_ROOT_CA}" > /usr/local/share/ca-certificates/balenaRootCA.crt
                update-ca-certificates
        fi
fi

but sadly, the environment variable BALENA_ROOT_CA does not exists

echo ${BALENA_ROOT_CA}
    nothing

So at startup, no certificates are added

I can confirm that my config.json is mounted and is valid and the variable balenaRootCa is correct

So the real question / bug here is:
How do we take cert from config.json and setup in $BALENA_ROOT_CA variable so that the entrypoint setup the certificate?

Hi, the certificate should be automatically inserted in your image when you run the balena os configure step from the user guide. However it is necessary for the certificate to be installed locally on the development machine according to the user guide for this to work. That means ensuring the NODE_EXTRA_CA_CERTS variable is exported and the rest of the described steps like making sure balena login works from the development machine. Can you please confirm that you reach the balena login step from you development machine successfully?
Thanks,
Zahari

Hello Zahari, yes indeed I followed the steps and has exported the NODE_EXTRA_CA_CERTS variable.

I can login, create an app, deploy app, add devices and list devices via the CLI.

However, it seems that the supervisor container on the device has no cert set up on it, so it cannot talk to the open-balena server

Today I came across a similar issue and I found a relevant commit. Can you please check my response there and let me know if it applies to you: balenaOS self-signed certificates error - #30 by majorz

I got confirmation from our team that this was a regression between around 2.54 and 2.66, so the current production version is indeed affected by it.

Excuse me. After updating the balena, there is no need to redeploy apps and reconnect devices?

Hi @kvshumsk

After a balenaOS upgrade, there should be no need to manually redeploy apps or “reconnect” devices. Are your containers not restarting after the upgrade?

Kind regards
Alida

Hallo! We upgraded to version 2.67 and the problem went away. Thank you for help!

I’m glad that it is now working for you.

I believe I’m encountering a related problem with Openbalena and a Jetson Nano 2GB (Nvidia Jetson Nano 2GB Devkit SD - BalenaOS 2.67.3+rev3) when trying to deploy an app to it.

[Info]    Pushing images to registry...
Retrying "registry.[redacted-domain]/v2/90238647833d2dd6c16138eea43b5c80:latest" after 2.0s (1 of 2) due to: Error: unauthorized: authentication required
Retrying "registry.[redacted-domain]/v2/90238647833d2dd6c16138eea43b5c80:latest" after 2.8s (2 of 2) due to: Error: unauthorized: authentication required
[Info]    Saving release...
[Error]   Deploy failed
unauthorized: authentication required

I followed the quickstart docs for openbalena, connected my local cli no problem, created an app and provisioned the devices all without issue - however deploying an app to the device I get the authentication error.

Hello,

Can you please provide more details:

  • on what OS are you running the cli?
  • which docker version did you install on the server? make sure to use docker.io
  • have you configured the root certificate on the local machine?
  • did you configure NODE_EXTRA_CA_CERTS for local machine?

You can see this previous forum thread if it’s helpful for you?

Let us know
Thanks

@barryjump Are you still encountering the problem with Openbalena? If so, please provide the additional information Rahul requested and we’ll try to help you get it fixed.

Morning guys, no I managed to get it working with a fresh install of openbalena on a new VM. Not sure what happened the first go around. Thanks for the help though!

@barryjump Great stuff, that’s what we’re here for :wink:

1 Like