Device not online / Production

Hi all,

First of all, I think it’s awesome that there is an open-source version of the Balena platform!

I know it’s in the beta stadium, but I have some questions.
First of all, I saw the issue that devices are appearing offline on Github. The issue is closed and I’ve followed the openBalena Guide on a server, but my device is still appearing offline in the balena-cli. I think it has something to do with the self-signed certificate on the server, but correct me if I’m wrong. How can I force the device to trust this certificate? Because the BalenaOS is read-only, I can’t place the certificate on the device.

And the BalenaOS, I’ve read on the BalenaOS docs that it’s the development image. So ssh is without a password and some other features that help developing the app. But how can I create production images? Or is this something for in the near-future?

I’m looking forward to use openBalena and BalenaOS for our (new) projects in the future!

Thanks,

Bart

Hello, there are two parts to the device offline issue — one was that we did not properly configure the API to forward the OpenVPN config to devices (the issue that you refer to) and the other is in balenaOS which you can track the status here: https://github.com/balena-os/meta-balena/issues/1101. I expect we’ll have this solved next week. As a temporary workaround, you could remount the boot partition as read/write, install the cert and reboot.

Regarding the dev variant, yes we have plans to make switching between dev and prod variants easy, hopefully real soon but I don’t have an ETA. It would be great if you could file an issue on the open-balena repo about this.

Hi @dfunckt,

Thanks for the quick reply.
I’ve looked everywhere for the issue regarding the certificate, but I couldn’t find it. I’m gonna try the workaround you suggested for the moment.

I’ll create an issue regarding the production variant in the open-balena repo and I’ll edit my post here with the link.

EDIT: Production image for supported boards

I’ve mentioned it in the ticket, but I’ll mention it here again for everyone else – you can now download production images from balena.io/os. Thanks for bringing this up @bversluijs!

After reading this topic, I’ve decided to post my progress with the certificate issue. I’ve tested the RPI 3 image. I’ve mounted the image on macOS and the “boot” partition mounted to my system, because that’s the only partition macOS can read. This partition is read/write.

I’ve added the openBalena ca.crt to this partition, ejected the image and flashed it to a SD card. I’ve booted the RPI 3 with the SD card and located the ca.crt at /mnt/boot. This is where I ran into problems. The next step is adding the ca.crt to the trusted certificates (obviously). In the openBalena guide, there is a section to add the cert to trusted certificates for Windows, MacOS and Linux. I’ve followed these steps for Linux, but because it’s a read-only system, it couldn’t add the certificate to the trusted section.

I haven’t had the time to do more research into this, and it’s obviously a workaround for the time-being (plz fix it), but these are the steps that “succeeded”. I hope someone can tell me and others the steps to bring it to a 100% success story!

And last but not least, @dfunckt thanks for the quick production support images! Once the openBalena is running okay, with the certificate issue fixed and a detailed guide, I’m surely going to use it for all of our projects! I really can’t wait to use this awesome product and helping to improve it!

I’ve had some time to investigate the issue further. I was able to remount / to a read/write state with the following command:
mount -o remount,rw /

After that, I’ve copied the ca.crt from /mnt/boot to /etc/ssl/certs and ran “update-ca-certificates”, but without success. So then I just added the ca.crt content to /etc/ssl/certs/ca-certificates.crt. Now I was able to connect to my server without an SSL issue. I’ve tested it with:
curl https://api.myserver.com/

But, the device still appeared to be offline :thinking:

So I’ve investigated the problem further, and logged in to the resin-supervisor container with:
balena exec -it [CONTAINER_ID] /bin/sh

I’ve found out that balena already has some support for the openBalena certificate. But that doesn’t work properly, because when I wanted to connect from the container to my server, I got an error. I’ve tested that with wget, because the container doesn’t have curl

I’ve tried to get it to work, but unfortunately I couldn’t fix it. Maybe the Balena staff can look into it, because the resin-supervisor is not open-source afaik.

I have a theory why the device keeps appearing offline, even after my workaround. The resin-supervisor uses a VPN to send all data. Because of this, the VPN connects to the openBalena server API, but the VPN doesn’t have the ca.crt of the openBalena server installed. So that connection fails.

Can someone (@dfunckt / balena team) confirm this? My only other solution is to make a Let’s Encrypt script that run’s and update the ca.crt with a signed certificate. That’d probably be the best solution for production of course. But I don’t know if the SSL certificate is the real problem why the devices appear offline…

EDIT: SSL EXPLANATION
I’m wrapping my brain around how the SSL certificates of openBalena are configured. There are so many certificates (openbalena/config/certs/[api,root,vpn]). Can somebody please explain how the SSL certficates are used? I think there’s a wildcard used for the SSL certificates for the API etc, but the VPN has it’s own certificate?

@bversluijs the VPN is only used to report online status to the backend and route SSH through it. The Supervisor does not use the VPN at all. It is open source BTW, like all balenaOS and openBalena software (which means if you’re wondering where something is, let us know and we can point you to the project :slight_smile: The Supervisor is here: https://github.com/balena-io/balena-supervisor

Regarding the offline issue – yes, it has to do with certificates in balenaOS – the balenaOS service that installs the certificates currently ignores the given cert so it can’t connect to the backend, retrieve the info it needs and configure the device. The fix is in this PR and will be coming with the next balenaOS release.

If you’re keen to do yourself the work that service does for you, you’ll need to SSH into the device, install the certs like you described, then curl https://api.yourinstance.io/os/v1/config and install the files at the appropriate locations on the device as described in the returned JSON.

@dfunckt Thanks for the update. I’ve tried to get Let’s Encrypt working, but I didn’t have much time. And I’ve checked the GitHub repositories, awesome that everything is open-source!

I’ll wait for the new BalenaOS update with openBalena. I’m wondering when are the new updates expected? Are the balenaOS updates weekly build or just whenever they’re ready?

I would expect the next release to be available within the next 2 weeks. BalenaOS releases are currently not on a strict schedule but we’ll soon be moving to a more predictable model. We’ll let you know more about it soon.

I’ve managed to get a Let’s Encrypt certificate to work with openBalena. I’ve tweaked the haproxy Dockerfile to use the Let’s Encrypt certificate instead of generated certificate. The certificate works (When I go to https://api.openBalenaServer.com/, the certificate is trusted and I can connect without any warnings. And it says it’s a Let’s Encrypt certificate).

However, the resin-supervisor can’t connect with it. When I do a curl/wget https://api.openBalenaServer.com/ from the HostOS, I get a good response. But when I exec into the resin-supervisor container, and I use wget, I get an error. Same as when the generated SSL certificate was installed. I thought that would fix the problem, but I got that wrong (Tested with a RPI 3B+ BalenaOS 2.27.0).

I was planning to use Let’s Encrypt when going to production, because a trusted SSL certificate is better (in most cases) than a self-signed SSL. certificate. I was hoping this would fix some problems, and it did! (Like balena login), but I was wondering why resin-supervisor still can’t connect with the server. I know in a new BalenaOS version, it’s going to be fixed, but I was wondering if I can use Let’s Encrypt in the future for the SSL connections?

Thanks in advance!

1 Like

Hi all,

Any updates regarding the offline status? I saw in the os-config repository that V1.0.0 is released a few days ago and according to this issue, that should fix the custom certificates issue, correct?

I understand you’re all focussing on fixing the custom root certificate first, but is there going to be support for other SSL certificates, for example like I tried with Let’s Encrypt?

I can’t wait to get started with openBalena and building an UI around it. For the development stages, I’m still using balenaCloud, but for some projects we would like to use openBalena for development and production devices :slight_smile:

Thanks in advance!

1 Like

I’m also wondering about this issue, re: devices that report as “offline”. I’ve used the getting started guide to setup an instance of open-balena, can login to the server, create apps, etc. But installing a production image to a device results in my server reporting the device in the assigned app, but always offline and inaccessible.

So if I’m understanding the thread here, is it that dev images work but production images don’t since the server’s certs are self-signed? In addition to the issue @bversluijs referenced, the PR that @dfunckt referenced (os-config#13) has also been merged.

If the issue is fixed, does the server need to be rebuilt to regenerate new certificates?

Hi @critzo,

It’s for development images and production images. Probably a new build for BalenaOS/openBalena should fix the issue, but that’s just guessing on my side. This guess is based on the issue that’s been solved like you mentioned.

We’re very excited to use openBalena and improving it, but probably we share the opinion that when SSH doesn’t work and deploying containers doesn’t work, it’s still unusable at the moment and not yet worth the while to invest time in openBalena. After investing some time in using Let’s Encrypt (so using valid SSL certificates), it still says offline. When this is fixed, we’re determined to invest some time and money in using openBalena.

But we’re still very positive about openBalena and using it for future projects, and nothing but love for the balena team in deciding for making it open source and investing their time in this open source project!

@critzo - the device showing as offline will be resolved soon with a new OpenBalena and balenaOS releases. I have been testing them over the last 24 hours and I can confirm that the device shows as Online (as it should) and that SSH into the host is also working. We are still doing bugfixes and further tests, but you can expect it will come as soon as it is ready.

@bversluijs - I am glad to read that more companies are looking at OpenBalena as an option and we are spending significant development effort in pursuit of improving the OpenBalena use-case. As above, you can expect improvements to come as soon as we can safely release them.

3 Likes

@richbayliss, This is great news! very much looking forward to this being solved.

Hi @richbayliss,

Awesome to hear! We’re very excited to use the new openBalena version! We really appreciate all the hard work from you guys to make such an awesome product open source.

We’re looking forward when the update is released!

Thanks @vedicum. I agree- the Balena team’s commitment to open source is a huge reason I’ve been using it too. The open balena server is super promising for self hosting and I’m looking forward to continuing to test it. Awesome service and project!

Thanks @richbayliss for the update. I’ve been really happy with the balena.io service and looking forward to continuing to test open balena. Does the team announce releases and other updates in some way or should I just watch for releases in the Github repositories (balena-os & open-balena).

Thanks for update @richbayliss,
Looking forward to use openBalena as a primary IoT container based solution. :+1: