Using real (not self-signed) certificates

#1

Thanks for providing openBalena, it’s a great addition to the family!

We’re having a few issues which I think mostly stem from the use of the self-signed certificates. I’m wondering if we need to use self-signed in the first place, or if we can use real ones, if so, where do we put them? I can see files in ~/open-balena/config/certs/api which look relevant.

I was thinking of using certbot (certonly) as we use cloudflare for DNS so can easily handle the auth side of generating certificates from Let’s Encrypt.

Is it going to be fairly simple to swap out for real certificates or is this not a good idea?

Again, thanks for the great products :slight_smile:

1 Like
#2

Hi @violuke

Firstly, thanks for trying out openBalena and providing us with your feedback; it’s great to see people starting to use it and getting excited about it.

The cert generation and usage can be tricky in openBalena so I would recommend the following:

  • unbind the ports 80 and 443 from haproxy container to the host.
  • add a new service in the compose file using “Caddy” which will host-bind on 80 and 443. I used this one https://github.com/abiosoft/caddy-docker.
  • configure the Caddy container to use TLS, listen for requests to *.your-domain.here, and use “haproxy:80” as the backend.
  • re-start the stack.

Caddy is a web server which will go off to LetsEncrypt and do the cert management for you; so no need to mess with the certs in openBalena. It can be tricky to setup if you’re new to containers, but I am hoping to make some tools & guides available to help with as soon as I get the time.

Let us know how you get on, and thanks again for using openBalena!

1 Like
#3

Hi,

Thanks for the pointers. I’ll have a look into this and let you know how I get on.

2 Likes
#4

Awesome, please report back here with your results as I am sure others will find it useful.

1 Like
#5

Hey @violuke, how did it go? I’m not super confortable with certificates in general, still trying to figure out what would be the best way to set up openBalena with signed certificates.

0 Likes
#8

Hi,

Sorry for the lack of response. I’ve not had much time to spend on this recently, but will give you an update when I do. Thanks.

0 Likes
#9

Hi @edorgeville
We are not using Cloud Flare, but rather Azure.
We also use a wildcard Let’s Encrypt (LE) certificate.
My suggestion based on @richbayliss’ post is the following:

!! If you try this, please make sure to use the LE’s staging environment so you don’t run into issues. !!

The great advantage of this is, it also works when you have your balena infrastructure in a private network since it uses DNS authentication for LE.

1 Like
#10

I’ve found some time today to look at this (before I saw UncleSam’s message) and went down the Caddy route. Like UncleSam, our entire infrastructure was within a firewall but instead of using DNS/Cloudflare auth, I swapped to a non-wildcard cert and opened just port 80 to the server. The changes ended up being pretty simple. Although I might close port 80 and use DNS auth down the line.

You can see them here https://github.com/balena-io/open-balena/compare/cec371f0b87d90eaf15462f6a8e961461a14405a...violuke:0e7a8b0c7b1ba53dc83f2732ce89f5df9a39e33e

Hopefully this is of some help to others

I can’t get a socket connection to the device (so cannot issues reboots for example), but it does show as online. I’m not sure if this is caused by my changes yet or not as I never got this far before on the self-signed certificates. When trying a reboot I get:

$ balena device reboot 4eXXXXX
BalenaRequestError: Request error: tunneling socket could not be established, cause=socket hang up

If you need help, don't hesitate in contacting us at:

  GitHub: https://github.com/balena-io/balena-cli/issues/new
  Forums: https://forums.balena.io
1 Like
#11

The issue I ran into doing this was that our VPN connects in on port 443, like HTTPs does, but HAproxy is configured to identify it and send it to the VPN service container. Because of the way Caddy binds to 443 in this solution, I think it will affect the VPN functionality.

The way I solved this was to change the main HAproxy container itself; including the acme.sh script and some supporting supervisord scripts to allow it to restart the haproxy daemon on cert acquisition/renewal.

The reason I haven’t pushed this live is because we have a separate project which will be doing this, or at least me close enough to add in this functionality and I don’t want to duplicate code/effort.

0 Likes
#12

Humm, yeah, this does look like a problem with the Caddy route.

@richbayliss how long do you think it will be until there’s an “official solution” as I don’t want to spend much more time on this if it’s close. Thanks.

EDIT:
I get these messages from Caddy’s logs (which presumably is the VPN being borked)

caddy_1_753cf1ffa561 | 2019/01/04 14:19:18 http: TLS handshake error from 10.0.3.140:59276: tls: first record does not look like a TLS handshake
caddy_1_753cf1ffa561 | 2019/01/04 14:20:39 http: TLS handshake error from 10.0.3.140:59328: tls: oversized record received with length 34295
caddy_1_753cf1ffa561 | 2019/01/04 14:22:40 http: TLS handshake error from 10.0.3.140:59330: tls: oversized record received with length 31740
caddy_1_753cf1ffa561 | 2019/01/04 14:24:41 http: TLS handshake error from 10.0.3.140:59342: tls: oversized record received with length 51482
0 Likes
#13

how long do you think it will be until there’s an “official solution”

I don’t have a time on anything official at this point, sorry.

(which presumably is the VPN being borked)

I would agree.

0 Likes
#14

Ok, so if it the official solution not just around the corner, I’ll tinker a bit longer…

I’ve made further changes (still building on Caddy - but not sure this is the best plan tbh) so 80 and 443 traffic goes to haproxy, then 80 and non-vpn 443 forwards to Caddy (and Caddy does it’s stuff and forwards back to haproxy). In theory the vpn traffic should do as it did before.

Changes here:

So actual TLS 443 still seems to be working ok, I’m not getting the Caddy TLS errors - suggesting that vpn traffic is being filtered out ok, but I still cannot use the reboot command. The error has changed slightly however, from:

BalenaRequestError: Request error: tunneling socket could not be established, cause=socket hang up

to

BalenaRequestError: Request error: tunneling socket could not be established, cause=connect ECONNREFUSED 172.18.0.7:3128

Any ideas where this is going wrong? Thanks.

1 Like
#15

So OpenBalena is actually quite useless for any situation except a small proof of concept setup. Or is there any other way to swap out the self signed certificate for a real world situation? It’s 2019, the use of certificates isn’t that much of a rocket science anymore…

0 Likes
#16

Hello, I won’t comment on your tone, but I do want to comment that openBalena is not production-ready as mentioned in the README. For any bugs you’d like to see resolved or features implemented, pull requests are very welcome, as well as tickets on which we can discuss.

0 Likes
#17

sorry for the tone. I should not have done that. I had to adjust my expectations beforehand. I may not have read that correctly in README.
Thanks for the work so far, I’m going to try to make a solution. If it works properly, I will make the share / pull request.

0 Likes
#18

My solution is working fine for us (it’s only a few devices so far). Some things like SSH and reboots are not, but I don’t think that is caused by the changes that I’ve made. Devices connect to openBalena fine and updates to those devices get pushed correctly on deploy.

0 Likes
#19

It feels to me like a unnecessary complex hack, adding a module in stead of fixing the real problem. It should work without Caddy when HAProxy uses the correct certificate chain directly.
I think i’ll give it a shot. Maybe some additional code in scripts/make-env to build (and update) a correct /config/activate file.

1 Like
#20

Update:
-got it working by modifying config/activate.
-now testing is everything is working well.
todo: automate it. And make it work with the Lets encrypt workflow.

Anyone an idea how to replace the HAProxy certificate without restarting the complete (docker) process?

1 Like
#22

I have PR’d a solution to the LetsEncrypt certificate requirement. It will need to go through review, but if you’re interested in taking a look you can find the PR here: https://github.com/balena-io/open-balena/pull/38
We may still make changes to this, so please don’t consider it to be final; I plan to make a post about how to utilise the changes once they are approved and merged.

4 Likes
#24

I am interested in using an internally signed CA for my stuff so leveraging @richbayliss’s solution now it is merged I did the following.

  • Generated a new private key
  • Created a CSR which uses subject alt name for each required CNAME (registry, vpn, etc) in the CSR
  • Signed the CSR with my internal CA
  • Created open-balena.pem which contains the full cert chain and the private key appended

You can either find the physical location of the named cert volume (docker volume inspect openbalena_certs and look for Mountpoint) or copy the pem to haproxy/open-balena.pem and add COPY open-balena.pem /certs/open-balena.pem to the haproxy/Dockerfile

I needed to rebuild my haproxy container using ./scripts/compose up -d --build haproxy

After that my cert CA signed cert is now used.

0 Likes