Using real (not self-signed) certificates

@phoenixmage whilst this will load your cert and expose it on port 443, you will have difficulty getting your devices to connect since they will not trust the CA which signed that certificate.

I will have to have a think about this mechanism; there is a way to do it, but I need to consult some people internally.

@richbayliss I am sure there would be a way to inject it into the BalenaOS image, I am just not sure which partition it has to happen in. I will experiment and see.

@richbayliss So if you add the base64 encoded CA cert to balenaRootCA in /mnt/bot/config.json and remove the resin-supervisor container and systemctl start resin-supervisor it will use the new root CA

@phoenixmage Hey, just a quick update for you. I have a working solution for this scenario and it is pending a PR which should happen this week. When it does I will update you here.

Basically the current VPN trust configuration is tied to the BALENA_ROOT_CA and so while you can change this value, you would also need to issue a new VPN Sub-CA and server cert. So I am PRing a change to remove the trust chain, meaning you can change the BALENA_ROOT_CA value in your configuration to match your enterprise CA and still keep a working VPN. I have tested it myself and it worked nicely. You will also be able to make your own cert-provider image to load in your server’s cert and then renew it/reload the HAproxy config without changing the HAproxy container.

1 Like

@richbayliss any update on this? I see its merged into master but I can’t find any instructions on how to use is. Using the quickstart script with the -c flag still results in self signed certs being generated.

Cheers
Chris

@richbayliss nevermind found the issue. Changes work perfectly, thanks for your efforts on this one as I think it really simplifies the deployment and usage of openBalena in a secure manner.

For anyone else trying this out add the -c flag to quickstart script when setting up your openBalena server.

If you have any issues tail to cert-provider containers logs. In my case it was indicated DNS resolution failure for the DNS name I provided to quickstart. This was due to delay in propagation of DNS changes I had made right prior to start openBalena. After waiting for DNS name to propagate and restarting openBalena environment it all works now.

The only other questions I had are in terms of certificate renewal for both the openBalena server and devices. Is that automated or is there some sort of process required.

Cheers
Chris

Hey @dash, glad it’s working for you - the renewal should be automatic, but please highlight if it isn’t :+1:

1 Like

Hi @richbayliss,

Just thought I would report that it appears renewal has stopped working as of November due to this:

Using openBalean v2.0.0 branch when certificate renewal is require log output is as follows:

    cert-provider_1_9a99587c14da | [Info] VALIDATION not set. Using default: http-01
cert-provider_1_9a99587c14da | [Info] Waiting for domain.com to be available via HTTP...
cert-provider_1_9a99587c14da | (1/6) Retrying in 5 seconds...
cert-provider_1_9a99587c14da | (2/6) Retrying in 5 seconds...
cert-provider_1_9a99587c14da | (3/6) Retrying in 5 seconds...
cert-provider_1_9a99587c14da | [Info] Using STAGING mode
cert-provider_1_9a99587c14da | [Info] Waiting for domian.com to be available via HTTP...
cert-provider_1_9a99587c14da | [Info] Issuing certificates...
cert-provider_1_9a99587c14da | [Sat Dec 28 00:20:00 UTC 2019] Using stage ACME_DIRECTORY: https://acme-staging.api.letsencrypt.org/directory
cert-provider_1_9a99587c14da | [Sat Dec 28 00:20:01 UTC 2019] Standalone mode.
cert-provider_1_9a99587c14da | [Sat Dec 28 00:20:01 UTC 2019] Standalone mode.
cert-provider_1_9a99587c14da | [Sat Dec 28 00:20:01 UTC 2019] Standalone mode.
cert-provider_1_9a99587c14da | [Sat Dec 28 00:20:01 UTC 2019] Standalone mode.
cert-provider_1_9a99587c14da | [Sat Dec 28 00:20:01 UTC 2019] Registering account
cert-provider_1_9a99587c14da | [Sat Dec 28 00:20:06 UTC 2019] Register account Error: {
cert-provider_1_9a99587c14da |   "type": "urn:acme:error:unauthorized",
cert-provider_1_9a99587c14da |   "detail": "Account creation on ACMEv1 is disabled. Please upgrade your ACME client to a version that supports ACMEv2 / RFC 8555. See https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430 for details.",
cert-provider_1_9a99587c14da |   "status": 403
cert-provider_1_9a99587c14da | }
cert-provider_1_9a99587c14da | [Sat Dec 28 00:20:06 UTC 2019] Please add '--debug' or '--log' to check more details.
cert-provider_1_9a99587c14da | [Sat Dec 28 00:20:06 UTC 2019] See: https://github.com/Neilpang/acme.sh/wiki/How-to-debug-acme.sh
cert-provider_1_9a99587c14da | [Info] Installing certificates...
cert-provider_1_9a99587c14da | [Sat Dec 28 00:20:06 UTC 2019] Installing cert to:/tmp/cert.pem
cert-provider_1_9a99587c14da | [Sat Dec 28 00:20:06 UTC 2019] Installing key to:/tmp/key.pem
cert-provider_1_9a99587c14da | [Sat Dec 28 00:20:06 UTC 2019] Installing full chain to:/tmp/fullchain.pem
cert-provider_1_9a99587c14da | [Sat Dec 28 00:20:06 UTC 2019] Run reload cmd: cat /tmp/fullchain.pem /tmp/key.pem > /certs/open-balena.pem
cert-provider_1_9a99587c14da | [Sat Dec 28 00:20:06 UTC 2019] Reload success
cert-provider_1_9a99587c14da | [Info] Waiting for domain.com to use a staging certificate...
cert-provider_1_9a99587c14da | (1/3) Retrying in 5 seconds...
cert-provider_1_9a99587c14da | (2/3) Retrying in 5 seconds...
cert-provider_1_9a99587c14da | (3/3) Retrying in 5 seconds...
cert-provider_1_9a99587c14da | [Error] Unable to detect certificate change over. Cannot issue a production certificate. [Stopping]

I’m investigating solution going forward. Will let you know if I find anything of use.

Cheers
Dashals

Sorry @richbayliss - false alarm. I went digging and found your commit:

Because you were clearly all over this I started digging and found that somehow the last time I pulled latest images the cert provider was not updated as expected. Manually pulled latest cert provider image and it all worked as expect.

Advice for anyone currently using cert provider and open balena is to update to v2.0.0 before next cert renewal is due otherwise you might have problems when this date occurs.

Cheers
Dashals

I tried the latest build and it still failed. I had to update src/cert-provider/Dockerfile git checkout version for acme.sh from 2.8.2 to 2.8.3 for it work. i hope this helps

Hey. Is there any doco on how to use with LetsEncrypt now? I’ve looked at the PRs but not sure where to start.

I was just reading Devices offline & VPN Error: Cannot load DH parameters from dh.pem which might be helpful for some.

It didn’t work for me, admittedly and I’m still getting SELF_SIGNED_CERT_IN_CHAIN: request to https://api.{mydomain}/login_ failed, reason: self signed certificate in certificate chain.

It has to be done when the quickstart script is run; there is a flag to enable the cert provider service. You could also manually alter the activate file to enable it.

I used the -c flag, if that’s the one you are referring to?

I’ve not come across the activate file yet, but will see if I can find it and what it contains :+1:

Found it; And it does contain export OPENBALENA_ACME_CERT_ENABLED=true.

I checked the pingability of api.{mydomain} and that is working.

Did the restart of the cert provider too. No luck

check the logs of the cert provider service, it might give some clues as to the reason it isn’t swapping in the ACME cert (if it has managed to acquire it)

You may also need to pull the latest master and rebuild the cert provider service as I know some older versions stopped working due to LetsEncrypt changing their protocol.

I’ve obfuscated my domain, but here’s what the cert provider logs is showing:

[Info] VALIDATION not set. Using default: http-01
[Info] Waiting for api.{mydomain} to be available via HTTP...
cat: can't open '/usr/src/app/certs/last_run_mode': No such file or directory
[Info] Last acquired certificate for
[Info] Using STAGING mode
[Info] Waiting for api.{mydomain} to be available via HTTP...
[Info] Issuing certificates...
[Wed Jan 15 14:14:20 UTC 2020] Using stage ACME_DIRECTORY: https://acme-staging.api.letsencrypt.org/directory
[Wed Jan 15 14:14:21 UTC 2020] Standalone mode.
[Wed Jan 15 14:14:21 UTC 2020] Standalone mode.
[Wed Jan 15 14:14:21 UTC 2020] Standalone mode.
[Wed Jan 15 14:14:21 UTC 2020] Standalone mode.
[Wed Jan 15 14:14:21 UTC 2020] Registering account
[Wed Jan 15 14:14:22 UTC 2020] Register account Error: {
  "type": "urn:acme:error:unauthorized",
  "detail": "Account creation on ACMEv1 is disabled. Please upgrade your ACME client to a version that supports ACMEv2 / RFC 8555. See https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430 for details.",
  "status": 403
}
[Wed Jan 15 14:14:22 UTC 2020] Please add '--debug' or '--log' to check more details.
[Wed Jan 15 14:14:22 UTC 2020] See: https://github.com/Neilpang/acme.sh/wiki/How-to-debug-acme.sh
[Info] Installing certificates...
[Wed Jan 15 14:14:23 UTC 2020] Installing cert to:/tmp/cert.pem
cat: can't open '/usr/src/app/certs/api.{mydomain}/api.{mydomain}.cer': No such file or directory
[Error] Unable to acquire a staging certificate. [Stopping]

Related to https://github.com/balena-io/open-balena/pull/72 perhaps?

Please upgrade your ACME client to a version that supports ACMEv2

You need the latest master I think :+1:

Latest master does not include the acme 2 upgrade as far as I can tell. I’m trying from the cert-provider-update branch now…

[edit] to be clear, I pulled that branch in, so it’s actually a merge on https://github.com/balena-io/open-balena/releases/tag/v2.0.0

Ah, I see - sure I will take a look at getting this 2.8.3 version into master later.