Let's Encrypt Certificate expired

openBalena
1

Hey,

My infrastructure has been running for one and a half year by now.
I’m using version 1.3.0 of OpenBalena.
I’m using a Let’s Encrypt certificate generated and managed by OpenBalena.

As I understand, that certificate is created for 90 days and automatically refreshed when needed.
It seems that has worked for 1,5 years because I never had any errors related to expired certificate.

However today I realized that one of my Raspberry Pi had an error because it is blipping it’s LED, indicating it doesn’t have access to the Balena instance.

When I tried to connect to the Balena CLI to investigate is when I realized my certificate was expired.

CERT_HAS_EXPIRED: request to https://api.<mydomaine>/login_ failed, reason: certificate has expired

If I go to the api from my web browser, same story. The certificate was generated 3 months ago, and has now been expired for 3 days.

Validity
Not Before: Fri, 16 Apr 2021 23:09:24 GMT
Not After: Thu, 15 Jul 2021 23:09:24 GMT

I connected to my Balena Instance (running on DigitalOcean with no issues since october 2019), and tried restarting the cert-provider container with:

./scripts/compose restart cert-provider

That didn’t change anything. I also looked at the logs of that container, but there is nothing suspicious. Extract since I restarted (I first restarted the server, then a bit later manually restarted only the cert-provider)

balena-cert-provider-logs
balena-cert-provider-logs768×384 5.39 KB

For now my infrastructure “works” because the actual work takes place outside the Balena Instance, so I’m not in a critical situation. However, I cannot access my devices from Balena, and cannot login to the Balena CLI because of this.

I would appreciate your help in forcing the the Let’s Encrypt certificate to update.
Thanks in advance

Tim

2

Hey Tim,

We should first check if the certificate in use by openBalena is valid. Can you give a try to ‘curl https://api.your_domain/ping’ from inside the openBalena server?

By the way, I think the server needs a restart after the cert-provider is restarted. Might be worth trying if the curl command doesn’t return OK

4

Hey,

the ping from withing the OpenBalena server returns the following:

curl-fails
curl-fails695×112 2.35 KB

An extra restart changes nothing sadly.

I left the full URL of my server in the screenshot, if you want to go check the certificate out or run any tests on your side.

Tim

5

Tim, how are you renewing our certificate? Do you have a LetsEncrypt agent running that is handling the renewal (along with the challenge that verifies domain control)? Is that agent still running? If the cert is expired, I would expect that is where the problem lies.

7

Tim, sorry, I meant to say “how are you renewing THE certificate”. I suspect that the agent might be down.

8

Hey,

I did not setup anything to renew the certificate. Since it’s creation was handled by OpenBalena, I figured it would also handle the renewal.

I can confirm it did handle the renewal, because as I said, every certificate lasts 3 months, and it renewed them without any intervention from me for the past 1,5 years.

But since the 15th of July, it hasn’t updated it.

After digging into more logs with ./scripts/compose logs, I think I finally found it:

cert-provider_1  | crond: USER root pid 4073 cmd "/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh" > /dev/null
cert-provider_1  | [Wed Jul 28 00:09:53 UTC 2021] Error, can not get domain token entry api.<your-domain>
cert-provider_1  | [Wed Jul 28 00:09:53 UTC 2021] Please add '--debug' or '--log' to check more details.
cert-provider_1  | [Wed Jul 28 00:09:53 UTC 2021] See: https://github.com/Neilpang/acme.sh/wiki/How-to-debug-acme.sh
cert-provider_1  | [Wed Jul 28 00:09:53 UTC 2021] Error renew api.<your-domain>.

I’ll post this now, but I’ll keep digging into it and I’ll keep you updated. If you have any idea what the issue is, I would appriciate your help.

Tim

9

Well looks like I found the error:

acme.sh version 2.8.1 (inside the cert-provider container) tries to get a new cert from:

https://acme-v01.api.letsencrypt.org/directory

{
  "type": "urn:acme:error:serverInternal",
  "detail": "ACMEv1 is deprecated and you can no longer get certificates from this endpoint. Please use the ACMEv2 endpoint, you may need to update your ACME client software to do so. Visit https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430/27 for more information."
}

So year. It’s deprecated. I guess that’s what happens when you run an old version of a software (OpenBalena in my case). I’m going to try and update acme.sh and keep you updated.

10

Okay, here is the solution for anyone running an old version of OpenmBalena:

The problem is that ACMEv1 was deprecated recently, but the cert-provider included in OpenBalena hasn’t been updatd to use ACMEv2.

Here is what I did to fix it. On the Server running OpenBalena:

# Find out the id of the cert-provider container
docker ps
# Run bash in the cert-provider container
docker exec -it <mycontainer> bash
cd /root/.acme.sh
# Get the latest version that will support ACMEv2
./acme.sh --upgrade
# Register a new account on LetsEncrypt
./acme.sh --register-account --server letsencrypt
# Set it as default
./acme.sh --set-default-ca --server letsencrypt
# Find the folder containing your cert and cert config
cd /usr/src/app/certs/api.<your-domain>
# Edit the config file. Only vim is available
vi api.<dour-domain>.conf

Replace all references to https://acme-v01.api.letsencrypt.org with https://acme-v02.api.letsencrypt.org.
Save and exit.
Run the certificate updater (with debug to see that everything is working)

"/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh" --debug

Main references for finding the solution:

3 Likes
11

Nice! Thanks @winkelmann ! :clap: I’ll document this and also make sure that the latest version of OpenBalena is doing the right thing.

27

hi @Winkelmann
I am running openbalena 2.0.0
I’m facing the same issue with my certificates but I can’t follow your steps.
here are the steps i could reproduce:

image
image847×313 11.7 KB

but then i cant seem to find any folder with domain specific configuration so im stuck at:
cd /usr/src/app/certs/api.

Edit the config file. Only vim is available

vi api..conf
what version of openbalena were you using?
I would appreciate any help thanks

28

Hey,

I was and am still using version 1.3.0.
I believe the path I gave should be the same for everyone,. Since we are inside the Docker Container, the folder containing the certificates is controlled by the people that developed Balena.

My best guess is, that folder was moved elsewhere between version 1.3.0 and 2.0.0

I don’t have that version or the time to deploy it for testing, so I’m afraid I can’t help you much.

You could do a file system search (inside the docker container) for the file “api.<dour-domain>.conf”. If you’re lucky, it might point you to the correct folder.
Otherwise, you are going to need to ask a Balena developper where the certificates are stored in version 2.0.0 (@toochevere ?)

Hope it helps
Tim