cert-provider error maybe typo?

[Mawiguk0]

Here is a fresh docker-compose up from a fresh cloned openBalena repo.

registry_1       | Systemd init system enabled.
vpn_1            | Systemd init system enabled.
s3_1             | Systemd init system enabled.
haproxy_1        | [WARNING] 313/120720 (19) : Server vpn-tunnel/balena_vpn is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
haproxy_1        | [ALERT] 313/120720 (19) : proxy 'vpn-tunnel' has no server available!
haproxy_1        | [WARNING] 313/120720 (19) : Server vpn-tunnel-tls/balena_vpn is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
haproxy_1        | [ALERT] 313/120720 (19) : proxy 'vpn-tunnel-tls' has no server available!
cert-provider_1  | [Info] (2/3) Connecting...
cert-provider_1  | [Info] (2/3) Failed. Retrying in 5 seconds...
cert-provider_1  | [Info] (3/3) Connecting...
cert-provider_1  | [Info] (3/3) Failed!
cert-provider_1  | [Info] Unable to access api.ob-test.redacted-domain.de on port 80. This is needed for certificate validation. Retrying in 30 seconds...
haproxy_1        | [WARNING] 313/120726 (19) : Server backend_api/balena_api_1 is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
haproxy_1        | [WARNING] 313/120738 (19) : Server vpn-tunnel/balena_vpn is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
haproxy_1        | [WARNING] 313/120738 (19) : Server vpn-tunnel-tls/balena_vpn is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
cert-provider_1  | [Info] Waiting for api.ob-test.redacted-domain.de to be available via HTTP...
cert-provider_1  | [Info] (1/3) Connecting...
cert-provider_1  | [Info] (1/3) Success!
cert-provider_1  | cat: can't open '/usr/src/app/certs/last_run_mode': No such file or directory
cert-provider_1  | [Info] Last acquired certificate for
cert-provider_1  | [Info] Using STAGING mode
cert-provider_1  | [Info] Waiting for api.ob-test.redacted-domain.de to be available via HTTP...
cert-provider_1  | [Info] (1/3) Connecting...
cert-provider_1  | [Info] (1/3) Success!
cert-provider_1  | [Info] Issuing certificates...
cert-provider_1  | [Wed Nov 10 12:07:56 UTC 2021] Using stage ACME_DIRECTORY: https://acme-staging-v02.api.letsencrypt.org/directory
cert-provider_1  | [Wed Nov 10 12:07:57 UTC 2021] Standalone mode.
cert-provider_1  | [Wed Nov 10 12:07:57 UTC 2021] Standalone mode.
cert-provider_1  | [Wed Nov 10 12:07:57 UTC 2021] Standalone mode.
cert-provider_1  | [Wed Nov 10 12:07:57 UTC 2021] Standalone mode.
cert-provider_1  | [Wed Nov 10 12:07:57 UTC 2021] Standalone mode.
cert-provider_1  | [Wed Nov 10 12:07:57 UTC 2021] Create account key ok.
cert-provider_1  | [Wed Nov 10 12:07:57 UTC 2021] Registering account
cert-provider_1  | [Wed Nov 10 12:07:58 UTC 2021] Register account Error: {
cert-provider_1  |   "type": "urn:ietf:params:acme:error:malformed",
cert-provider_1  |   "detail": "JWS verification error",
cert-provider_1  |   "status": 400
cert-provider_1  | }
cert-provider_1  | [Wed Nov 10 12:07:58 UTC 2021] Please add '--debug' or '--log' to check more details.
cert-provider_1  | [Wed Nov 10 12:07:58 UTC 2021] See: https://github.com/Neilpang/acme.sh/wiki/How-to-debug-acme.sh
cert-provider_1  | [Info] Installing certificates...
cert-provider_1  | [Wed Nov 10 12:07:58 UTC 2021] Installing cert to:/tmp/cert.pem
cert-provider_1  | cat: can't open '/usr/src/app/certs/api.ob-test.redacted-domain.de/api.ob-test.redacted-domain.de.cer': No such file or directory
cert-provider_1  | [Error] Unable to acquire a staging certificate. [Stopping]

The second last line is some bogus, any idea what went wrong?

[Mawiguk0]

Hi there,
So this is probably my last attempt to get help or information in the openBalena Forums.

I always get this error, the deployment is based on a KVM-VM with a public Ip address and DNS entries set for :

• api.ob-test.redacted-domainname.de
• registry.ob-test.redacted-domainname.de
• vpn.ob-test.redacted-domainname.de
• s3.ob-test.redacted-domainname.de
• tunnel.ob-test.redacted-domainname.de
  all pointing to its public IP-Address of my Server/VM

The Docker Compose Up runs smoothly, with no errors or warnings that aren’t expected so far.

Is there a fix for this misbehavior? Because obviously, that .crt file is not in place, and I would figure that it is most likely the acme that is failing, if I wouldn’t get an “OK” on the API endpoint:

I already asked a question related to the balena-CLI which ultimately leads to this error because there is obviously a self-signed certificate installed, even though I used the “-c” for acme in the installation process:

./scripts/quickstart -c -p -d ob-test.redacted-domainname.de -U ******** -P **********

Here is a link to my other Thread about the CLI problem: (it is still unanswered anyways)

openBalena quickstart -c but still cant balena login without SELF_SIGNED_CERT_IN_CHAIN error

Hi, I used the getting started guide on a freshly installed box. I used the -c option on the quickstart script, everything ran without any errors. I thought -c would use letsencrypt + acme to get a cert. I’m kind of new to openBalena but would really like to get more into it.

Please don’t get me wrong or anything, I like the idea of how balena wants to solve IOT related situations, we in general here at our company really would love to test compatibility to maybe even switch completely with 300+ Devices to balenaCloud, but as long as I can tell, support is most likely the only way to get help from balena. This is not what I was expecting, especially because other threads, with sometimes really easy beginner issues get an answer and help. Whereas myself was greatly ignored for issued by now.

Have a good one, I would love to hear at least a “hi” or “we don’t have a clue either” or “please give more information”

[Mawiguk0]

Even with ‘domain…’ in the quickstart command it is not working

[Mawiguk0]

Even with a fresh deploy after a docker system prune no change in progress, I’m no looking into the repo another time

[builder555]

Hello,

My colleague has responded in your original thread, so we can continue our support there.

[bkaindl]

@Mawiguk0 The version of the acme.sh pulled by the current open-balena is still the original version added by @richbayliss in 2019. Since letsencrypt.org stopped supporting ACMEv1, it cannot work anymore and open-balena needs an update. See Cert Provider - Cannot issue a production certificate · Issue #108 · balena-io/open-balena · GitHub for reference.

@richbayliss @builder555 This PR should fix it: Get cert-provider working again by danzel · Pull Request #131 · balena-io/open-balena · GitHub

But in my case, this is not enough, below are my results with the fix of PR131:

It seems my VPS is rather slow and it needs a restart, these are the logs from “scripts/compose up -d”
Creating openbalena_db_1 … done
Creating openbalena_redis_1 … done
Creating openbalena_s3_1 … done
Creating openbalena_cert-provider_1 … done
Creating openbalena_api_1 … done
Creating openbalena_registry_1 … done
Creating openbalena_vpn_1 … done
Creating openbalena_haproxy_1 … done
…
./scripts/compose logs
cert-provider_1 | [Info] VALIDATION not set. Using default: http-01
cert-provider_1 | [Info] Waiting for api.openbalena. to be available via HTTP…
cert-provider_1 | [Info] (1/3) Connecting…
cert-provider_1 | [Info] (1/3) Failed. Retrying in 5 seconds…
cert-provider_1 | [Info] (2/3) Connecting…
cert-provider_1 | [Info] (2/3) Failed. Retrying in 5 seconds…
cert-provider_1 | [Info] (3/3) Connecting…
cert-provider_1 | [Info] (3/3) Failed!
cert-provider_1 | [Info] Unable to access api.openbalena. on port 80. This is needed for certificate validation. Retrying in 30 seconds…
cert-provider_1 | [Info] Waiting for api.openbalena. to be available via HTTP…
cert-provider_1 | [Info] (1/3) Connecting…
cert-provider_1 | [Info] (1/3) Failed. Retrying in 5 seconds…
cert-provider_1 | [Info] (2/3) Connecting…
cert-provider_1 | [Info] (2/3) Failed. Retrying in 5 seconds…
cert-provider_1 | [Info] (3/3) Connecting…
cert-provider_1 | [Info] (3/3) Failed!
cert-provider_1 | [Info] Unable to access api.openbalena. on port 80. This is needed for certificate validation. Retrying in 30 seconds…

When I do a "docker restart <container-id-if-openbalena_cert-provider>, I get:
…
Info] Installing certificates…
[Fri Jan 14 15:20:19 UTC 2022] Installing cert to: /tmp/cert.pem
[Fri Jan 14 15:20:19 UTC 2022] Installing key to: /tmp/key.pem
[Fri Jan 14 15:20:19 UTC 2022] Installing full chain to: /tmp/fullchain.pem
[Fri Jan 14 15:20:19 UTC 2022] Run reload cmd: cat /tmp/fullchain.pem /tmp/key.pem > /certs/open-balena.pem
[Fri Jan 14 15:20:19 UTC 2022] Reload success
[Info] Waiting for api.openbalena.teamo.at to use a staging certificate…
[Info] (1/3) Connecting…
[Info] (1/3) Failed. Retrying in 5 seconds…
[Info] (2/3) Connecting…
[Info] (2/3) Failed. Retrying in 5 seconds…
[Info] (3/3) Connecting…
[Info] (3/3) Failed!
[Error] Unable to detect certificate change over. Cannot issue a production certificate. [Stopping]
[Info] VALIDATION not set. Using default: http-01
[Info] Waiting for api.openbalena.teamo.at to be available via HTTP…
[Info] (1/3) Connecting…
[Info] (1/3) Success!
[Info] Last acquired certificate for STAGING
[Info] Using STAGING mode
[Info] Waiting for api.openbalena.teamo.at to be available via HTTP…
[Info] (1/3) Connecting…
[Info] (1/3) Success!
[Info] Issuing certificates…
[Fri Jan 14 15:24:55 UTC 2022] Domains not changed.
[Fri Jan 14 15:24:55 UTC 2022] Skip, Next renewal time is: Tue Mar 15 15:20:19 UTC 2022
[Fri Jan 14 15:24:55 UTC 2022] Add ‘–force’ to force to renew.
[Info] Installing certificates…
[Fri Jan 14 15:24:56 UTC 2022] Installing cert to: /tmp/cert.pem
[Fri Jan 14 15:24:56 UTC 2022] Installing key to: /tmp/key.pem
[Fri Jan 14 15:24:56 UTC 2022] Installing full chain to: /tmp/fullchain.pem
[Fri Jan 14 15:24:56 UTC 2022] Run reload cmd: cat /tmp/fullchain.pem /tmp/key.pem > /certs/open-balena.pem
[Fri Jan 14 15:24:56 UTC 2022] Reload success
[Info] Waiting for api.openbalena.teamo.at to use a staging certificate…
[Info] (1/3) Connecting…
[Info] (1/3) Failed. Retrying in 5 seconds…
[Info] (2/3) Connecting…
[Info] (2/3) Failed. Retrying in 5 seconds…
[Info] (3/3) Connecting…
[Info] (3/3) Failed!
[Error] Unable to detect certificate change over. Cannot issue a production certificate. [Stopping]

PR: RFC: cert-provider: skip staging and issue a production certificate directly by bernhardkaindl · Pull Request #134 · balena-io/open-balena · GitHub

[Mawiguk0]

Sad to see that such problems stay as long as months in the codebase…

Hope this will get better in the future

[Mawiguk0]

For now, I stopped evaluating openBalena because of missing features and lack of time. I will freeze this account and will continue to participate with @Markus.Kohn.GWA which is my account when it comes to the company I work for.

[ab77]

@mawiguk0 we have a draft PR open against openBalena, which aims to address a lot of existing concerns. It is blocked on internal tech-debt, but hopefully, once that is cleared, we’ll have openBalena published to balenaHub with a much easier interface.

[Markus.Kohn.GWA]

Thanks for the update @ab77,

I will close the Topic as soon as the PR is closed.

Regards,
Markus