api.domainname.com not generating let's encrypt certificate

Hi all,

I have just started a new instance of OpenBalena, using the ./scripts/quickstart -c … command with a custom domain.

Everything seems to have gone through ok, no errors, although when I access https://api.mydomain.com my browser tells me it can’t establish a secure connection. More specifically:

Cannot communicate securely with peer: no common encryption algorithm(s).

Error code: SSL_ERROR_NO_CYPHER_OVERLAP

When I visit vpn.mydomain.com; s3.mydomain.com; registry.mydomain.com it accesses the pages no problem, and I get a nice shiny padlock sign that shows the Let’s encrypt certificates behind it.

I have tried compose up and compose stop from the scripts folder, rebuilt from scratch removing images and volumes before doing the quickstart script again, and tried removing the api folder from the /config/certs/ folder and redoing the quickstart, followed by a compose stop and compose up but still no change.

The api folder in the config/certs folder has a .crt, .kid and .pem file in it. Significantly less in this folder than the vpn and root folder in config/certs but nevertheless appears to have these keys.

When I try to login to OpenBalena from my local machine via the CLI, I get the following error message:

EPROTO: request to https://api.mydomain.com/login_ failed, reason: write EPROTO: request to https://api.mydomain.com/login_ failed, reason: write EPROTO 4524355008:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1536:SSL alert number 40

I have just built this server, so it is the most up to date OpenBalena and containers.

On the first compose up it runs through a whole bunch of tasks, and doesn’t appear to have any error messages. I see a generated key, I see it trying to verify the api domain and shows success, and haproxy detects certificate changes and restarts.

Any ideas?

Hey there @Maggie this is a strange one as the certificate is common between the API and the registry so I would think that if the registry is working fine the API would be too.

I talked to my colleague who is considerably more knowledgable about openBalena and his initial suggestion was that perhaps a browser certificate caching issue is in play, but since you’re seeing similar messages from the CLI that’s ruled out. I wonder if there’s anything going on with your machine as a whole with regard to certificate caching? It seems additionally strange that you’re rebuilding from scratch and the same issue persists, so could you try from another device just to rule that out?

Hi Chris,

That is particularly strange then, but perhaps suggests I should stop trying rebuilds at least.

I had considered if it was some sort of caching issue too, but have tried it through several browsers, Safari, Firefox and TOR which should be the ultimate of non-caching and getting the same issue. I can’t think what could be caching at the machine level, but I had a friend try it from their system yesterday and they were seeing the same issue. They had tried a few times as well though, so I logged into a VPS that has had nothing to do with balena and tried to curl the URL. I got the following:

curl: (35) error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure

Hi, we had a look and are not sure what it might be, I’m currently setting up open-balena myself as a sanity check to make sure everything is working correctly. With this we can rule out a configuration issue of the repo.

Thanks. I am going to check a few other things this end, namely with the domain name, it’s via Cloudflare and I am wondering if it is proxied rather than DNS only, and whether this may make a difference.

Here is the OS it’s running on:

Description: Ubuntu 18.04.4 LTS
Release: 18.04
Codename: bionic

Everything else installed as per: https://www.balena.io/open/docs/getting-started

Hi, It would be good to rule out Cloudflare. If that is not the one causing issues could you manually confirm that the certificate is valid? it can be found in the shared volume of the haproxy container. Could you also share the config of the haproxy container? there might be something there that is causing trouble.

Can you share the publicly accessible domain for this? It might help to understand what is exposed etc?

Seems to be related to the CloudFlare proxy service. When I disable the proxy service and set it to ‘DNS only’ it is accessible (this isn’t exclusively for the API url. It just so happened the proxy was enabled for API and not for VPN, which was why the issue was only there. The same issue would apply to all the URLs behind the Cloudflare proxy). Before when I pinged the URL it would return cloud flares IP address, now on DNS only mode it returns the IP address of the server. Apparently the same happens the other way around.

It has also been suggested by a colleague that this shouldn’t be an issue behind the proxies, although the root domain doesn’t have a cert issued for it and may be causing issues. Not sure how much validity there is to it though.

In case it is of importance, I am using sub-sub domains. I.e.:

ob.domainname.com
api.ob.domainname.com
vpn.ob.domainname.com

Don’t see how that would make a difference, but I should have included the info.

Here are a few things I found on it, but getting beyond my realm:

Now I have disabled the CloudFlare proxy everything is up and running with valid certs, so definitely the root of the problem. Would like to have the proxies enabled for security reasons if possible, so any input on how to do so would be most helpful.

Hi @Maggie, this is the response from one of the openBalena maintainers: “It wouldn’t work with HTTPs proxies as the VPN uses port 443 and isn’t HTTPS proxyable. Straight TCP to the VPS only”. Please let us know if you have any further questions.
Thanks,
Zahari

The comments above are very accurate answers so I don’t need to mention them anymore.