Hi everyone,
I went through all the steps in this thread, but nothing worked for me.
My cert-provider always stops with the following logs:
cert-provider_1 | [Info] Waiting for api.{domain} to use a staging certificate...
cert-provider_1 | [Info] (1/3) Connecting...
cert-provider_1 | [Info] (1/3) Failed. Retrying in 5 seconds...
cert-provider_1 | [Info] (2/3) Connecting...
cert-provider_1 | [Info] (2/3) Failed. Retrying in 5 seconds...
cert-provider_1 | [Info] (3/3) Connecting...
cert-provider_1 | [Info] (3/3) Failed!
cert-provider_1 | [Error] Unable to detect certificate change over. Cannot issue a production certificate. [Stopping]
My haproxy container shows the following logs:
haproxy_1 | Building certificate from environment variables...
haproxy_1 | Setting up watches. Beware: since -r was given, this may take a while!
haproxy_1 | Watches established.
haproxy_1 | [NOTICE] 131/104238 (16) : New worker #1 (18) forked
haproxy_1 | [WARNING] 131/104238 (18) : Server backend_api/balena_api_1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
haproxy_1 | [ALERT] 131/104238 (18) : backend 'backend_api' has no server available!
haproxy_1 | [WARNING] 131/104239 (18) : Server vpn-tunnel/balena_vpn is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
haproxy_1 | [ALERT] 131/104239 (18) : proxy 'vpn-tunnel' has no server available!
haproxy_1 | [WARNING] 131/104239 (18) : Server vpn-tunnel-tls/balena_vpn is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
haproxy_1 | [ALERT] 131/104239 (18) : proxy 'vpn-tunnel-tls' has no server available!
haproxy_1 | Building certificate from environment variables...
haproxy_1 | Setting up watches. Beware: since -r was given, this may take a while!
haproxy_1 | Watches established.
haproxy_1 | [NOTICE] 131/104528 (17) : New worker #1 (19) forked
haproxy_1 | [WARNING] 131/104528 (19) : Server backend_api/balena_api_1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
haproxy_1 | [ALERT] 131/104528 (19) : backend 'backend_api' has no server available!
haproxy_1 | [WARNING] 131/104530 (19) : Server vpn-tunnel/balena_vpn is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
haproxy_1 | [ALERT] 131/104530 (19) : proxy 'vpn-tunnel' has no server available!
haproxy_1 | [WARNING] 131/104530 (19) : Server vpn-tunnel-tls/balena_vpn is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
haproxy_1 | [ALERT] 131/104530 (19) : proxy 'vpn-tunnel-tls' has no server available!
haproxy_1 | Building certificate from environment variables...
haproxy_1 | Setting up watches. Beware: since -r was given, this may take a while!
haproxy_1 | Watches established.
haproxy_1 | [NOTICE] 131/104743 (17) : New worker #1 (19) forked
haproxy_1 | [WARNING] 131/104743 (19) : Server backend_api/balena_api_1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
haproxy_1 | [ALERT] 131/104743 (19) : backend 'backend_api' has no server available!
haproxy_1 | [WARNING] 131/104745 (19) : Server backend_s3/balena_s3_1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 606ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
haproxy_1 | [ALERT] 131/104745 (19) : backend 'backend_s3' has no server available!
haproxy_1 | [WARNING] 131/104745 (19) : Server vpn-tunnel/balena_vpn is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
haproxy_1 | [ALERT] 131/104745 (19) : proxy 'vpn-tunnel' has no server available!
haproxy_1 | [WARNING] 131/104745 (19) : Server vpn-tunnel-tls/balena_vpn is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 2ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
haproxy_1 | [ALERT] 131/104745 (19) : proxy 'vpn-tunnel-tls' has no server available!
haproxy_1 | [WARNING] 131/104749 (19) : Server backend_s3/balena_s3_1 is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
haproxy_1 | [WARNING] 131/104809 (19) : Server backend_api/balena_api_1 is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
haproxy_1 | [WARNING] 131/104823 (19) : Server vpn-tunnel/balena_vpn is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
haproxy_1 | [WARNING] 131/104823 (19) : Server vpn-tunnel-tls/balena_vpn is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
haproxy_1 | /certs/ CREATE open-balena.pem
haproxy_1 | Updating certificate from cert-provider...
haproxy_1 | Certificate change detected. Reloading...
haproxy_1 | [WARNING] 131/104859 (17) : Reexecuting Master process
haproxy_1 | [WARNING] 131/104859 (19) : Stopping frontend http-in in 0 ms.
haproxy_1 | [WARNING] 131/104859 (19) : Stopping frontend ssl-in in 0 ms.
haproxy_1 | [WARNING] 131/104859 (19) : Stopping backend redirect-to-https-in in 0 ms.
haproxy_1 | [WARNING] 131/104859 (19) : Stopping backend redirect-to-tunnel-in in 0 ms.
haproxy_1 | [NOTICE] 131/104859 (17) : New worker #1 (22) forked
haproxy_1 | [WARNING] 131/104859 (19) : Stopping frontend https-in in 0 ms.
haproxy_1 | [WARNING] 131/104859 (19) : Stopping backend backend_api in 0 ms.
haproxy_1 | [WARNING] 131/104859 (19) : Stopping backend backend_registry in 0 ms.
haproxy_1 | [WARNING] 131/104859 (19) : Stopping backend backend_vpn in 0 ms.
haproxy_1 | [WARNING] 131/104859 (19) : Stopping backend backend_s3 in 0 ms.
haproxy_1 | [WARNING] 131/104859 (19) : Stopping backend cert-provider in 0 ms.
haproxy_1 | [WARNING] 131/104859 (19) : Stopping backend vpn-devices in 0 ms.
haproxy_1 | [WARNING] 131/104859 (19) : Stopping frontend db in 0 ms.
haproxy_1 | [WARNING] 131/104859 (19) : Stopping backend backend_db in 0 ms.
haproxy_1 | [WARNING] 131/104859 (19) : Stopping frontend redis in 0 ms.
haproxy_1 | [WARNING] 131/104859 (19) : Stopping backend backend_redis in 0 ms.
haproxy_1 | [WARNING] 131/104859 (19) : Stopping proxy vpn-tunnel in 0 ms.
haproxy_1 | [WARNING] 131/104859 (19) : Stopping proxy vpn-tunnel-tls in 0 ms.
haproxy_1 | [WARNING] 131/104859 (19) : Stopping frontend GLOBAL in 0 ms.
haproxy_1 | [WARNING] 131/104859 (19) : Proxy http-in stopped (FE: 22 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/104859 (19) : Proxy ssl-in stopped (FE: 10 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/104859 (19) : Proxy redirect-to-https-in stopped (FE: 0 conns, BE: 10 conns).
haproxy_1 | [WARNING] 131/104859 (19) : Proxy redirect-to-tunnel-in stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/104859 (19) : Proxy https-in stopped (FE: 10 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/104859 (19) : Proxy backend_api stopped (FE: 0 conns, BE: 12 conns).
haproxy_1 | [WARNING] 131/104859 (19) : Proxy backend_registry stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/104859 (19) : Proxy backend_vpn stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/104859 (19) : Proxy backend_s3 stopped (FE: 0 conns, BE: 1 conns).
haproxy_1 | [WARNING] 131/104859 (19) : Proxy cert-provider stopped (FE: 0 conns, BE: 18 conns).
haproxy_1 | [WARNING] 131/104859 (19) : Proxy vpn-devices stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/104859 (19) : Proxy db stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/104859 (19) : Proxy backend_db stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/104859 (19) : Proxy redis stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/104859 (19) : Proxy backend_redis stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/104859 (19) : Proxy vpn-tunnel stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/104859 (19) : Proxy vpn-tunnel-tls stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/104859 (19) : Proxy GLOBAL stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/104859 (17) : Former worker #1 (19) exited with code 0 (Exit)
haproxy_1 | Setting up watches. Beware: since -r was given, this may take a while!
haproxy_1 | Watches established.
haproxy_1 | /certs/ MODIFY open-balena.pem
haproxy_1 | Updating certificate from cert-provider...
haproxy_1 | Certificate change detected. Reloading...
haproxy_1 | [WARNING] 131/110109 (17) : Reexecuting Master process
haproxy_1 | [WARNING] 131/110109 (22) : Stopping frontend http-in in 0 ms.
haproxy_1 | [WARNING] 131/110109 (22) : Stopping frontend ssl-in in 0 ms.
haproxy_1 | [WARNING] 131/110109 (22) : Stopping backend redirect-to-https-in in 0 ms.
haproxy_1 | [WARNING] 131/110109 (22) : Stopping backend redirect-to-tunnel-in in 0 ms.
haproxy_1 | [WARNING] 131/110109 (22) : Stopping frontend https-in in 0 ms.
haproxy_1 | [WARNING] 131/110109 (22) : Stopping backend backend_api in 0 ms.
haproxy_1 | [WARNING] 131/110109 (22) : Stopping backend backend_registry in 0 ms.
haproxy_1 | [WARNING] 131/110109 (22) : Stopping backend backend_vpn in 0 ms.
haproxy_1 | [WARNING] 131/110109 (22) : Stopping backend backend_s3 in 0 ms.
haproxy_1 | [WARNING] 131/110109 (22) : Stopping backend cert-provider in 0 ms.
haproxy_1 | [NOTICE] 131/110109 (17) : New worker #1 (26) forked
haproxy_1 | [WARNING] 131/110109 (22) : Stopping backend vpn-devices in 0 ms.
haproxy_1 | [WARNING] 131/110109 (22) : Stopping frontend db in 0 ms.
haproxy_1 | [WARNING] 131/110109 (22) : Stopping backend backend_db in 0 ms.
haproxy_1 | [WARNING] 131/110109 (22) : Stopping frontend redis in 0 ms.
haproxy_1 | [WARNING] 131/110109 (22) : Stopping backend backend_redis in 0 ms.
haproxy_1 | [WARNING] 131/110109 (22) : Stopping proxy vpn-tunnel in 0 ms.
haproxy_1 | [WARNING] 131/110109 (22) : Stopping proxy vpn-tunnel-tls in 0 ms.
haproxy_1 | [WARNING] 131/110109 (22) : Stopping frontend GLOBAL in 0 ms.
haproxy_1 | [WARNING] 131/110109 (22) : Proxy http-in stopped (FE: 3 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/110109 (22) : Proxy ssl-in stopped (FE: 77 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/110109 (22) : Proxy redirect-to-https-in stopped (FE: 0 conns, BE: 77 conns).
haproxy_1 | [WARNING] 131/110109 (22) : Proxy redirect-to-tunnel-in stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/110109 (22) : Proxy https-in stopped (FE: 77 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/110109 (22) : Proxy backend_api stopped (FE: 0 conns, BE: 4 conns).
haproxy_1 | [WARNING] 131/110109 (22) : Proxy backend_registry stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/110109 (22) : Proxy backend_vpn stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/110109 (22) : Proxy backend_s3 stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/110109 (22) : Proxy cert-provider stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/110109 (22) : Proxy vpn-devices stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/110109 (22) : Proxy db stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/110109 (22) : Proxy backend_db stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/110109 (22) : Proxy redis stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/110109 (22) : Proxy backend_redis stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/110109 (22) : Proxy vpn-tunnel stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/110109 (22) : Proxy vpn-tunnel-tls stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/110109 (22) : Proxy GLOBAL stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/110109 (17) : Former worker #1 (22) exited with code 0 (Exit)
haproxy_1 | Setting up watches. Beware: since -r was given, this may take a while!
haproxy_1 | Watches established.
The part where it says
haproxy_1 | /certs/ MODIFY open-balena.pem
haproxy_1 | Updating certificate from cert-provider...
haproxy_1 | Certificate change detected. Reloading...
tells me, that the haproxy recognizes the certificate changes. But I don’t know how to debug further.
When I run the command
openssl s_client -host "api.{domain}" -port 443 -showcerts | awk '/BEGIN CERT/ p=1} ; p==1; /END CERT/ {p=0}' | openssl verify -CAfile "/usr/src/app/fake-le-bundle.pem"
inside the cert-provider container (it is the command, that checks the staging cert in the cert-provider.sh file without the 2>/dev/null parts) I get the following output:
depth=2 C = US, O = (STAGING) Internet Security Research Group, CN = (STAGING) Pretend Pear X1
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=1 C = US, O = (STAGING) Let's Encrypt, CN = (STAGING) Artificial Apricot R3
verify return:1
depth=0 CN = api.{domain}
verify return:1
CN = api.{domain}
error 20 at 0 depth lookup: unable to get local issuer certificate
error stdin: verification failed
Maybe I should mention, that the open balena instatnce is running in a VM and because of the port forwarding I added every balena url (like api.{domain}, …) to the hosts file.
Can anyone help me with that?
Thanks! 