Great, thanks.
Fyi, that worked as far as the logs are concerned.
I also did a full clean and rebuild (purged containers, images, networks, and volumes in docker and deleted the config dir). But my cli is still whingeing. Possibly a local cache of the cert… I’ll check
Btw, thanks for the awesome support engagement 
5 stars! 
You’re welcome.
The way that cert provider is working, is that when a new ACME cert is available the HAproxy service sees the file change and reloads its configuration/certs. That HAproxy service is the frontend for the API, proxy, registry etc - so check the logs for that and see that it reloaded
Yeah, I can confirm that it’s picking up the letsencrypt cert now But of course, in the world of software, the yak always needs a little extra shaving, right? In other words:
$ BALENARC_BALENA_URL="{mydomain}" balena login
gives
UNABLE_TO_GET_ISSUER_CERT_LOCALLY: request to https://api.{mydomain}/login_ failed, reason: unable to get local issuer certificate
FetchError: request to https://api.{mydomain}/login_ failed, reason: unable to get local issuer certificate
at ClientRequest.<anonymous> (/Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/node-fetch/index.js:133:11)
at ClientRequest.emit (events.js:182:13)
at ClientRequest.EventEmitter.emit (domain.js:442:20)
at TLSSocket.socketErrorListener (_http_client.js:391:9)
at TLSSocket.emit (events.js:182:13)
at TLSSocket.EventEmitter.emit (domain.js:442:20)
at emitErrorNT (internal/streams/destroy.js:82:8)
at emitErrorAndCloseNT (internal/streams/destroy.js:50:3)
at process._tickCallback (internal/process/next_tick.js:63:19)
From previous event:
at new Fetch (/Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/node-fetch/index.js:49:9)
at Fetch (/Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/node-fetch/index.js:37:10)
at /Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/fetch-ponyfill/fetch-node.js:15:12
at requestAsync (/Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/balena-request/build/utils.js:333:7)
From previous event:
at requestAsync (/Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/balena-request/build/utils.js:352:22)
at /Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/balena-request/build/utils.js:402:12
at /Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/balena-request/build/request.js:184:14
From previous event:
at Object.exports.send (/Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/balena-request/build/request.js:183:89)
at Object.exports.authenticate (/Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/balena-sdk/build/auth.js:146:20)
at exports.login (/Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/balena-sdk/build/auth.js:183:20)
From previous event:
at Object.authenticate (/Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/build/utils/patterns.js:31:10)
at login (/Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/build/actions/auth.js:79:25)
at /Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/build/actions/auth.js:100:14
From previous event:
at Command.action (/Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/build/actions/auth.js:97:45)
at /Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/capitano/build/command.js:98:37
at Command.module.exports.Command.applyPermissions (/Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/capitano/build/command.js:43:14)
at /Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/capitano/build/command.js:92:24
at Command.module.exports.Command._checkElevation (/Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/capitano/build/command.js:63:14)
at /Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/capitano/build/command.js:76:22
at /Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/capitano/build/signature.js:177:14
at /Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/capitano/node_modules/async/lib/async.js:52:16
at /Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/capitano/node_modules/async/lib/async.js:269:32
at /Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/capitano/node_modules/async/lib/async.js:44:16
at /Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/capitano/build/signature.js:171:16
at /Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/capitano/node_modules/async/lib/async.js:181:20
at iterate (/Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/capitano/node_modules/async/lib/async.js:262:13)
at Object.async.forEachOfSeries.async.eachOfSeries (/Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/capitano/node_modules/async/lib/async.js:281:9)
at Object.async.forEachSeries.async.eachSeries (/Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/capitano/node_modules/async/lib/async.js:214:22)
at Signature.module.exports.Signature.compileParameters (/Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/capitano/build/signature.js:125:18)
at Command.module.exports.Command.execute (/Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/capitano/build/command.js:71:27)
at /Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/capitano/build/capitano.js:64:22
at /Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/capitano/build/state.js:31:14
at /Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/capitano/build/signature.js:105:16
at /Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/capitano/build/signature.js:177:14
at /Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/capitano/node_modules/async/lib/async.js:52:16
at /Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/capitano/node_modules/async/lib/async.js:269:32
at /Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/capitano/node_modules/async/lib/async.js:44:16
at /Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/capitano/build/signature.js:171:16
at /Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/capitano/node_modules/async/lib/async.js:181:20
at iterate (/Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/capitano/node_modules/async/lib/async.js:262:13)
at Object.async.forEachOfSeries.async.eachOfSeries (/Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/capitano/node_modules/async/lib/async.js:281:9)
at Object.async.forEachSeries.async.eachSeries (/Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/capitano/node_modules/async/lib/async.js:214:22)
at Signature.module.exports.Signature.compileParameters (/Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/capitano/build/signature.js:125:18)
at Signature.module.exports.Signature.matches (/Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/capitano/build/signature.js:103:17)
at /Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/capitano/build/state.js:27:30
at /Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/capitano/node_modules/async/lib/async.js:181:20
at Immediate.iterate (/Users/al/.nvm/versions/node/v10.9.0/lib/node_modules/balena-cli/node_modules/capitano/node_modules/async/lib/async.js:262:13)
at runCallback (timers.js:693:18)
at tryOnImmediate (timers.js:664:5)
at processImmediate (timers.js:646:5)
at process.topLevelDomainCallback (domain.js:121:23)
Ah! So this is nodejs complaining in the CLI tool - remove the environment variable you exported, NODE_EXTRA_CA_CERTS from memory…
Doh! I forgot about that 
… 5 minutes later. Um, no difference. I both unset the variabled, removed it from my bashrc, and opened a new shell. I don’t think it can be picking that up anymore unless there’s a stored reference to it somewhere that I don’t know about.
Not sure then; I would open the url https://api.{your domain}/ping and see what certificate is served - it should be your ACME one
Ok, so I tried from both my local machine and another machine up in the cloud. I got different errors, annoyingly.
Local:
curl -vv https://api.{mydomain}
* Trying 18.200.67.184...
* TCP_NODELAY set
* Connected to api.{mydomain} (18.200.67.184) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: unable to get local issuer certificate
* stopped the pause stream!
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
HTTPS-proxy has similar options --proxy-cacert and --proxy-insecure.
and
$ echo | openssl s_client -showcerts -servername registry.{mydomain} -connect registry.{mydomain}:443 2>/dev/null | openssl x509 -inform pem -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
fa:ca:ac:e8:88:62:46:6f:01:95:6d:e0:13:df:4e:94:6b:23
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=Fake LE Intermediate X1
Validity
Not Before: Jan 15 13:44:29 2020 GMT
Not After : Apr 14 13:44:29 2020 GMT
Subject: CN=api.{mydomain}
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:bf:f5:67:17:f5:a2:01:79:68:89:cb:9e:fe:38:
1e:42:68:25:a0:ff:1d:16:f5:ef:8c:d2:ce:60:48:
40:d9:54:d6:3c:ad:c3:f2:de:8e:b4:f9:1f:48:ff:
de:21:1d:13:d5:3d:e6:ed:84:1a:16:d7:50:0b:39:
ca:58:b0:84:6a:5c:bb:95:83:a9:bd:7b:72:b8:3b:
29:cc:39:c0:e0:bf:2f:9f:3c:d2:bc:13:51:a1:ac:
87:b4:6c:3f:97:a6:dd:77:26:91:ef:4d:91:33:0e:
50:d6:02:65:c5:0c:46:34:0e:1c:bc:39:b4:db:9c:
9c:6f:38:86:0d:0a:6a:36:db:25:b6:a6:b1:47:00:
ec:ac:de:cb:8f:0f:19:cd:ef:40:12:a0:0e:53:d5:
78:2a:da:12:20:e1:f9:c2:6c:06:31:51:7f:3e:c9:
cf:45:a3:48:f4:b0:5e:5a:d5:6f:62:8c:79:5a:29:
05:b6:34:48:1b:e8:e1:73:57:0b:59:86:80:20:61:
95:61:a6:e8:69:8b:51:d4:8e:bc:07:27:58:af:62:
b2:a0:a5:3c:be:41:ee:b6:57:fd:0a:4d:77:8d:18:
dd:98:a0:9d:f3:ee:cf:f0:c7:4a:95:0d:b6:3e:ad:
11:35:48:ca:c8:e2:27:34:dd:38:ba:25:1a:d9:67:
19:ab
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Key Identifier:
A0:B0:BD:3F:AC:2A:04:69:49:AA:0C:69:37:C4:18:82:18:AD:65:67
X509v3 Authority Key Identifier:
keyid:C0:CC:03:46:B9:58:20:CC:5C:72:70:F3:E1:2E:CB:20:A6:F5:68:3A
Authority Information Access:
OCSP - URI:http://ocsp.stg-int-x1.letsencrypt.org
CA Issuers - URI:http://cert.stg-int-x1.letsencrypt.org/
X509v3 Subject Alternative Name:
DNS:api.{mydomain}, DNS:registry.{mydomain}, DNS:s3.{mydomain}, DNS:vpn.{mydomain}
X509v3 Certificate Policies:
Policy: 2.23.140.1.2.1
Policy: 1.3.6.1.4.1.44947.1.1.1
CPS: http://cps.letsencrypt.org
1.3.6.1.4.1.11129.2.4.2:
......v...4...$..Vh}.4..I.I..i....?\..nd...o...q.....G0E. ...pP.l[S................oX..
.....f.c{.T...;.B.1..."..a^5o.6.S.+.p.v...i...........v......1.R....1......o.........G0E.!......j...b...96.............<..a. e\c...E....aw+.W
Signature Algorithm: sha256WithRSAEncryption
ce:7f:8c:e8:4b:59:98:11:fb:e9:3b:b9:6e:a8:a3:1e:13:84:
99:a5:1d:f1:86:f2:45:43:a1:b3:76:fe:83:67:2d:f7:70:59:
69:51:eb:31:82:37:b1:ed:ff:4e:38:51:4d:41:64:3c:6a:f7:
20:c6:10:11:0a:3c:75:b2:d5:96:af:f4:4b:b0:ad:cf:56:3f:
0c:66:eb:2f:03:8c:30:fd:95:26:2a:42:36:4b:98:34:d3:99:
b5:b3:c7:8b:a9:a6:ec:6c:bb:4b:49:eb:11:70:a3:db:61:34:
4f:87:32:d5:c2:e4:59:df:2f:74:32:76:c7:27:5e:f9:e0:cc:
82:5f:0f:22:9f:8e:36:d4:66:b2:2d:82:86:75:84:a6:d3:1a:
29:d2:98:08:bd:14:a7:61:0e:fc:16:ac:ac:49:79:43:30:ac:
92:a7:02:85:da:36:f2:2c:a5:f3:77:13:7e:78:a2:1f:67:37:
df:45:f2:5b:cb:8b:1c:38:9c:1e:70:57:8e:99:e8:76:56:3c:
2e:d2:84:ca:18:44:f7:ec:6d:57:d1:c5:2a:40:a6:ea:f4:27:
02:23:59:40:5e:9b:4d:41:91:aa:01:42:73:2e:97:ea:64:2d:
b9:44:4a:9a:be:28:8b:53:42:86:25:32:a7:9d:b9:d9:b4:73:
41:0e:33:b2
But on a remote machine (one that never had NODE_EXTRA_CA_CERTS or the old self-signed certificate installed):
$ curl -vv https://api.{mydomain}/ping
* Trying 18.200.67.184...
* Connected to api.{mydomain} (18.200.67.184) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 592 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
* server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
* Closing connection 0
curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
and
$ echo | openssl s_client -showcerts -servername registry.{mydomain} -connect registry.{mydomain}:443 2>/dev/null | openssl x509 -inform pem -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
fa:ca:ac:e8:88:62:46:6f:01:95:6d:e0:13:df:4e:94:6b:23
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=Fake LE Intermediate X1
Validity
Not Before: Jan 15 13:44:29 2020 GMT
Not After : Apr 14 13:44:29 2020 GMT
Subject: CN=api.{mydomain}
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:bf:f5:67:17:f5:a2:01:79:68:89:cb:9e:fe:38:
1e:42:68:25:a0:ff:1d:16:f5:ef:8c:d2:ce:60:48:
40:d9:54:d6:3c:ad:c3:f2:de:8e:b4:f9:1f:48:ff:
de:21:1d:13:d5:3d:e6:ed:84:1a:16:d7:50:0b:39:
ca:58:b0:84:6a:5c:bb:95:83:a9:bd:7b:72:b8:3b:
29:cc:39:c0:e0:bf:2f:9f:3c:d2:bc:13:51:a1:ac:
87:b4:6c:3f:97:a6:dd:77:26:91:ef:4d:91:33:0e:
50:d6:02:65:c5:0c:46:34:0e:1c:bc:39:b4:db:9c:
9c:6f:38:86:0d:0a:6a:36:db:25:b6:a6:b1:47:00:
ec:ac:de:cb:8f:0f:19:cd:ef:40:12:a0:0e:53:d5:
78:2a:da:12:20:e1:f9:c2:6c:06:31:51:7f:3e:c9:
cf:45:a3:48:f4:b0:5e:5a:d5:6f:62:8c:79:5a:29:
05:b6:34:48:1b:e8:e1:73:57:0b:59:86:80:20:61:
95:61:a6:e8:69:8b:51:d4:8e:bc:07:27:58:af:62:
b2:a0:a5:3c:be:41:ee:b6:57:fd:0a:4d:77:8d:18:
dd:98:a0:9d:f3:ee:cf:f0:c7:4a:95:0d:b6:3e:ad:
11:35:48:ca:c8:e2:27:34:dd:38:ba:25:1a:d9:67:
19:ab
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Key Identifier:
A0:B0:BD:3F:AC:2A:04:69:49:AA:0C:69:37:C4:18:82:18:AD:65:67
X509v3 Authority Key Identifier:
keyid:C0:CC:03:46:B9:58:20:CC:5C:72:70:F3:E1:2E:CB:20:A6:F5:68:3A
Authority Information Access:
OCSP - URI:http://ocsp.stg-int-x1.letsencrypt.org
CA Issuers - URI:http://cert.stg-int-x1.letsencrypt.org/
X509v3 Subject Alternative Name:
DNS:api.{mydomain}, DNS:registry.{mydomain}, DNS:s3.{mydomain}, DNS:vpn.{mydomain}
X509v3 Certificate Policies:
Policy: 2.23.140.1.2.1
Policy: 1.3.6.1.4.1.44947.1.1.1
CPS: http://cps.letsencrypt.org
CT Precertificate SCTs:
Signed Certificate Timestamp:
Version : v1(0)
Log ID : DD:99:34:FC:A5:E7:24:80:C9:56:68:7D:81:34:99:08:
49:B2:49:F7:B5:69:D8:C7:BC:AB:3F:5C:C1:F3:6E:64
Timestamp : Jan 15 14:44:29.937 2020 GMT
Extensions: none
Signature : ecdsa-with-SHA256
30:45:02:20:12:C4:EC:70:50:90:6C:5B:53:F0:D4:0B:
1D:8D:D6:13:C0:86:97:D9:EB:02:8E:D9:90:6F:58:96:
84:0A:47:1F:02:21:00:FE:44:BD:FC:75:36:E3:5B:A2:
02:EF:42:EC:31:06:97:81:22:88:A1:61:5E:35:6F:08:
36:92:53:97:2B:B4:70
Signed Certificate Timestamp:
Version : v1(0)
Log ID : 16:E8:69:C1:D1:95:EA:D7:C3:F8:97:1A:E3:F0:76:01:
F7:8C:E1:B6:9D:31:A8:52:18:B6:83:7F:31:A8:15:08
Timestamp : Jan 15 14:44:29.971 2020 GMT
Extensions: none
Signature : ecdsa-with-SHA256
30:45:02:21:00:F3:04:A5:9F:90:6A:8D:C7:F2:62:F0:
8C:F3:39:36:1F:C7:89:F1:FD:A7:E4:C4:0F:CA:E0:A9:
0B:3C:98:DC:61:02:20:65:5C:63:F4:93:F6:45:C9:AF:
EF:EB:61:77:2B:B6:57:0D:0E:B6:0C:1E:E9:66:02:63:
7B:D0:54:90:0C:CB:3B
Signature Algorithm: sha256WithRSAEncryption
ce:7f:8c:e8:4b:59:98:11:fb:e9:3b:b9:6e:a8:a3:1e:13:84:
99:a5:1d:f1:86:f2:45:43:a1:b3:76:fe:83:67:2d:f7:70:59:
69:51:eb:31:82:37:b1:ed:ff:4e:38:51:4d:41:64:3c:6a:f7:
20:c6:10:11:0a:3c:75:b2:d5:96:af:f4:4b:b0:ad:cf:56:3f:
0c:66:eb:2f:03:8c:30:fd:95:26:2a:42:36:4b:98:34:d3:99:
b5:b3:c7:8b:a9:a6:ec:6c:bb:4b:49:eb:11:70:a3:db:61:34:
4f:87:32:d5:c2:e4:59:df:2f:74:32:76:c7:27:5e:f9:e0:cc:
82:5f:0f:22:9f:8e:36:d4:66:b2:2d:82:86:75:84:a6:d3:1a:
29:d2:98:08:bd:14:a7:61:0e:fc:16:ac:ac:49:79:43:30:ac:
92:a7:02:85:da:36:f2:2c:a5:f3:77:13:7e:78:a2:1f:67:37:
df:45:f2:5b:cb:8b:1c:38:9c:1e:70:57:8e:99:e8:76:56:3c:
2e:d2:84:ca:18:44:f7:ec:6d:57:d1:c5:2a:40:a6:ea:f4:27:
02:23:59:40:5e:9b:4d:41:91:aa:01:42:73:2e:97:ea:64:2d:
b9:44:4a:9a:be:28:8b:53:42:86:25:32:a7:9d:b9:d9:b4:73:
41:0e:33:b2
Which to my reading means that both are getting the letsencrypt certificate.
In the remote case, I found https://community.letsencrypt.org/t/certificate-verification-errors-when-using-curl/105335, which may be relevant or not.
I patched the cert provider container to get this far. I’m also wondering if there are any other patches required for the haproxy or api containers, for example?
Thanks again for you help. I must be completely trashing your roadmap for today with all my questions 
I can see the following: Fake LE Intermediate X1 – this is the staging certificate from LetEncrypt.
The cert provider service first acquires a staging cert, then, and ONLY then, will it try to get a production one. Check the cert provider logs again - I suspect something is preventing LE issuing you a production cert. It might be that you’ve run into their rate limits…
Looks like it created prod certs ok:
[Info] VALIDATION not set. Using default: http-01
[Info] Waiting for api.{mydomain} to be available via HTTP...
[Info] (1/3) Connecting...
[Info] (1/3) Success!
cat: can't open '/usr/src/app/certs/last_run_mode': No such file or directory
[Info] Last acquired certificate for
[Info] Using STAGING mode
[Info] Waiting for api.{mydomain} to be available via HTTP...
[Info] (1/3) Connecting...
[Info] (1/3) Success!
[Info] Issuing certificates...
[Wed Jan 15 14:44:06 UTC 2020] Using stage ACME_DIRECTORY: https://acme-staging-v02.api.letsencrypt.org/directory
[Wed Jan 15 14:44:06 UTC 2020] Standalone mode.
[Wed Jan 15 14:44:06 UTC 2020] Standalone mode.
[Wed Jan 15 14:44:06 UTC 2020] Standalone mode.
[Wed Jan 15 14:44:06 UTC 2020] Standalone mode.
[Wed Jan 15 14:44:07 UTC 2020] Create account key ok.
[Wed Jan 15 14:44:07 UTC 2020] Registering account
[Wed Jan 15 14:44:08 UTC 2020] Registered
[Wed Jan 15 14:44:08 UTC 2020] ACCOUNT_THUMBPRINT='00MQ_OXndVzQFrcJFHG0pGK40matY7ak2ezeUsgqQ5o'
[Wed Jan 15 14:44:08 UTC 2020] Creating domain key
[Wed Jan 15 14:44:08 UTC 2020] The domain key is here: /usr/src/app/certs/api.{mydomain}/api.{mydomain}.key
[Wed Jan 15 14:44:08 UTC 2020] Multi domain='DNS:api.{mydomain},DNS:registry.{mydomain},DNS:s3.{mydomain},DNS:vpn.{mydomain}'
[Wed Jan 15 14:44:08 UTC 2020] Getting domain auth token for each domain
[Wed Jan 15 14:44:11 UTC 2020] Getting webroot for domain='api.{mydomain}'
[Wed Jan 15 14:44:11 UTC 2020] Getting webroot for domain='registry.{mydomain}'
[Wed Jan 15 14:44:11 UTC 2020] Getting webroot for domain='s3.{mydomain}'
[Wed Jan 15 14:44:11 UTC 2020] Getting webroot for domain='vpn.{mydomain}'
[Wed Jan 15 14:44:12 UTC 2020] Verifying: api.{mydomain}
[Wed Jan 15 14:44:12 UTC 2020] Standalone mode server
[Wed Jan 15 14:44:16 UTC 2020] Success
[Wed Jan 15 14:44:16 UTC 2020] Verifying: registry.{mydomain}
[Wed Jan 15 14:44:16 UTC 2020] Standalone mode server
[Wed Jan 15 14:44:20 UTC 2020] Success
[Wed Jan 15 14:44:20 UTC 2020] Verifying: s3.{mydomain}
[Wed Jan 15 14:44:20 UTC 2020] Standalone mode server
[Wed Jan 15 14:44:24 UTC 2020] Success
[Wed Jan 15 14:44:24 UTC 2020] Verifying: vpn.{mydomain}
[Wed Jan 15 14:44:24 UTC 2020] Standalone mode server
[Wed Jan 15 14:44:29 UTC 2020] Success
[Wed Jan 15 14:44:29 UTC 2020] Verify finished, start to sign.
[Wed Jan 15 14:44:29 UTC 2020] Lets finalize the order, Le_OrderFinalize: https://acme-staging-v02.api.letsencrypt.org/acme/finalize/12103655/70204066
[Wed Jan 15 14:44:30 UTC 2020] Download cert, Le_LinkCert: https://acme-staging-v02.api.letsencrypt.org/acme/cert/facaace88862466f01956de013df4e946b23
[Wed Jan 15 14:44:31 UTC 2020] Cert success.
-----BEGIN CERTIFICATE-----
MIIFkTCCBHmgAwIBAgITAPrKrOiIYkZvAZVt4BPfTpRrIzANBgkqhkiG9w0BAQsF
ADAiMSAwHgYDVQQDDBdGYWtlIExFIEludGVybWVkaWF0ZSBYMTAeFw0yMDAxMTUx
MzQ0MjlaFw0yMDA0MTQxMzQ0MjlaMCExHzAdBgNVBAMTFmFwaS5iYWxlbmEuZGdj
c2Rldi5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC/9WcX9aIB
eWiJy57+OB5CaCWg/x0W9e+M0s5gSEDZVNY8rcPy3o60+R9I/94hHRPVPebthBoW
11ALOcpYsIRqXLuVg6m9e3K4OynMOcDgvy+fPNK8E1GhrIe0bD+Xpt13JpHvTZEz
DlDWAmXFDEY0Dhy8ObTbnJxvOIYNCmo22yW2prFHAOys3suPDxnN70ASoA5T1Xgq
2hIg4fnCbAYxUX8+yc9Fo0j0sF5a1W9ijHlaKQW2NEgb6OFzVwtZhoAgYZVhpuhp
i1HUjrwHJ1ivYrKgpTy+Qe62V/0KTXeNGN2YoJ3z7s/wx0qVDbY+rRE1SMrI4ic0
3Ti6JRrZZxmrAgMBAAGjggK/MIICuzAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYw
FAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFKCw
vT+sKgRpSaoMaTfEGIIYrWVnMB8GA1UdIwQYMBaAFMDMA0a5WCDMXHJw8+EuyyCm
9Wg6MHcGCCsGAQUFBwEBBGswaTAyBggrBgEFBQcwAYYmaHR0cDovL29jc3Auc3Rn
LWludC14MS5sZXRzZW5jcnlwdC5vcmcwMwYIKwYBBQUHMAKGJ2h0dHA6Ly9jZXJ0
LnN0Zy1pbnQteDEubGV0c2VuY3J5cHQub3JnLzBtBgNVHREEZjBkghZhcGkuYmFs
ZW5hLmRnY3NkZXYuY29tghtyZWdpc3RyeS5iYWxlbmEuZGdjc2Rldi5jb22CFXMz
LmJhbGVuYS5kZ2NzZGV2LmNvbYIWdnBuLmJhbGVuYS5kZ2NzZGV2LmNvbTBMBgNV
HSAERTBDMAgGBmeBDAECATA3BgsrBgEEAYLfEwEBATAoMCYGCCsGAQUFBwIBFhpo
dHRwOi8vY3BzLmxldHNlbmNyeXB0Lm9yZzCCAQQGCisGAQQB1nkCBAIEgfUEgfIA
8AB2AN2ZNPyl5ySAyVZofYE0mQhJskn3tWnYx7yrP1zB825kAAABb6mpuHEAAAQD
AEcwRQIgEsTscFCQbFtT8NQLHY3WE8CGl9nrAo7ZkG9YloQKRx8CIQD+RL38dTbj
W6IC70LsMQaXgSKIoWFeNW8INpJTlyu0cAB2ABboacHRlerXw/iXGuPwdgH3jOG2
nTGoUhi2g38xqBUIAAABb6mpuJMAAAQDAEcwRQIhAPMEpZ+Qao3H8mLwjPM5Nh/H
ifH9p+TED8rgqQs8mNxhAiBlXGP0k/ZFya/v62F3K7ZXDQ62DB7pZgJje9BUkAzL
OzANBgkqhkiG9w0BAQsFAAOCAQEAzn+M6EtZmBH76Tu5bqijHhOEmaUd8YbyRUOh
s3b+g2ct93BZaVHrMYI3se3/TjhRTUFkPGr3IMYQEQo8dbLVlq/0S7Ctz1Y/DGbr
LwOMMP2VJipCNkuYNNOZtbPHi6mm7Gy7S0nrEXCj22E0T4cy1cLkWd8vdDJ2xyde
+eDMgl8PIp+ONtRmsi2ChnWEptMaKdKYCL0Up2EO/BasrEl5QzCskqcChdo28iyl
83cTfniiH2c330XyW8uLHDicHnBXjpnodlY8LtKEyhhE9+xtV9HFKkCm6vQnAiNZ
QF6bTUGRqgFCcy6X6mQtuURKmr4oi1NChiUyp5252bRzQQ4zsg==
-----END CERTIFICATE-----
[Wed Jan 15 14:44:31 UTC 2020] Your cert is in /usr/src/app/certs/api.{mydomain}/api.{mydomain}.cer
[Wed Jan 15 14:44:31 UTC 2020] Your cert key is in /usr/src/app/certs/api.{mydomain}/api.{mydomain}.key
[Wed Jan 15 14:44:31 UTC 2020] The intermediate CA cert is in /usr/src/app/certs/api.{mydomain}/ca.cer
[Wed Jan 15 14:44:31 UTC 2020] And the full chain certs is there: /usr/src/app/certs/api.{mydomain}/fullchain.cer
[Info] Installing certificates...
[Wed Jan 15 14:44:31 UTC 2020] Installing cert to:/tmp/cert.pem
[Wed Jan 15 14:44:31 UTC 2020] Installing key to:/tmp/key.pem
[Wed Jan 15 14:44:31 UTC 2020] Installing full chain to:/tmp/fullchain.pem
[Wed Jan 15 14:44:31 UTC 2020] Run reload cmd: cat /tmp/fullchain.pem /tmp/key.pem > /certs/open-balena.pem
[Wed Jan 15 14:44:31 UTC 2020] Reload success
[Info] Waiting for api.{mydomain} to use a staging certificate...
[Info] (1/3) Connecting...
[Info] (1/3) Success!
[Info] Using PRODUCTION mode
[Info] Waiting for api.{mydomain} to be available via HTTP...
[Info] (1/3) Connecting...
[Info] (1/3) Success!
[Info] Issuing certificates...
[Wed Jan 15 14:44:32 UTC 2020] Standalone mode.
[Wed Jan 15 14:44:32 UTC 2020] Standalone mode.
[Wed Jan 15 14:44:32 UTC 2020] Standalone mode.
[Wed Jan 15 14:44:32 UTC 2020] Standalone mode.
[Wed Jan 15 14:44:32 UTC 2020] Create account key ok.
[Wed Jan 15 14:44:32 UTC 2020] Registering account
[Wed Jan 15 14:44:33 UTC 2020] Registered
[Wed Jan 15 14:44:33 UTC 2020] ACCOUNT_THUMBPRINT='H7FVTNref0zjffSY6t46VVgluR6yIiEayP6irRb6ghQ'
[Wed Jan 15 14:44:33 UTC 2020] Multi domain='DNS:api.{mydomain},DNS:registry.{mydomain},DNS:s3.{mydomain},DNS:vpn.{mydomain}'
[Wed Jan 15 14:44:33 UTC 2020] Getting domain auth token for each domain
[Wed Jan 15 14:44:36 UTC 2020] Getting webroot for domain='api.{mydomain}'
[Wed Jan 15 14:44:36 UTC 2020] Getting webroot for domain='registry.{mydomain}'
[Wed Jan 15 14:44:36 UTC 2020] Getting webroot for domain='s3.{mydomain}'
[Wed Jan 15 14:44:36 UTC 2020] Getting webroot for domain='vpn.{mydomain}'
[Wed Jan 15 14:44:36 UTC 2020] Verifying: api.{mydomain}
[Wed Jan 15 14:44:36 UTC 2020] Standalone mode server
[Wed Jan 15 14:44:41 UTC 2020] Success
[Wed Jan 15 14:44:41 UTC 2020] Verifying: registry.{mydomain}
[Wed Jan 15 14:44:41 UTC 2020] Standalone mode server
[Wed Jan 15 14:44:45 UTC 2020] Success
[Wed Jan 15 14:44:45 UTC 2020] Verifying: s3.{mydomain}
[Wed Jan 15 14:44:45 UTC 2020] Standalone mode server
[Wed Jan 15 14:44:49 UTC 2020] Success
[Wed Jan 15 14:44:49 UTC 2020] Verifying: vpn.{mydomain}
[Wed Jan 15 14:44:49 UTC 2020] Standalone mode server
[Wed Jan 15 14:44:54 UTC 2020] Success
[Wed Jan 15 14:44:54 UTC 2020] Verify finished, start to sign.
[Wed Jan 15 14:44:54 UTC 2020] Lets finalize the order, Le_OrderFinalize: https://acme-v02.api.letsencrypt.org/acme/finalize/75899581/2039203337
[Wed Jan 15 14:44:55 UTC 2020] Download cert, Le_LinkCert: https://acme-v02.api.letsencrypt.org/acme/cert/0442bf520f9639506b1e53d29eb593c58548
[Wed Jan 15 14:44:55 UTC 2020] Cert success.
-----BEGIN CERTIFICATE-----
MIIFrzCCBJegAwIBAgISBEK/Ug+WOVBrHlPSnrWTxYVIMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0yMDAxMTUxMzQ0NTRaFw0y
MDA0MTQxMzQ0NTRaMCExHzAdBgNVBAMTFmFwaS5iYWxlbmEuZGdjc2Rldi5jb20w
ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC/9WcX9aIBeWiJy57+OB5C
aCWg/x0W9e+M0s5gSEDZVNY8rcPy3o60+R9I/94hHRPVPebthBoW11ALOcpYsIRq
XLuVg6m9e3K4OynMOcDgvy+fPNK8E1GhrIe0bD+Xpt13JpHvTZEzDlDWAmXFDEY0
Dhy8ObTbnJxvOIYNCmo22yW2prFHAOys3suPDxnN70ASoA5T1Xgq2hIg4fnCbAYx
UX8+yc9Fo0j0sF5a1W9ijHlaKQW2NEgb6OFzVwtZhoAgYZVhpuhpi1HUjrwHJ1iv
YrKgpTy+Qe62V/0KTXeNGN2YoJ3z7s/wx0qVDbY+rRE1SMrI4ic03Ti6JRrZZxmr
AgMBAAGjggK2MIICsjAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUH
AwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFKCwvT+sKgRpSaoM
aTfEGIIYrWVnMB8GA1UdIwQYMBaAFKhKamMEfd265tE5t6ZFZe/zqOyhMG8GCCsG
AQUFBwEBBGMwYTAuBggrBgEFBQcwAYYiaHR0cDovL29jc3AuaW50LXgzLmxldHNl
bmNyeXB0Lm9yZzAvBggrBgEFBQcwAoYjaHR0cDovL2NlcnQuaW50LXgzLmxldHNl
bmNyeXB0Lm9yZy8wbQYDVR0RBGYwZIIWYXBpLmJhbGVuYS5kZ2NzZGV2LmNvbYIb
cmVnaXN0cnkuYmFsZW5hLmRnY3NkZXYuY29tghVzMy5iYWxlbmEuZGdjc2Rldi5j
b22CFnZwbi5iYWxlbmEuZGdjc2Rldi5jb20wTAYDVR0gBEUwQzAIBgZngQwBAgEw
NwYLKwYBBAGC3xMBAQEwKDAmBggrBgEFBQcCARYaaHR0cDovL2Nwcy5sZXRzZW5j
cnlwdC5vcmcwggEDBgorBgEEAdZ5AgQCBIH0BIHxAO8AdQCyHgXMi6LNiiBOh2b5
K7mKJSBna9r6cOeySVMt74uQXgAAAW+pqho1AAAEAwBGMEQCIC5wweFtr6KHzHI/
vC3DS0W23zs41c04pUsdDXvVfXMxAiBfItOJp1fBM8l8+e0HJogQ9glTjetAkKwK
sUrw37z1owB2AG9Tdqwx8DEZ2JkApFEV/3cVHBHZAsEAKQaNsgiaN9kTAAABb6mq
GmoAAAQDAEcwRQIgGFcbH/bV0UOD6yXilehdZ9gExLa3t4yDgYZwuJr6ZVECIQCe
xC31/9rP2N6qPOX1VSpG3H63w/5uLoo0g+N7/JWG+DANBgkqhkiG9w0BAQsFAAOC
AQEAZNG+QoffyKc6NQlNDkzY1vCGbJdmKDRAOwVKFNajQH5A4eW9Ef7hzU/Z4w3m
noBrBnJYgSAlmOXm4wCCNj3OMtbje/wgTSLT3nrnXTjrZXc2AoAykVUc3wJeKFZ7
VgBJWesQf0mFWpLCuY3SJvuVjs62cYkXAODZgIvu4p+5put3AWKDAK54Zm5qxk5f
busqhTRcK1kgVWEUCuqodQY/zg/2FEx+k1xMN4P5MQzNUNiSlQ5CjA5d0Lzj2sBM
F1dHbyp6HRocAbXkGWCl3EjW62cxmo9Es3QkvejhVzRn+5iSXTxhqrpYhvC+hkAZ
9GOSxF2nQ61w1HLtG8X7ztGnpA==
-----END CERTIFICATE-----
[Wed Jan 15 14:44:55 UTC 2020] Your cert is in /usr/src/app/certs/api.{mydomain}/api.{mydomain}.cer
[Wed Jan 15 14:44:55 UTC 2020] Your cert key is in /usr/src/app/certs/api.{mydomain}/api.{mydomain}.key
[Wed Jan 15 14:44:55 UTC 2020] The intermediate CA cert is in /usr/src/app/certs/api.{mydomain}/ca.cer
[Wed Jan 15 14:44:55 UTC 2020] And the full chain certs is there: /usr/src/app/certs/api.{mydomain}/fullchain.cer
[Info] Installing certificates...
[Wed Jan 15 14:44:56 UTC 2020] Installing cert to:/tmp/cert.pem
[Wed Jan 15 14:44:56 UTC 2020] Installing key to:/tmp/key.pem
[Wed Jan 15 14:44:56 UTC 2020] Installing full chain to:/tmp/fullchain.pem
[Wed Jan 15 14:44:56 UTC 2020] Run reload cmd: cat /tmp/fullchain.pem /tmp/key.pem > /certs/open-balena.pem
[Wed Jan 15 14:44:56 UTC 2020] Reload success
[Success] Done!
OK - that’s good - so I guess the HAproxy container didn’t pick it up - check those logs, and then if you jump into the container and touch the cert file, then it should reload again.
Arrgh. Lost my internet, and my reply.
I didn’t know how to jump into the container, as a docker exec -it openbalena_haproxy_1 /bin/bash didn’t work. So I just restarted and found the following:
$ docker logs openbalena_haproxy_1
...
Setting up watches. Beware: since -r was given, this may take a while!
Watches established.
/certs/ MODIFY open-balena.pem
Updating certificate from cert-provider...
Certificate change detected. Reloading...
[WARNING] 014/144456 (14) : Reexecuting Master process
[ALERT] 014/144456 (14) : parsing [/usr/local/etc/haproxy/haproxy.cfg:48] : 'bind 127.0.0.1:444' : unable to load SSL private key from PEM file '/etc/ssl/private/open-balena.pem'.
[ALERT] 014/144456 (14) : Error(s) found in configuration file : /usr/local/etc/haproxy/haproxy.cfg
[ALERT] 014/144456 (14) : Fatal errors found in configuration.
[WARNING] 014/144456 (14) : Reexecuting Master process in waitpid mode
[WARNING] 014/144456 (14) : Reexecuting Master process
Setting up watches. Beware: since -r was given, this may take a while!
Watches established.
Using certificate from cert-provider...
Setting up watches. Beware: since -r was given, this may take a while!
Watches established.
[NOTICE] 014/161559 (8) : New worker #1 (10) forked
So I checked out the haproxy.cfg and the whole section including line 48 reads:
frontend https-in
mode http
option forwardfor
bind 127.0.0.1:444 ssl crt /etc/ssl/private/open-balena.pem accept-proxy
reqadd X-Forwarded-Proto:\ https
acl host_api hdr_dom(host) -i "api.${HAPROXY_HOSTNAME}"
use_backend backend_api if host_api
acl host_registry hdr_dom(host) -i "registry.${HAPROXY_HOSTNAME}"
use_backend backend_registry if host_registry
acl host_vpn hdr_dom(host) -i "vpn.${HAPROXY_HOSTNAME}"
use_backend backend_vpn if host_vpn
acl host_s3 hdr_dom(host) -i "s3.${HAPROXY_HOSTNAME}"
use_backend backend_s3 if host_s3
I’m in unfamiliar territory with haproxy; more of an nginx chap meself 
I’ll keep poking around.
[edit] I do notice that the $OUTPUT_PEM variable defined in the compose dir is /certs/open-balena.pem, and the volume corresponding to certs is loaded into haproxy also as /certs. So I’m wondering if haproxy should really be looking in /certs not /etc/ssl/certs? I’m trying to make that wire up now
[edit 2] So, I looked at start_haproxy.pem and realised you have code in there already to move the pem.
How did you restart the container, and how do you generally interact with the services? You must always go through ./scripts/compose.
Hey @richbayliss. Apologies for bugging you again, but I’m not making headway understanding what’s going on with these certs.
(I’ve got access to the containers now, with good old sh)
Running cert-provider.sh in the cert-provider container claims that the last production keys were production-mode.
The contents of open-balena.pem (in both container’s /cert/… and haproxy’s /etc/ssl/private/…) is as follows: (I’ve obfuscated the private key)
-----BEGIN CERTIFICATE-----
MIIFkDCCBHigAwIBAgITAPqdwc75g7ISvdiTj30k8MF7zzANBgkqhkiG9w0BAQsF
ADAiMSAwHgYDVQQDDBdGYWtlIExFIEludGVybWVkaWF0ZSBYMTAeFw0yMDAxMTUx
NzA3MTFaFw0yMDA0MTQxNzA3MTFaMCExHzAdBgNVBAMTFmFwaS5iYWxlbmEuZGdj
c2Rldi5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCzGIfDAMBA
Yuo8Xhdtw5oUzRntBZp30ifNSOP0xGZZ9BPrdw2U2lvC7b1+SuqhR2vhqK47uYhz
3RI7CFgGE0ryNp4WcDjbmvJyDGenYNN9MK0RLw81m7eVURTWzgzWsR2gUqjBv+Uf
JyGX+JgSxfT5MeBzcYZHTJ6IhakqUV1Z++enEVfGlB/FZMlJxiZvd/wG0irJJb75
yYQ+WNvEVGlz+mvJihsEDN/V3yVvBYT4wKQc1Eo+4Y+7xCeP7+spU+b16UkCCA67
y8GJmzetZkahqcy65U0MgxiXi2p8kG4mbDCM2S4+a/CLtGQXjjDGh8cxxS07H2TW
kD7GN5e4sXC/AgMBAAGjggK+MIICujAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYw
FAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFF/E
E6OetCaS/LI8PTlyOhaEhH4QMB8GA1UdIwQYMBaAFMDMA0a5WCDMXHJw8+EuyyCm
9Wg6MHcGCCsGAQUFBwEBBGswaTAyBggrBgEFBQcwAYYmaHR0cDovL29jc3Auc3Rn
LWludC14MS5sZXRzZW5jcnlwdC5vcmcwMwYIKwYBBQUHMAKGJ2h0dHA6Ly9jZXJ0
LnN0Zy1pbnQteDEubGV0c2VuY3J5cHQub3JnLzBtBgNVHREEZjBkghZhcGkuYmFs
ZW5hLmRnY3NkZXYuY29tghtyZWdpc3RyeS5iYWxlbmEuZGdjc2Rldi5jb22CFXMz
LmJhbGVuYS5kZ2NzZGV2LmNvbYIWdnBuLmJhbGVuYS5kZ2NzZGV2LmNvbTBMBgNV
HSAERTBDMAgGBmeBDAECATA3BgsrBgEEAYLfEwEBATAoMCYGCCsGAQUFBwIBFhpo
dHRwOi8vY3BzLmxldHNlbmNyeXB0Lm9yZzCCAQMGCisGAQQB1nkCBAIEgfQEgfEA
7wB1ABboacHRlerXw/iXGuPwdgH3jOG2nTGoUhi2g38xqBUIAAABb6pjSwgAAAQD
AEYwRAIgH6NwveXI4M34FLF3v/YK8tKeFiq7btLLip5sW4AbKfYCIAzE0nASxdc/
0/WFoyqId/QKcEwfvERag1nwjeU71A6VAHYAxj8iGMN9VqaqBrWW2o5T1NcVbR6b
rI5E0iAt5k1p2dwAAAFvqmNLDgAABAMARzBFAiEAopaXt1EO6IDCj2a+pBlxf/m5
lqCfqWcP0laXDOYCx30CIBSgiu198UP+N+GwkoD6sXzlsgm5NlRDjyKBxys/Btf4
MA0GCSqGSIb3DQEBCwUAA4IBAQCkbjsFsiMkhcFl+AzgxcPZw2KeB9cCT7LZF25b
C6Wod1zekCptrRsCQTL4ujSZQShPzAbWvHsy1O7xGmbSjHYsfjjj79zu5QH6rQv5
/gFr1ektZ+ffVy5fxz3F/Hy+sOD/GC19V4yTsvbz5IE7pFGCZAVlynwJnqi4RqHY
pntNLixnJsJlRbl8o2iav0InOI98QBokBN/gfd6UF+XN38vRY/nQgni17dMrIEiW
aJlSTHbH5gazZb38EhZ/PJRdtiSbx+cNu9V+Kg7pRf/CsAL1cA2pscLMC3HCfed+
0QqRndCaXNok04Qs51/+xDMVoRYCD+iD0cPw4bMKoi5tbv5i
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIEqzCCApOgAwIBAgIRAIvhKg5ZRO08VGQx8JdhT+UwDQYJKoZIhvcNAQELBQAw
GjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMB4XDTE2MDUyMzIyMDc1OVoXDTM2
MDUyMzIyMDc1OVowIjEgMB4GA1UEAwwXRmFrZSBMRSBJbnRlcm1lZGlhdGUgWDEw
ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDtWKySDn7rWZc5ggjz3ZB0
8jO4xti3uzINfD5sQ7Lj7hzetUT+wQob+iXSZkhnvx+IvdbXF5/yt8aWPpUKnPym
oLxsYiI5gQBLxNDzIec0OIaflWqAr29m7J8+NNtApEN8nZFnf3bhehZW7AxmS1m0
ZnSsdHw0Fw+bgixPg2MQ9k9oefFeqa+7Kqdlz5bbrUYV2volxhDFtnI4Mh8BiWCN
xDH1Hizq+GKCcHsinDZWurCqder/afJBnQs+SBSL6MVApHt+d35zjBD92fO2Je56
dhMfzCgOKXeJ340WhW3TjD1zqLZXeaCyUNRnfOmWZV8nEhtHOFbUCU7r/KkjMZO9
AgMBAAGjgeMwgeAwDgYDVR0PAQH/BAQDAgGGMBIGA1UdEwEB/wQIMAYBAf8CAQAw
HQYDVR0OBBYEFMDMA0a5WCDMXHJw8+EuyyCm9Wg6MHoGCCsGAQUFBwEBBG4wbDA0
BggrBgEFBQcwAYYoaHR0cDovL29jc3Auc3RnLXJvb3QteDEubGV0c2VuY3J5cHQu
b3JnLzA0BggrBgEFBQcwAoYoaHR0cDovL2NlcnQuc3RnLXJvb3QteDEubGV0c2Vu
Y3J5cHQub3JnLzAfBgNVHSMEGDAWgBTBJnSkikSg5vogKNhcI5pFiBh54DANBgkq
hkiG9w0BAQsFAAOCAgEABYSu4Il+fI0MYU42OTmEj+1HqQ5DvyAeyCA6sGuZdwjF
UGeVOv3NnLyfofuUOjEbY5irFCDtnv+0ckukUZN9lz4Q2YjWGUpW4TTu3ieTsaC9
AFvCSgNHJyWSVtWvB5XDxsqawl1KzHzzwr132bF2rtGtazSqVqK9E07sGHMCf+zp
DQVDVVGtqZPHwX3KqUtefE621b8RI6VCl4oD30Olf8pjuzG4JKBFRFclzLRjo/h7
IkkfjZ8wDa7faOjVXx6n+eUQ29cIMCzr8/rNWHS9pYGGQKJiY2xmVC9h12H99Xyf
zWE9vb5zKP3MVG6neX1hSdo7PEAb9fqRhHkqVsqUvJlIRmvXvVKTwNCP3eCjRCCI
PTAvjV+4ni786iXwwFYNz8l3PmPLCyQXWGohnJ8iBm+5nk7O2ynaPVW0U2W+pt2w
SVuvdDM5zGv2f9ltNWUiYZHJ1mmO97jSY/6YfdOUH66iRtQtDkHBRdkNBsMbD+Em
2TgBldtHNSJBfB3pm9FblgOcJ0FSWcUDWJ7vO0+NTXlgrRofRT6pVywzxVo6dND0
WzYlTWeUVsO40xJqhgUQRER9YLOLxJ0O6C8i0xFxAMKOtSdodMB3RIwt7RFQ0uyt
n5Z5MqkYhlMI3J1tPRTp1nEt9fyGspBOO05gi148Qasp+3N+svqKomoQglNoAxU=
-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----
I’ve then in the haproxy container killed the haproxy process manually and started it again. I assume this will guarantee that it picks up the latest open-balena.pem. All starts ok.
Still getting the same error:
UNABLE_TO_GET_ISSUER_CERT_LOCALLY: request to https://api.{mydomain}/login_ failed, reason: unable to get local issuer certificate
And the output from the ssl query still shows a fake intermediate cert:
$ echo | openssl s_client -showcerts -servername registry.{mydomain} -connect registry.{mydomain}:443 2>/dev/null | openssl x509 -inform pem -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
fa:9d:c1:ce:f9:83:b2:12:bd:d8:93:8f:7d:24:f0:c1:7b:cf
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=Fake LE Intermediate X1
Validity
Not Before: Jan 15 17:07:11 2020 GMT
Not After : Apr 14 17:07:11 2020 GMT
Subject: CN=api.{mydomain}
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:b3:18:87:c3:00:c0:40:62:ea:3c:5e:17:6d:c3:
9a:14:cd:19:ed:05:9a:77:d2:27:cd:48:e3:f4:c4:
66:59:f4:13:eb:77:0d:94:da:5b:c2:ed:bd:7e:4a:
ea:a1:47:6b:e1:a8:ae:3b:b9:88:73:dd:12:3b:08:
58:06:13:4a:f2:36:9e:16:70:38:db:9a:f2:72:0c:
67:a7:60:d3:7d:30:ad:11:2f:0f:35:9b:b7:95:51:
14:d6:ce:0c:d6:b1:1d:a0:52:a8:c1:bf:e5:1f:27:
21:97:f8:98:12:c5:f4:f9:31:e0:73:71:86:47:4c:
9e:88:85:a9:2a:51:5d:59:fb:e7:a7:11:57:c6:94:
1f:c5:64:c9:49:c6:26:6f:77:fc:06:d2:2a:c9:25:
be:f9:c9:84:3e:58:db:c4:54:69:73:fa:6b:c9:8a:
1b:04:0c:df:d5:df:25:6f:05:84:f8:c0:a4:1c:d4:
4a:3e:e1:8f:bb:c4:27:8f:ef:eb:29:53:e6:f5:e9:
49:02:08:0e:bb:cb:c1:89:9b:37:ad:66:46:a1:a9:
cc:ba:e5:4d:0c:83:18:97:8b:6a:7c:90:6e:26:6c:
30:8c:d9:2e:3e:6b:f0:8b:b4:64:17:8e:30:c6:87:
c7:31:c5:2d:3b:1f:64:d6:90:3e:c6:37:97:b8:b1:
70:bf
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Key Identifier:
5F:C4:13:A3:9E:B4:26:92:FC:B2:3C:3D:39:72:3A:16:84:84:7E:10
X509v3 Authority Key Identifier:
keyid:C0:CC:03:46:B9:58:20:CC:5C:72:70:F3:E1:2E:CB:20:A6:F5:68:3A
Authority Information Access:
OCSP - URI:http://ocsp.stg-int-x1.letsencrypt.org
CA Issuers - URI:http://cert.stg-int-x1.letsencrypt.org/
X509v3 Subject Alternative Name:
DNS:api.{mydomain}, DNS:registry.{mydomain}, DNS:s3.{mydomain}, DNS:vpn.{mydomain}
X509v3 Certificate Policies:
Policy: 2.23.140.1.2.1
Policy: 1.3.6.1.4.1.44947.1.1.1
CPS: http://cps.letsencrypt.org
1.3.6.1.4.1.11129.2.4.2:
......u...i...........v......1.R....1......o.cK......F0D. ..p........w..
....*.n....l[..).. ...p...?....*.w.
pL..DZ.Y...;....v..?"..}V.......S...m....D. -.Mi.....o.cK......G0E.!.....Q.....f...q.......g..V.....}. ....}.C.7......|....6TC."..+?...
Signature Algorithm: sha256WithRSAEncryption
a4:6e:3b:05:b2:23:24:85:c1:65:f8:0c:e0:c5:c3:d9:c3:62:
9e:07:d7:02:4f:b2:d9:17:6e:5b:0b:a5:a8:77:5c:de:90:2a:
6d:ad:1b:02:41:32:f8:ba:34:99:41:28:4f:cc:06:d6:bc:7b:
32:d4:ee:f1:1a:66:d2:8c:76:2c:7e:38:e3:ef:dc:ee:e5:01:
fa:ad:0b:f9:fe:01:6b:d5:e9:2d:67:e7:df:57:2e:5f:c7:3d:
c5:fc:7c:be:b0:e0:ff:18:2d:7d:57:8c:93:b2:f6:f3:e4:81:
3b:a4:51:82:64:05:65:ca:7c:09:9e:a8:b8:46:a1:d8:a6:7b:
4d:2e:2c:67:26:c2:65:45:b9:7c:a3:68:9a:bf:42:27:38:8f:
7c:40:1a:24:04:df:e0:7d:de:94:17:e5:cd:df:cb:d1:63:f9:
d0:82:78:b5:ed:d3:2b:20:48:96:68:99:52:4c:76:c7:e6:06:
b3:65:bd:fc:12:16:7f:3c:94:5d:b6:24:9b:c7:e7:0d:bb:d5:
7e:2a:0e:e9:45:ff:c2:b0:02:f5:70:0d:a9:b1:c2:cc:0b:71:
c2:7d:e7:7e:d1:0a:91:9d:d0:9a:5c:da:24:d3:84:2c:e7:5f:
fe:c4:33:15:a1:16:02:0f:e8:83:d1:c3:f0:e1:b3:0a:a2:2e:
6d:6e:fe:62
I found the fake-le-bundle.pem in the repo, and noticed that it contains the same second certificate (MIIEqzCCApOg…). I’m wondering if this a fake cert or a shared public reference certificate? Should it appear in the production certs?
-----BEGIN CERTIFICATE-----
MIIFATCCAumgAwIBAgIRAKc9ZKBASymy5TLOEp57N98wDQYJKoZIhvcNAQELBQAw
GjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMB4XDTE2MDMyMzIyNTM0NloXDTM2
MDMyMzIyNTM0NlowGjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMIICIjANBgkq
hkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA+pYHvQw5iU3v2b3iNuYNKYgsWD6KU7aJ
diddtZQxSWYzUI3U0I1UsRPTxnhTifs/M9NW4ZlV13ZfB7APwC8oqKOIiwo7IwlP
xg0VKgyz+kT8RJfYr66PPIYP0fpTeu42LpMJ+CKo9sbpgVNDZN2z/qiXrRNX/VtG
TkPV7a44fZ5bHHVruAxvDnylpQxJobtCBWlJSsbIRGFHMc2z88eUz9NmIOWUKGGj
EmP76x8OfRHpIpuxRSCjn0+i9+hR2siIOpcMOGd+40uVJxbRRP5ZXnUFa2fF5FWd
O0u0RPI8HON0ovhrwPJY+4eWKkQzyC611oLPYGQ4EbifRsTsCxUZqyUuStGyp8oa
aoSKfF6X0+KzGgwwnrjRTUpIl19A92KR0Noo6h622OX+4sZiO/JQdkuX5w/HupK0
A0M0WSMCvU6GOhjGotmh2VTEJwHHY4+TUk0iQYRtv1crONklyZoAQPD76hCrC8Cr
IbgsZLfTMC8TWUoMbyUDgvgYkHKMoPm0VGVVuwpRKJxv7+2wXO+pivrrUl2Q9fPe
Kk055nJLMV9yPUdig8othUKrRfSxli946AEV1eEOhxddfEwBE3Lt2xn0hhiIedbb
Ftf/5kEWFZkXyUmMJK8Ra76Kus2ABueUVEcZ48hrRr1Hf1N9n59VbTUaXgeiZA50
qXf2bymE6F8CAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMB
Af8wHQYDVR0OBBYEFMEmdKSKRKDm+iAo2FwjmkWIGHngMA0GCSqGSIb3DQEBCwUA
A4ICAQBCPw74M9X/Xx04K1VAES3ypgQYH5bf9FXVDrwhRFSVckria/7dMzoF5wln
uq9NGsjkkkDg17AohcQdr8alH4LvPdxpKr3BjpvEcmbqF8xH+MbbeUEnmbSfLI8H
sefuhXF9AF/9iYvpVNC8FmJ0OhiVv13VgMQw0CRKkbtjZBf8xaEhq/YqxWVsgOjm
dm5CAQ2X0aX7502x8wYRgMnZhA5goC1zVWBVAi8yhhmlhhoDUfg17cXkmaJC5pDd
oenZ9NVhW8eDb03MFCrWNvIh89DDeCGWuWfDltDq0n3owyL0IeSn7RfpSclpxVmV
/53jkYjwIgxIG7Gsv0LKMbsf6QdBcTjhvfZyMIpBRkTe3zuHd2feKzY9lEkbRvRQ
zbh4Ps5YBnG6CKJPTbe2hfi3nhnw/MyEmF3zb0hzvLWNrR9XW3ibb2oL3424XOwc
VjrTSCLzO9Rv6s5wi03qoWvKAQQAElqTYRHhynJ3w6wuvKYF5zcZF3MDnrVGLbh1
Q9ePRFBCiXOQ6wPLoUhrrbZ8LpFUFYDXHMtYM7P9sc9IAWoONXREJaO08zgFtMp4
8iyIYUyQAbsvx8oD2M8kRvrIRSrRJSl6L957b4AFiLIQ/GgV2curs0jje7Edx34c
idWw1VrejtwclobqNMVtG3EiPUIpJGpbMcJgbiLSmKkrvQtGng==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIEqzCCApOgAwIBAgIRAIvhKg5ZRO08VGQx8JdhT+UwDQYJKoZIhvcNAQELBQAw
GjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMB4XDTE2MDUyMzIyMDc1OVoXDTM2
MDUyMzIyMDc1OVowIjEgMB4GA1UEAwwXRmFrZSBMRSBJbnRlcm1lZGlhdGUgWDEw
ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDtWKySDn7rWZc5ggjz3ZB0
8jO4xti3uzINfD5sQ7Lj7hzetUT+wQob+iXSZkhnvx+IvdbXF5/yt8aWPpUKnPym
oLxsYiI5gQBLxNDzIec0OIaflWqAr29m7J8+NNtApEN8nZFnf3bhehZW7AxmS1m0
ZnSsdHw0Fw+bgixPg2MQ9k9oefFeqa+7Kqdlz5bbrUYV2volxhDFtnI4Mh8BiWCN
xDH1Hizq+GKCcHsinDZWurCqder/afJBnQs+SBSL6MVApHt+d35zjBD92fO2Je56
dhMfzCgOKXeJ340WhW3TjD1zqLZXeaCyUNRnfOmWZV8nEhtHOFbUCU7r/KkjMZO9
AgMBAAGjgeMwgeAwDgYDVR0PAQH/BAQDAgGGMBIGA1UdEwEB/wQIMAYBAf8CAQAw
HQYDVR0OBBYEFMDMA0a5WCDMXHJw8+EuyyCm9Wg6MHoGCCsGAQUFBwEBBG4wbDA0
BggrBgEFBQcwAYYoaHR0cDovL29jc3Auc3RnLXJvb3QteDEubGV0c2VuY3J5cHQu
b3JnLzA0BggrBgEFBQcwAoYoaHR0cDovL2NlcnQuc3RnLXJvb3QteDEubGV0c2Vu
Y3J5cHQub3JnLzAfBgNVHSMEGDAWgBTBJnSkikSg5vogKNhcI5pFiBh54DANBgkq
hkiG9w0BAQsFAAOCAgEABYSu4Il+fI0MYU42OTmEj+1HqQ5DvyAeyCA6sGuZdwjF
UGeVOv3NnLyfofuUOjEbY5irFCDtnv+0ckukUZN9lz4Q2YjWGUpW4TTu3ieTsaC9
AFvCSgNHJyWSVtWvB5XDxsqawl1KzHzzwr132bF2rtGtazSqVqK9E07sGHMCf+zp
DQVDVVGtqZPHwX3KqUtefE621b8RI6VCl4oD30Olf8pjuzG4JKBFRFclzLRjo/h7
IkkfjZ8wDa7faOjVXx6n+eUQ29cIMCzr8/rNWHS9pYGGQKJiY2xmVC9h12H99Xyf
zWE9vb5zKP3MVG6neX1hSdo7PEAb9fqRhHkqVsqUvJlIRmvXvVKTwNCP3eCjRCCI
PTAvjV+4ni786iXwwFYNz8l3PmPLCyQXWGohnJ8iBm+5nk7O2ynaPVW0U2W+pt2w
SVuvdDM5zGv2f9ltNWUiYZHJ1mmO97jSY/6YfdOUH66iRtQtDkHBRdkNBsMbD+Em
2TgBldtHNSJBfB3pm9FblgOcJ0FSWcUDWJ7vO0+NTXlgrRofRT6pVywzxVo6dND0
WzYlTWeUVsO40xJqhgUQRER9YLOLxJ0O6C8i0xFxAMKOtSdodMB3RIwt7RFQ0uyt
n5Z5MqkYhlMI3J1tPRTp1nEt9fyGspBOO05gi148Qasp+3N+svqKomoQglNoAxU=
-----END CERTIFICATE-----
fyi, I’ve tried curl -v https://api.{mydomain}/login_ from a number of different OSes and neworks, so pretty convinced it’s not an issue with my local environment.
What am I missing? 
^^ @dfunckt Yup, always with ./scripts/compose.
Right! I finally got it working. Here’s how:
I killed absolutely everything in Docker.
$ docker stop [....all containers]
$ docker system prune -a
$ docker volume prune
Then rebuilt everything:
$ ./scripts/quickstart -U {myemail} -P '{mypass}' -c -d {mydomain}
$ ./scripts/compose up -d
Then I tailed all the logs as soon as I was able:
...
[Wed Jan 15 18:07:12 UTC 2020] Installing cert to:/tmp/cert.pem
[Wed Jan 15 18:07:12 UTC 2020] Installing key to:/tmp/key.pem
[Wed Jan 15 18:07:12 UTC 2020] Installing full chain to:/tmp/fullchain.pem
[Wed Jan 15 18:07:12 UTC 2020] Run reload cmd: cat /tmp/fullchain.pem /tmp/key.pem > /certs/open-balena.pem
[Wed Jan 15 18:07:12 UTC 2020] Reload success
[Info] Waiting for api.{mydomain} to use a staging certificate...
[Info] (1/3) Connecting...
[Info] (1/3) Failed. Retrying in 5 seconds...
[Info] (2/3) Connecting...
[Info] (2/3) Failed. Retrying in 5 seconds...
[Info] (3/3) Connecting...
[Info] (3/3) Failed!
[Error] Unable to detect certificate change over. Cannot issue a production certificate. [Stopping]
...
[ALERT] 015/160934 (14) : parsing [/usr/local/etc/haproxy/haproxy.cfg:48] : 'bind 127.0.0.1:444' : unable to load SSL private key from PEM file '/etc/ssl/private/open-balena.pem'.
...
I noticed in an earlier post that someone had success with restarting haproxy (which in this case I didn’t need to restart).
I’ve been round the loop of wiping and restarting things a bunch of times. I cannot honestly remember for sure whether I’ve tried this exact sequence before (start -> wait -> restart cert_provider -> success). But I have definitely restarted various containers in various loops to see what happens. In any case, I don’t think I changed anything in the end beyond adding the -c flag and patching with the cert-provider-update branch, although I do have a bunch of debug log lines scattered about now too.
My conclusion is that this feels an awful lot like a race condition, with potential to enter an inconsistent state - e.g. the cert provider re-running and thinking it already has a prod cert. I might suggest the cert_provider should wait longer before bailing the first time.
I’m sorry I can’t provide a better answer for why it worked this time round, when I’ve been round the wipe&restart loop the best part of a dozen times. For anyone following me with the same error, I suggest watching the docker logs on startup like a hawk and restarting containers in various orders until it works
In any case, thank you very much @richbayliss for all your support and help. Hugely appreciated. I think this particular workflow has felt a little flaky, but a great product otherwise. We’re using it at my work, but I actually liked it so much I just went and created a Balena sound setup (with an Allo Boss DAC) for a bunch of spotify-connect clients at home just for kicks (using balena-cloud, of course 
). Big fan 
Hi @al, thanks for sharing you solution !
Hi there @DistgenAl , @dfunckt . I have tried more than 10 times with different scenarios that I have found on this topic and others on balena forum and other links on web.
Does anybody has a clue why the Lets Encrypt cert issuer is using the staging certificates?
Here is an output from the logs:
balena@cloud:~/open-balena$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a2230b7a6342 openbalena_haproxy "/docker-entrypoint.…" 43 minutes ago Up 43 minutes 0.0.0.0:80->80/tcp, 222/tcp, 3128/tcp, 5432/tcp, 0.0.0.0:443->443/tcp, 6379/tcp openbalena_haproxy_1
486751d77f00 balena/open-balena-vpn:v9.17.4 "/usr/bin/entry.sh" 43 minutes ago Up 43 minutes 80/tcp, 443/tcp, 3128/tcp openbalena_vpn_1
909ecaaf9eb2 balena/open-balena-api:v0.119.5 "/usr/bin/entry.sh" 43 minutes ago Up 43 minutes 80/tcp openbalena_api_1
9d70b256beca balena/open-balena-registry:v2.16.0 "/usr/bin/entry.sh" 43 minutes ago Up 43 minutes 80/tcp openbalena_registry_1
9d64f5633524 balena/open-balena-db:v4.1.0 "/balena-entrypoint.…" 43 minutes ago Up 43 minutes 5432/tcp openbalena_db_1
1e3e0ec5250b balena/open-balena-s3:v2.9.9 "/usr/bin/entry.sh" 43 minutes ago Up 43 minutes 80/tcp openbalena_s3_1
d133e823c22f redis:alpine "docker-entrypoint.s…" 43 minutes ago Up 43 minutes 6379/tcp openbalena_redis_1
8bb3fd329a21 openbalena_cert-provider "/entry.sh /usr/src/…" 43 minutes ago Up 43 minutes 80/tcp openbalena_cert-provider_1
balena@cloud:~/open-balena$ docker logs 8bb3fd329a21
[Info] VALIDATION not set. Using default: http-01
[Info] Waiting for api.cloud.mydomain.com to be available via HTTP...
[Info] (1/3) Connecting...
[Info] (1/3) Failed. Retrying in 5 seconds...
[Info] (2/3) Connecting...
[Info] (2/3) Failed. Retrying in 5 seconds...
[Info] (3/3) Connecting...
[Info] (3/3) Failed!
[Info] Unable to access api.cloud.mydomain.com on port 80. This is needed for certificate validation. Retrying in 30 seconds...
[Info] Waiting for api.cloud.mydomain.com to be available via HTTP...
[Info] (1/3) Connecting...
[Info] (1/3) Success!
cat: can't open '/usr/src/app/certs/last_run_mode': No such file or directory
[Info] Last acquired certificate for
[Info] Using STAGING mode
[Info] Waiting for api.cloud.mydomain.com to be available via HTTP...
[Info] (1/3) Connecting...
[Info] (1/3) Success!
[Info] Issuing certificates...
[Fri May 7 22:46:06 UTC 2021] Using stage ACME_DIRECTORY: https://acme-staging-v02.api.letsencrypt.org/directory
Whats even stranger if I’m running the following command the following command:
balena@cloud:~/open-balena$ curl -k https://api.cloud.mydomain.com/ping
OKbalena@cloud:~/open-balena$
Please note that https://api.cloud.mydomain.com or https://cloud.mydomain.com prompts a self-signed certificate warning in browser.
What am I missing? Could anyone please specify the specific versions that are working? Like guys from Balena team have provided in this video?
Looking forward any idea!??!
Hi Alex,
I’m not a maintainer of openBalena, but I’m using it myself.
The test with the curl command used the -k option, which allows curl to use insecure connections (in other words, the certificate doesn’t have to be valid). So that’s why curl doesn’t show a warning.
How did you use the quickstart command? You’ve used the -c option? So like:
./scripts/quickstart -U <user-email> -P <password> -d <domain> -c
Hope you’ll figure it out and will start using openbalena! 
Hi everyone,
I went through all the steps in this thread, but nothing worked for me.
My cert-provider always stops with the following logs:
cert-provider_1 | [Info] Waiting for api.{domain} to use a staging certificate...
cert-provider_1 | [Info] (1/3) Connecting...
cert-provider_1 | [Info] (1/3) Failed. Retrying in 5 seconds...
cert-provider_1 | [Info] (2/3) Connecting...
cert-provider_1 | [Info] (2/3) Failed. Retrying in 5 seconds...
cert-provider_1 | [Info] (3/3) Connecting...
cert-provider_1 | [Info] (3/3) Failed!
cert-provider_1 | [Error] Unable to detect certificate change over. Cannot issue a production certificate. [Stopping]
My haproxy container shows the following logs:
haproxy_1 | Building certificate from environment variables...
haproxy_1 | Setting up watches. Beware: since -r was given, this may take a while!
haproxy_1 | Watches established.
haproxy_1 | [NOTICE] 131/104238 (16) : New worker #1 (18) forked
haproxy_1 | [WARNING] 131/104238 (18) : Server backend_api/balena_api_1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
haproxy_1 | [ALERT] 131/104238 (18) : backend 'backend_api' has no server available!
haproxy_1 | [WARNING] 131/104239 (18) : Server vpn-tunnel/balena_vpn is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
haproxy_1 | [ALERT] 131/104239 (18) : proxy 'vpn-tunnel' has no server available!
haproxy_1 | [WARNING] 131/104239 (18) : Server vpn-tunnel-tls/balena_vpn is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
haproxy_1 | [ALERT] 131/104239 (18) : proxy 'vpn-tunnel-tls' has no server available!
haproxy_1 | Building certificate from environment variables...
haproxy_1 | Setting up watches. Beware: since -r was given, this may take a while!
haproxy_1 | Watches established.
haproxy_1 | [NOTICE] 131/104528 (17) : New worker #1 (19) forked
haproxy_1 | [WARNING] 131/104528 (19) : Server backend_api/balena_api_1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
haproxy_1 | [ALERT] 131/104528 (19) : backend 'backend_api' has no server available!
haproxy_1 | [WARNING] 131/104530 (19) : Server vpn-tunnel/balena_vpn is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
haproxy_1 | [ALERT] 131/104530 (19) : proxy 'vpn-tunnel' has no server available!
haproxy_1 | [WARNING] 131/104530 (19) : Server vpn-tunnel-tls/balena_vpn is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
haproxy_1 | [ALERT] 131/104530 (19) : proxy 'vpn-tunnel-tls' has no server available!
haproxy_1 | Building certificate from environment variables...
haproxy_1 | Setting up watches. Beware: since -r was given, this may take a while!
haproxy_1 | Watches established.
haproxy_1 | [NOTICE] 131/104743 (17) : New worker #1 (19) forked
haproxy_1 | [WARNING] 131/104743 (19) : Server backend_api/balena_api_1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
haproxy_1 | [ALERT] 131/104743 (19) : backend 'backend_api' has no server available!
haproxy_1 | [WARNING] 131/104745 (19) : Server backend_s3/balena_s3_1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 606ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
haproxy_1 | [ALERT] 131/104745 (19) : backend 'backend_s3' has no server available!
haproxy_1 | [WARNING] 131/104745 (19) : Server vpn-tunnel/balena_vpn is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
haproxy_1 | [ALERT] 131/104745 (19) : proxy 'vpn-tunnel' has no server available!
haproxy_1 | [WARNING] 131/104745 (19) : Server vpn-tunnel-tls/balena_vpn is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 2ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
haproxy_1 | [ALERT] 131/104745 (19) : proxy 'vpn-tunnel-tls' has no server available!
haproxy_1 | [WARNING] 131/104749 (19) : Server backend_s3/balena_s3_1 is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
haproxy_1 | [WARNING] 131/104809 (19) : Server backend_api/balena_api_1 is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
haproxy_1 | [WARNING] 131/104823 (19) : Server vpn-tunnel/balena_vpn is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
haproxy_1 | [WARNING] 131/104823 (19) : Server vpn-tunnel-tls/balena_vpn is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
haproxy_1 | /certs/ CREATE open-balena.pem
haproxy_1 | Updating certificate from cert-provider...
haproxy_1 | Certificate change detected. Reloading...
haproxy_1 | [WARNING] 131/104859 (17) : Reexecuting Master process
haproxy_1 | [WARNING] 131/104859 (19) : Stopping frontend http-in in 0 ms.
haproxy_1 | [WARNING] 131/104859 (19) : Stopping frontend ssl-in in 0 ms.
haproxy_1 | [WARNING] 131/104859 (19) : Stopping backend redirect-to-https-in in 0 ms.
haproxy_1 | [WARNING] 131/104859 (19) : Stopping backend redirect-to-tunnel-in in 0 ms.
haproxy_1 | [NOTICE] 131/104859 (17) : New worker #1 (22) forked
haproxy_1 | [WARNING] 131/104859 (19) : Stopping frontend https-in in 0 ms.
haproxy_1 | [WARNING] 131/104859 (19) : Stopping backend backend_api in 0 ms.
haproxy_1 | [WARNING] 131/104859 (19) : Stopping backend backend_registry in 0 ms.
haproxy_1 | [WARNING] 131/104859 (19) : Stopping backend backend_vpn in 0 ms.
haproxy_1 | [WARNING] 131/104859 (19) : Stopping backend backend_s3 in 0 ms.
haproxy_1 | [WARNING] 131/104859 (19) : Stopping backend cert-provider in 0 ms.
haproxy_1 | [WARNING] 131/104859 (19) : Stopping backend vpn-devices in 0 ms.
haproxy_1 | [WARNING] 131/104859 (19) : Stopping frontend db in 0 ms.
haproxy_1 | [WARNING] 131/104859 (19) : Stopping backend backend_db in 0 ms.
haproxy_1 | [WARNING] 131/104859 (19) : Stopping frontend redis in 0 ms.
haproxy_1 | [WARNING] 131/104859 (19) : Stopping backend backend_redis in 0 ms.
haproxy_1 | [WARNING] 131/104859 (19) : Stopping proxy vpn-tunnel in 0 ms.
haproxy_1 | [WARNING] 131/104859 (19) : Stopping proxy vpn-tunnel-tls in 0 ms.
haproxy_1 | [WARNING] 131/104859 (19) : Stopping frontend GLOBAL in 0 ms.
haproxy_1 | [WARNING] 131/104859 (19) : Proxy http-in stopped (FE: 22 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/104859 (19) : Proxy ssl-in stopped (FE: 10 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/104859 (19) : Proxy redirect-to-https-in stopped (FE: 0 conns, BE: 10 conns).
haproxy_1 | [WARNING] 131/104859 (19) : Proxy redirect-to-tunnel-in stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/104859 (19) : Proxy https-in stopped (FE: 10 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/104859 (19) : Proxy backend_api stopped (FE: 0 conns, BE: 12 conns).
haproxy_1 | [WARNING] 131/104859 (19) : Proxy backend_registry stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/104859 (19) : Proxy backend_vpn stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/104859 (19) : Proxy backend_s3 stopped (FE: 0 conns, BE: 1 conns).
haproxy_1 | [WARNING] 131/104859 (19) : Proxy cert-provider stopped (FE: 0 conns, BE: 18 conns).
haproxy_1 | [WARNING] 131/104859 (19) : Proxy vpn-devices stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/104859 (19) : Proxy db stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/104859 (19) : Proxy backend_db stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/104859 (19) : Proxy redis stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/104859 (19) : Proxy backend_redis stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/104859 (19) : Proxy vpn-tunnel stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/104859 (19) : Proxy vpn-tunnel-tls stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/104859 (19) : Proxy GLOBAL stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/104859 (17) : Former worker #1 (19) exited with code 0 (Exit)
haproxy_1 | Setting up watches. Beware: since -r was given, this may take a while!
haproxy_1 | Watches established.
haproxy_1 | /certs/ MODIFY open-balena.pem
haproxy_1 | Updating certificate from cert-provider...
haproxy_1 | Certificate change detected. Reloading...
haproxy_1 | [WARNING] 131/110109 (17) : Reexecuting Master process
haproxy_1 | [WARNING] 131/110109 (22) : Stopping frontend http-in in 0 ms.
haproxy_1 | [WARNING] 131/110109 (22) : Stopping frontend ssl-in in 0 ms.
haproxy_1 | [WARNING] 131/110109 (22) : Stopping backend redirect-to-https-in in 0 ms.
haproxy_1 | [WARNING] 131/110109 (22) : Stopping backend redirect-to-tunnel-in in 0 ms.
haproxy_1 | [WARNING] 131/110109 (22) : Stopping frontend https-in in 0 ms.
haproxy_1 | [WARNING] 131/110109 (22) : Stopping backend backend_api in 0 ms.
haproxy_1 | [WARNING] 131/110109 (22) : Stopping backend backend_registry in 0 ms.
haproxy_1 | [WARNING] 131/110109 (22) : Stopping backend backend_vpn in 0 ms.
haproxy_1 | [WARNING] 131/110109 (22) : Stopping backend backend_s3 in 0 ms.
haproxy_1 | [WARNING] 131/110109 (22) : Stopping backend cert-provider in 0 ms.
haproxy_1 | [NOTICE] 131/110109 (17) : New worker #1 (26) forked
haproxy_1 | [WARNING] 131/110109 (22) : Stopping backend vpn-devices in 0 ms.
haproxy_1 | [WARNING] 131/110109 (22) : Stopping frontend db in 0 ms.
haproxy_1 | [WARNING] 131/110109 (22) : Stopping backend backend_db in 0 ms.
haproxy_1 | [WARNING] 131/110109 (22) : Stopping frontend redis in 0 ms.
haproxy_1 | [WARNING] 131/110109 (22) : Stopping backend backend_redis in 0 ms.
haproxy_1 | [WARNING] 131/110109 (22) : Stopping proxy vpn-tunnel in 0 ms.
haproxy_1 | [WARNING] 131/110109 (22) : Stopping proxy vpn-tunnel-tls in 0 ms.
haproxy_1 | [WARNING] 131/110109 (22) : Stopping frontend GLOBAL in 0 ms.
haproxy_1 | [WARNING] 131/110109 (22) : Proxy http-in stopped (FE: 3 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/110109 (22) : Proxy ssl-in stopped (FE: 77 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/110109 (22) : Proxy redirect-to-https-in stopped (FE: 0 conns, BE: 77 conns).
haproxy_1 | [WARNING] 131/110109 (22) : Proxy redirect-to-tunnel-in stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/110109 (22) : Proxy https-in stopped (FE: 77 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/110109 (22) : Proxy backend_api stopped (FE: 0 conns, BE: 4 conns).
haproxy_1 | [WARNING] 131/110109 (22) : Proxy backend_registry stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/110109 (22) : Proxy backend_vpn stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/110109 (22) : Proxy backend_s3 stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/110109 (22) : Proxy cert-provider stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/110109 (22) : Proxy vpn-devices stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/110109 (22) : Proxy db stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/110109 (22) : Proxy backend_db stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/110109 (22) : Proxy redis stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/110109 (22) : Proxy backend_redis stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/110109 (22) : Proxy vpn-tunnel stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/110109 (22) : Proxy vpn-tunnel-tls stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/110109 (22) : Proxy GLOBAL stopped (FE: 0 conns, BE: 0 conns).
haproxy_1 | [WARNING] 131/110109 (17) : Former worker #1 (22) exited with code 0 (Exit)
haproxy_1 | Setting up watches. Beware: since -r was given, this may take a while!
haproxy_1 | Watches established.
The part where it says
haproxy_1 | /certs/ MODIFY open-balena.pem
haproxy_1 | Updating certificate from cert-provider...
haproxy_1 | Certificate change detected. Reloading...
tells me, that the haproxy recognizes the certificate changes. But I don’t know how to debug further.
When I run the command
openssl s_client -host "api.{domain}" -port 443 -showcerts | awk '/BEGIN CERT/ p=1} ; p==1; /END CERT/ {p=0}' | openssl verify -CAfile "/usr/src/app/fake-le-bundle.pem"
inside the cert-provider container (it is the command, that checks the staging cert in the cert-provider.sh file without the 2>/dev/null parts) I get the following output:
depth=2 C = US, O = (STAGING) Internet Security Research Group, CN = (STAGING) Pretend Pear X1
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=1 C = US, O = (STAGING) Let's Encrypt, CN = (STAGING) Artificial Apricot R3
verify return:1
depth=0 CN = api.{domain}
verify return:1
CN = api.{domain}
error 20 at 0 depth lookup: unable to get local issuer certificate
error stdin: verification failed
Maybe I should mention, that the open balena instatnce is running in a VM and because of the port forwarding I added every balena url (like api.{domain}, …) to the hosts file.
Can anyone help me with that?
Thanks!
Hi @ppoth
Not sure from the debug if this is the issue but one thing to be aware of when using letsencrypt is that letsencrypts external servers need to be able to access the server via the domain given to verify it which means you need a DNS setup for you domain to forward the request to you server and that your local server needs to have port 80 and 443 open to accept traffic via port forwarding/however requests to the domain from the internet are being directed to it.
If you are just testing locally or its for your internal network only you can get away with using the self signed certificates created by the quick start script when you dont specify using letsencrypt or you can manually setup each certificate with a valid certificate from any issuer by modifying the environment variables in config/activate
Cheers
dashals