Using real (not self-signed) certificates

Hey @richbayliss. Apologies for bugging you again, but I’m not making headway understanding what’s going on with these certs.

(I’ve got access to the containers now, with good old sh)

Running cert-provider.sh in the cert-provider container claims that the last production keys were production-mode.

The contents of open-balena.pem (in both container’s /cert/… and haproxy’s /etc/ssl/private/…) is as follows: (I’ve obfuscated the private key)

-----BEGIN CERTIFICATE-----
MIIFkDCCBHigAwIBAgITAPqdwc75g7ISvdiTj30k8MF7zzANBgkqhkiG9w0BAQsF
ADAiMSAwHgYDVQQDDBdGYWtlIExFIEludGVybWVkaWF0ZSBYMTAeFw0yMDAxMTUx
NzA3MTFaFw0yMDA0MTQxNzA3MTFaMCExHzAdBgNVBAMTFmFwaS5iYWxlbmEuZGdj
c2Rldi5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCzGIfDAMBA
Yuo8Xhdtw5oUzRntBZp30ifNSOP0xGZZ9BPrdw2U2lvC7b1+SuqhR2vhqK47uYhz
3RI7CFgGE0ryNp4WcDjbmvJyDGenYNN9MK0RLw81m7eVURTWzgzWsR2gUqjBv+Uf
JyGX+JgSxfT5MeBzcYZHTJ6IhakqUV1Z++enEVfGlB/FZMlJxiZvd/wG0irJJb75
yYQ+WNvEVGlz+mvJihsEDN/V3yVvBYT4wKQc1Eo+4Y+7xCeP7+spU+b16UkCCA67
y8GJmzetZkahqcy65U0MgxiXi2p8kG4mbDCM2S4+a/CLtGQXjjDGh8cxxS07H2TW
kD7GN5e4sXC/AgMBAAGjggK+MIICujAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYw
FAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFF/E
E6OetCaS/LI8PTlyOhaEhH4QMB8GA1UdIwQYMBaAFMDMA0a5WCDMXHJw8+EuyyCm
9Wg6MHcGCCsGAQUFBwEBBGswaTAyBggrBgEFBQcwAYYmaHR0cDovL29jc3Auc3Rn
LWludC14MS5sZXRzZW5jcnlwdC5vcmcwMwYIKwYBBQUHMAKGJ2h0dHA6Ly9jZXJ0
LnN0Zy1pbnQteDEubGV0c2VuY3J5cHQub3JnLzBtBgNVHREEZjBkghZhcGkuYmFs
ZW5hLmRnY3NkZXYuY29tghtyZWdpc3RyeS5iYWxlbmEuZGdjc2Rldi5jb22CFXMz
LmJhbGVuYS5kZ2NzZGV2LmNvbYIWdnBuLmJhbGVuYS5kZ2NzZGV2LmNvbTBMBgNV
HSAERTBDMAgGBmeBDAECATA3BgsrBgEEAYLfEwEBATAoMCYGCCsGAQUFBwIBFhpo
dHRwOi8vY3BzLmxldHNlbmNyeXB0Lm9yZzCCAQMGCisGAQQB1nkCBAIEgfQEgfEA
7wB1ABboacHRlerXw/iXGuPwdgH3jOG2nTGoUhi2g38xqBUIAAABb6pjSwgAAAQD
AEYwRAIgH6NwveXI4M34FLF3v/YK8tKeFiq7btLLip5sW4AbKfYCIAzE0nASxdc/
0/WFoyqId/QKcEwfvERag1nwjeU71A6VAHYAxj8iGMN9VqaqBrWW2o5T1NcVbR6b
rI5E0iAt5k1p2dwAAAFvqmNLDgAABAMARzBFAiEAopaXt1EO6IDCj2a+pBlxf/m5
lqCfqWcP0laXDOYCx30CIBSgiu198UP+N+GwkoD6sXzlsgm5NlRDjyKBxys/Btf4
MA0GCSqGSIb3DQEBCwUAA4IBAQCkbjsFsiMkhcFl+AzgxcPZw2KeB9cCT7LZF25b
C6Wod1zekCptrRsCQTL4ujSZQShPzAbWvHsy1O7xGmbSjHYsfjjj79zu5QH6rQv5
/gFr1ektZ+ffVy5fxz3F/Hy+sOD/GC19V4yTsvbz5IE7pFGCZAVlynwJnqi4RqHY
pntNLixnJsJlRbl8o2iav0InOI98QBokBN/gfd6UF+XN38vRY/nQgni17dMrIEiW
aJlSTHbH5gazZb38EhZ/PJRdtiSbx+cNu9V+Kg7pRf/CsAL1cA2pscLMC3HCfed+
0QqRndCaXNok04Qs51/+xDMVoRYCD+iD0cPw4bMKoi5tbv5i
-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----
MIIEqzCCApOgAwIBAgIRAIvhKg5ZRO08VGQx8JdhT+UwDQYJKoZIhvcNAQELBQAw
GjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMB4XDTE2MDUyMzIyMDc1OVoXDTM2
MDUyMzIyMDc1OVowIjEgMB4GA1UEAwwXRmFrZSBMRSBJbnRlcm1lZGlhdGUgWDEw
ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDtWKySDn7rWZc5ggjz3ZB0
8jO4xti3uzINfD5sQ7Lj7hzetUT+wQob+iXSZkhnvx+IvdbXF5/yt8aWPpUKnPym
oLxsYiI5gQBLxNDzIec0OIaflWqAr29m7J8+NNtApEN8nZFnf3bhehZW7AxmS1m0
ZnSsdHw0Fw+bgixPg2MQ9k9oefFeqa+7Kqdlz5bbrUYV2volxhDFtnI4Mh8BiWCN
xDH1Hizq+GKCcHsinDZWurCqder/afJBnQs+SBSL6MVApHt+d35zjBD92fO2Je56
dhMfzCgOKXeJ340WhW3TjD1zqLZXeaCyUNRnfOmWZV8nEhtHOFbUCU7r/KkjMZO9
AgMBAAGjgeMwgeAwDgYDVR0PAQH/BAQDAgGGMBIGA1UdEwEB/wQIMAYBAf8CAQAw
HQYDVR0OBBYEFMDMA0a5WCDMXHJw8+EuyyCm9Wg6MHoGCCsGAQUFBwEBBG4wbDA0
BggrBgEFBQcwAYYoaHR0cDovL29jc3Auc3RnLXJvb3QteDEubGV0c2VuY3J5cHQu
b3JnLzA0BggrBgEFBQcwAoYoaHR0cDovL2NlcnQuc3RnLXJvb3QteDEubGV0c2Vu
Y3J5cHQub3JnLzAfBgNVHSMEGDAWgBTBJnSkikSg5vogKNhcI5pFiBh54DANBgkq
hkiG9w0BAQsFAAOCAgEABYSu4Il+fI0MYU42OTmEj+1HqQ5DvyAeyCA6sGuZdwjF
UGeVOv3NnLyfofuUOjEbY5irFCDtnv+0ckukUZN9lz4Q2YjWGUpW4TTu3ieTsaC9
AFvCSgNHJyWSVtWvB5XDxsqawl1KzHzzwr132bF2rtGtazSqVqK9E07sGHMCf+zp
DQVDVVGtqZPHwX3KqUtefE621b8RI6VCl4oD30Olf8pjuzG4JKBFRFclzLRjo/h7
IkkfjZ8wDa7faOjVXx6n+eUQ29cIMCzr8/rNWHS9pYGGQKJiY2xmVC9h12H99Xyf
zWE9vb5zKP3MVG6neX1hSdo7PEAb9fqRhHkqVsqUvJlIRmvXvVKTwNCP3eCjRCCI
PTAvjV+4ni786iXwwFYNz8l3PmPLCyQXWGohnJ8iBm+5nk7O2ynaPVW0U2W+pt2w
SVuvdDM5zGv2f9ltNWUiYZHJ1mmO97jSY/6YfdOUH66iRtQtDkHBRdkNBsMbD+Em
2TgBldtHNSJBfB3pm9FblgOcJ0FSWcUDWJ7vO0+NTXlgrRofRT6pVywzxVo6dND0
WzYlTWeUVsO40xJqhgUQRER9YLOLxJ0O6C8i0xFxAMKOtSdodMB3RIwt7RFQ0uyt
n5Z5MqkYhlMI3J1tPRTp1nEt9fyGspBOO05gi148Qasp+3N+svqKomoQglNoAxU=
-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----
  ... 
-----END RSA PRIVATE KEY-----

I’ve then in the haproxy container killed the haproxy process manually and started it again. I assume this will guarantee that it picks up the latest open-balena.pem. All starts ok.

Still getting the same error:

UNABLE_TO_GET_ISSUER_CERT_LOCALLY: request to https://api.{mydomain}/login_ failed, reason: unable to get local issuer certificate

And the output from the ssl query still shows a fake intermediate cert:

$ echo | openssl s_client -showcerts -servername registry.{mydomain} -connect registry.{mydomain}:443 2>/dev/null | openssl x509 -inform pem -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            fa:9d:c1:ce:f9:83:b2:12:bd:d8:93:8f:7d:24:f0:c1:7b:cf
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=Fake LE Intermediate X1
        Validity
            Not Before: Jan 15 17:07:11 2020 GMT
            Not After : Apr 14 17:07:11 2020 GMT
        Subject: CN=api.{mydomain}
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:b3:18:87:c3:00:c0:40:62:ea:3c:5e:17:6d:c3:
                    9a:14:cd:19:ed:05:9a:77:d2:27:cd:48:e3:f4:c4:
                    66:59:f4:13:eb:77:0d:94:da:5b:c2:ed:bd:7e:4a:
                    ea:a1:47:6b:e1:a8:ae:3b:b9:88:73:dd:12:3b:08:
                    58:06:13:4a:f2:36:9e:16:70:38:db:9a:f2:72:0c:
                    67:a7:60:d3:7d:30:ad:11:2f:0f:35:9b:b7:95:51:
                    14:d6:ce:0c:d6:b1:1d:a0:52:a8:c1:bf:e5:1f:27:
                    21:97:f8:98:12:c5:f4:f9:31:e0:73:71:86:47:4c:
                    9e:88:85:a9:2a:51:5d:59:fb:e7:a7:11:57:c6:94:
                    1f:c5:64:c9:49:c6:26:6f:77:fc:06:d2:2a:c9:25:
                    be:f9:c9:84:3e:58:db:c4:54:69:73:fa:6b:c9:8a:
                    1b:04:0c:df:d5:df:25:6f:05:84:f8:c0:a4:1c:d4:
                    4a:3e:e1:8f:bb:c4:27:8f:ef:eb:29:53:e6:f5:e9:
                    49:02:08:0e:bb:cb:c1:89:9b:37:ad:66:46:a1:a9:
                    cc:ba:e5:4d:0c:83:18:97:8b:6a:7c:90:6e:26:6c:
                    30:8c:d9:2e:3e:6b:f0:8b:b4:64:17:8e:30:c6:87:
                    c7:31:c5:2d:3b:1f:64:d6:90:3e:c6:37:97:b8:b1:
                    70:bf
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier:
                5F:C4:13:A3:9E:B4:26:92:FC:B2:3C:3D:39:72:3A:16:84:84:7E:10
            X509v3 Authority Key Identifier:
                keyid:C0:CC:03:46:B9:58:20:CC:5C:72:70:F3:E1:2E:CB:20:A6:F5:68:3A

            Authority Information Access:
                OCSP - URI:http://ocsp.stg-int-x1.letsencrypt.org
                CA Issuers - URI:http://cert.stg-int-x1.letsencrypt.org/

            X509v3 Subject Alternative Name:
                DNS:api.{mydomain}, DNS:registry.{mydomain}, DNS:s3.{mydomain}, DNS:vpn.{mydomain}
            X509v3 Certificate Policies:
                Policy: 2.23.140.1.2.1
                Policy: 1.3.6.1.4.1.44947.1.1.1
                  CPS: http://cps.letsencrypt.org

            1.3.6.1.4.1.11129.2.4.2:
                ......u...i...........v......1.R....1......o.cK......F0D. ..p........w..
....*.n....l[..).. ...p...?....*.w.
pL..DZ.Y...;....v..?"..}V.......S...m....D. -.Mi.....o.cK......G0E.!.....Q.....f...q.......g..V.....}. ....}.C.7......|....6TC."..+?...
    Signature Algorithm: sha256WithRSAEncryption
         a4:6e:3b:05:b2:23:24:85:c1:65:f8:0c:e0:c5:c3:d9:c3:62:
         9e:07:d7:02:4f:b2:d9:17:6e:5b:0b:a5:a8:77:5c:de:90:2a:
         6d:ad:1b:02:41:32:f8:ba:34:99:41:28:4f:cc:06:d6:bc:7b:
         32:d4:ee:f1:1a:66:d2:8c:76:2c:7e:38:e3:ef:dc:ee:e5:01:
         fa:ad:0b:f9:fe:01:6b:d5:e9:2d:67:e7:df:57:2e:5f:c7:3d:
         c5:fc:7c:be:b0:e0:ff:18:2d:7d:57:8c:93:b2:f6:f3:e4:81:
         3b:a4:51:82:64:05:65:ca:7c:09:9e:a8:b8:46:a1:d8:a6:7b:
         4d:2e:2c:67:26:c2:65:45:b9:7c:a3:68:9a:bf:42:27:38:8f:
         7c:40:1a:24:04:df:e0:7d:de:94:17:e5:cd:df:cb:d1:63:f9:
         d0:82:78:b5:ed:d3:2b:20:48:96:68:99:52:4c:76:c7:e6:06:
         b3:65:bd:fc:12:16:7f:3c:94:5d:b6:24:9b:c7:e7:0d:bb:d5:
         7e:2a:0e:e9:45:ff:c2:b0:02:f5:70:0d:a9:b1:c2:cc:0b:71:
         c2:7d:e7:7e:d1:0a:91:9d:d0:9a:5c:da:24:d3:84:2c:e7:5f:
         fe:c4:33:15:a1:16:02:0f:e8:83:d1:c3:f0:e1:b3:0a:a2:2e:
         6d:6e:fe:62

I found the fake-le-bundle.pem in the repo, and noticed that it contains the same second certificate (MIIEqzCCApOg…). I’m wondering if this a fake cert or a shared public reference certificate? Should it appear in the production certs?

-----BEGIN CERTIFICATE-----
MIIFATCCAumgAwIBAgIRAKc9ZKBASymy5TLOEp57N98wDQYJKoZIhvcNAQELBQAw
GjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMB4XDTE2MDMyMzIyNTM0NloXDTM2
MDMyMzIyNTM0NlowGjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMIICIjANBgkq
hkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA+pYHvQw5iU3v2b3iNuYNKYgsWD6KU7aJ
diddtZQxSWYzUI3U0I1UsRPTxnhTifs/M9NW4ZlV13ZfB7APwC8oqKOIiwo7IwlP
xg0VKgyz+kT8RJfYr66PPIYP0fpTeu42LpMJ+CKo9sbpgVNDZN2z/qiXrRNX/VtG
TkPV7a44fZ5bHHVruAxvDnylpQxJobtCBWlJSsbIRGFHMc2z88eUz9NmIOWUKGGj
EmP76x8OfRHpIpuxRSCjn0+i9+hR2siIOpcMOGd+40uVJxbRRP5ZXnUFa2fF5FWd
O0u0RPI8HON0ovhrwPJY+4eWKkQzyC611oLPYGQ4EbifRsTsCxUZqyUuStGyp8oa
aoSKfF6X0+KzGgwwnrjRTUpIl19A92KR0Noo6h622OX+4sZiO/JQdkuX5w/HupK0
A0M0WSMCvU6GOhjGotmh2VTEJwHHY4+TUk0iQYRtv1crONklyZoAQPD76hCrC8Cr
IbgsZLfTMC8TWUoMbyUDgvgYkHKMoPm0VGVVuwpRKJxv7+2wXO+pivrrUl2Q9fPe
Kk055nJLMV9yPUdig8othUKrRfSxli946AEV1eEOhxddfEwBE3Lt2xn0hhiIedbb
Ftf/5kEWFZkXyUmMJK8Ra76Kus2ABueUVEcZ48hrRr1Hf1N9n59VbTUaXgeiZA50
qXf2bymE6F8CAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMB
Af8wHQYDVR0OBBYEFMEmdKSKRKDm+iAo2FwjmkWIGHngMA0GCSqGSIb3DQEBCwUA
A4ICAQBCPw74M9X/Xx04K1VAES3ypgQYH5bf9FXVDrwhRFSVckria/7dMzoF5wln
uq9NGsjkkkDg17AohcQdr8alH4LvPdxpKr3BjpvEcmbqF8xH+MbbeUEnmbSfLI8H
sefuhXF9AF/9iYvpVNC8FmJ0OhiVv13VgMQw0CRKkbtjZBf8xaEhq/YqxWVsgOjm
dm5CAQ2X0aX7502x8wYRgMnZhA5goC1zVWBVAi8yhhmlhhoDUfg17cXkmaJC5pDd
oenZ9NVhW8eDb03MFCrWNvIh89DDeCGWuWfDltDq0n3owyL0IeSn7RfpSclpxVmV
/53jkYjwIgxIG7Gsv0LKMbsf6QdBcTjhvfZyMIpBRkTe3zuHd2feKzY9lEkbRvRQ
zbh4Ps5YBnG6CKJPTbe2hfi3nhnw/MyEmF3zb0hzvLWNrR9XW3ibb2oL3424XOwc
VjrTSCLzO9Rv6s5wi03qoWvKAQQAElqTYRHhynJ3w6wuvKYF5zcZF3MDnrVGLbh1
Q9ePRFBCiXOQ6wPLoUhrrbZ8LpFUFYDXHMtYM7P9sc9IAWoONXREJaO08zgFtMp4
8iyIYUyQAbsvx8oD2M8kRvrIRSrRJSl6L957b4AFiLIQ/GgV2curs0jje7Edx34c
idWw1VrejtwclobqNMVtG3EiPUIpJGpbMcJgbiLSmKkrvQtGng==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIEqzCCApOgAwIBAgIRAIvhKg5ZRO08VGQx8JdhT+UwDQYJKoZIhvcNAQELBQAw
GjEYMBYGA1UEAwwPRmFrZSBMRSBSb290IFgxMB4XDTE2MDUyMzIyMDc1OVoXDTM2
MDUyMzIyMDc1OVowIjEgMB4GA1UEAwwXRmFrZSBMRSBJbnRlcm1lZGlhdGUgWDEw
ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDtWKySDn7rWZc5ggjz3ZB0
8jO4xti3uzINfD5sQ7Lj7hzetUT+wQob+iXSZkhnvx+IvdbXF5/yt8aWPpUKnPym
oLxsYiI5gQBLxNDzIec0OIaflWqAr29m7J8+NNtApEN8nZFnf3bhehZW7AxmS1m0
ZnSsdHw0Fw+bgixPg2MQ9k9oefFeqa+7Kqdlz5bbrUYV2volxhDFtnI4Mh8BiWCN
xDH1Hizq+GKCcHsinDZWurCqder/afJBnQs+SBSL6MVApHt+d35zjBD92fO2Je56
dhMfzCgOKXeJ340WhW3TjD1zqLZXeaCyUNRnfOmWZV8nEhtHOFbUCU7r/KkjMZO9
AgMBAAGjgeMwgeAwDgYDVR0PAQH/BAQDAgGGMBIGA1UdEwEB/wQIMAYBAf8CAQAw
HQYDVR0OBBYEFMDMA0a5WCDMXHJw8+EuyyCm9Wg6MHoGCCsGAQUFBwEBBG4wbDA0
BggrBgEFBQcwAYYoaHR0cDovL29jc3Auc3RnLXJvb3QteDEubGV0c2VuY3J5cHQu
b3JnLzA0BggrBgEFBQcwAoYoaHR0cDovL2NlcnQuc3RnLXJvb3QteDEubGV0c2Vu
Y3J5cHQub3JnLzAfBgNVHSMEGDAWgBTBJnSkikSg5vogKNhcI5pFiBh54DANBgkq
hkiG9w0BAQsFAAOCAgEABYSu4Il+fI0MYU42OTmEj+1HqQ5DvyAeyCA6sGuZdwjF
UGeVOv3NnLyfofuUOjEbY5irFCDtnv+0ckukUZN9lz4Q2YjWGUpW4TTu3ieTsaC9
AFvCSgNHJyWSVtWvB5XDxsqawl1KzHzzwr132bF2rtGtazSqVqK9E07sGHMCf+zp
DQVDVVGtqZPHwX3KqUtefE621b8RI6VCl4oD30Olf8pjuzG4JKBFRFclzLRjo/h7
IkkfjZ8wDa7faOjVXx6n+eUQ29cIMCzr8/rNWHS9pYGGQKJiY2xmVC9h12H99Xyf
zWE9vb5zKP3MVG6neX1hSdo7PEAb9fqRhHkqVsqUvJlIRmvXvVKTwNCP3eCjRCCI
PTAvjV+4ni786iXwwFYNz8l3PmPLCyQXWGohnJ8iBm+5nk7O2ynaPVW0U2W+pt2w
SVuvdDM5zGv2f9ltNWUiYZHJ1mmO97jSY/6YfdOUH66iRtQtDkHBRdkNBsMbD+Em
2TgBldtHNSJBfB3pm9FblgOcJ0FSWcUDWJ7vO0+NTXlgrRofRT6pVywzxVo6dND0
WzYlTWeUVsO40xJqhgUQRER9YLOLxJ0O6C8i0xFxAMKOtSdodMB3RIwt7RFQ0uyt
n5Z5MqkYhlMI3J1tPRTp1nEt9fyGspBOO05gi148Qasp+3N+svqKomoQglNoAxU=
-----END CERTIFICATE-----

fyi, I’ve tried curl -v https://api.{mydomain}/login_ from a number of different OSes and neworks, so pretty convinced it’s not an issue with my local environment.

What am I missing? :worried: