Connecting to a proxy behind a firewall

Hey all,

we are currently evaluating Balena. The devices (RPi4s) will be rolled out in a very (network) controlled environment.

Since we need to allow specific IPs and have no option to allow wildcard-domains, we are considering the proxy setup as recommended here: Firewall configuration incorrect

Here is what we did to test:

  1. On my MacBook I configured one ethernet port to have a static IP. I then started glider: glider -listen socks5://user1:pass1@:1080 -verbose
  2. On the Balena image we are flashing to the RPi 4 we added two files (see below) to configure static IP, DNS resolution and redsocks. The device is connected via ethernet to my MacBook.

Thus the RPi4 does not have a direct connection to the Internet (on my MacBook I have no forwarding enabled). It can however connect to the proxy.

Connectivity seems to work (running curl icanhazip.com from the Pi is successful).
However we don’t see the device in the dashboard so the OpenVPN connection does not seem to work.

In journalctl I see the following log messages:

Aug 29 10:11:13 be452ad resin-supervisor[1235]: [event]   Event: Device bootstrap {}
Aug 29 10:11:13 be452ad ba6ec9d3d967[791]: [info]    New device detected. Provisioning...
Aug 29 10:11:13 be452ad resin-supervisor[1235]: [info]    New device detected. Provisioning...
Aug 29 10:11:13 be452ad resin-supervisor[1235]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Aug 29 10:11:13 be452ad ba6ec9d3d967[791]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Aug 29 10:11:19 be452ad os-config[786]: Awaiting service configuration...
Aug 29 10:11:29 be452ad os-config[786]: Awaiting service configuration...
Aug 29 10:11:40 be452ad os-config[786]: Awaiting service configuration...
Aug 29 10:11:42 be452ad systemd[1]: Created slice system-sshd.slice.
Aug 29 10:11:42 be452ad systemd[1]: Starting OpenSSH Per-Connection Daemon (192.168.1.1:64200)...
Aug 29 10:11:42 be452ad systemd[1]: Started OpenSSH Per-Connection Daemon (192.168.1.1:64200).
Aug 29 10:11:43 be452ad ba6ec9d3d967[791]: [event]   Event: Device bootstrap {}
Aug 29 10:11:43 be452ad resin-supervisor[1235]: [event]   Event: Device bootstrap {}
Aug 29 10:11:43 be452ad ba6ec9d3d967[791]: [info]    New device detected. Provisioning...
Aug 29 10:11:43 be452ad resin-supervisor[1235]: [info]    New device detected. Provisioning...
Aug 29 10:11:44 be452ad ba6ec9d3d967[791]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}
Aug 29 10:11:44 be452ad resin-supervisor[1235]: [event]   Event: Device bootstrap failed, retrying {"delay":30000,"error":{"message":""}}

In the proxy log, I see that it is trying to connect:

2019/10/29 11:47:47 socks5.go:145: [socks5] 192.168.1.2:42302 <-> 8.8.8.8:53 via DIRECT
2019/10/29 11:47:47 socks5.go:145: [socks5] 192.168.1.2:42306 <-> 3.215.161.68:443 via DIRECT
2019/10/29 11:47:57 socks5.go:145: [socks5] 192.168.1.2:42310 <-> 8.8.8.8:53 via DIRECT
2019/10/29 11:47:57 socks5.go:145: [socks5] 192.168.1.2:42314 <-> 34.200.210.179:443 via DIRECT
2019/10/29 11:48:07 socks5.go:145: [socks5] 192.168.1.2:42318 <-> 8.8.8.8:53 via DIRECT
2019/10/29 11:48:07 socks5.go:145: [socks5] 192.168.1.2:42322 <-> 3.215.161.68:443 via DIRECT
2019/10/29 11:48:10 socks5.go:145: [socks5] 192.168.1.2:42326 <-> 8.8.8.8:53 via DIRECT
2019/10/29 11:48:10 socks5.go:145: [socks5] 192.168.1.2:42330 <-> 34.200.210.179:443 via DIRECT
2019/10/29 11:48:18 socks5.go:145: [socks5] 192.168.1.2:42334 <-> 8.8.8.8:53 via DIRECT
2019/10/29 11:48:18 socks5.go:145: [socks5] 192.168.1.2:42338 <-> 34.200.210.179:443 via DIRECT

The content of /system-connections/test

[connection]
id=test
type=ethernet
interface-name=eth0
permissions=
secondaries=

[ethernet]
mac-address-blacklist=

[ipv4]
address1=192.168.1.2/24,192.168.1.1
dns="127.0.0.1#5313"
dns-search=
method=manual

[ipv6]
addr-gen-mode=stable-privacy
dns-search=
method=auto

The content of /system-proxy/redsocks.conf:

base {
log_debug = off;
log_info = on;
log = "syslog:local7";
daemon = off;
redirector = iptables;
}

redsocks {
type = socks5;
ip = 192.168.1.1;
login = user1;
password = pass1;
port = 1080;
local_ip = 127.0.0.1;
local_port = 12345;
}

dnsu2t {
local_ip = 127.0.0.1;
local_port = 5313;

remote_ip = 8.8.8.8;
remote_port = 53;
}

Where do we go wrong?

Thanks
Michael

Hi,
Could you check the output of journalctl -u openvpn and see if you can find any relevant errors?
Kind regards,
Theodor

@telphan Unfortunately, there are none:

root@4316ae0:~# journalctl -u openvpn
-- Logs begin at Mon 2019-08-26 09:19:56 UTC, end at Thu 2019-08-29 10:18:49 UTC. --
Aug 29 10:10:17 4316ae0 systemd[1]: Condition check resulted in OpenVPN being skipped.
Aug 29 10:10:41 4316ae0 systemd[1]: Condition check resulted in OpenVPN being skipped.

I traced down the APIs the Pi is trying to connect to and that gave me a new lead:

As mentioned above another SSL domain works well:

root@4316ae0:~# curl https://icanhazip.com
37.24.14.114

But not https://api.balena-cloud.com:

root@4316ae0:~# curl https://api.balena-cloud.com
curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Running curl https://api.balena-cloud.com from my MacBook directly works fine.

I am currently trying to figure out how the proxy would mess up a SSL connection (especially only in that one case).

Thanks
Michael

As expected ignoring certificate warnings in curl works:

root@4316ae0:~# curl https://api.balena-cloud.com -k           
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>Cannot GET /</pre>
</body>
</html>

Hey @mschae , that’s interesting. Is there a DPI system on your device network?
Can you share the output of openssl s_client -showcerts -servername api.balena-cloud.com -connect api.balena-cloud.com:443 | openssl x509 -inform pem -noout -text from the device? (and maybe openssl s_client -showcerts -servername icanhazip.com -connect icanhazip.com:443 | openssl x509 -inform pem -noout -text too) ? Thanks!

root@4316ae0:~# openssl s_client -showcerts -servername api.balena-cloud.com -connect api.balena-cloud.com:443 | openssl x509 -inform pem -noout -text
depth=2 C = US, O = Amazon, CN = Amazon Root CA 1
verify return:1
depth=1 C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
verify return:1
depth=0 CN = balena.io
verify error:num=9:certificate is not yet valid
notBefore=Sep 27 00:00:00 2019 GMT
verify return:1
depth=0 CN = balena.io
notBefore=Sep 27 00:00:00 2019 GMT
verify return:1
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            09:9f:b9:ee:39:ac:b9:48:c6:4d:1e:50:cf:3c:cf:fd
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
        Validity
            Not Before: Sep 27 00:00:00 2019 GMT
            Not After : Oct 27 12:00:00 2020 GMT
        Subject: CN = balena.io
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:83:02:69:77:3e:41:68:8d:81:ee:01:aa:1d:58:
                    36:8b:6e:2b:b2:a5:d4:e6:f7:2b:09:85:b3:fb:54:
                    e8:a6:83:38:cf:26:2d:9a:51:19:7c:49:2c:f3:4d:
                    a5:c7:d6:41:b7:b3:dd:ba:26:a1:8e:06:ff:a6:ee:
                    fb:f6:22:02:fd:0b:19:c0:7d:f6:f7:25:7b:7e:7c:
                    80:07:50:90:24:4d:ea:61:98:6f:8e:3d:86:b9:96:
                    dd:82:18:42:61:7e:3f:89:e7:de:f1:1f:cf:c3:8d:
                    9c:65:bb:85:64:b2:28:3b:ca:ff:72:00:03:45:9d:
                    c2:e6:3e:3e:e9:41:ed:3c:85:2d:e6:14:47:b3:cd:
                    58:ff:3e:24:a4:e4:58:67:ea:da:0d:79:d9:bd:fc:
                    49:1f:c9:4c:7e:f5:fb:3d:eb:6d:3c:f4:c0:d3:e2:
                    3e:fc:5e:08:ca:4f:38:84:c3:98:3b:b9:ce:15:be:
                    15:1a:9a:f0:2f:81:d3:3c:f5:73:f6:8f:28:df:c1:
                    2f:32:e4:61:2b:20:18:c0:74:f2:68:b8:b4:04:a8:
                    7b:51:15:a7:f9:78:3e:3a:0c:d0:02:0f:e7:d8:32:
                    2b:5a:14:cd:13:86:d9:65:1d:ec:8a:e9:e7:bb:e4:
                    df:41:88:65:9b:75:71:53:49:35:3c:7b:1a:88:4e:
                    ae:f7
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Authority Key Identifier: 
                keyid:59:A4:66:06:52:A0:7B:95:92:3C:A3:94:07:27:96:74:5B:F9:3D:D0

            X509v3 Subject Key Identifier: 
                9A:42:D2:5C:1C:51:E3:0F:55:B7:E8:EE:0E:D1:5E:C8:11:4F:51:3B
            X509v3 Subject Alternative Name: 
                DNS:balena.io, DNS:*.balena.io, DNS:resin.io, DNS:*.resin.io, DNS:resindevice.io, DNS:*.resindevice.io, DNS:balena-cloud.com, DNS:*.balena-cloud
.com, DNS:balena-devices.com, DNS:*.balena-devices.com
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 CRL Distribution Points: 

                Full Name:
                  URI:http://crl.sca1b.amazontrust.com/sca1b.crl

            X509v3 Certificate Policies: 
                Policy: 2.16.840.1.114412.1.2
                Policy: 2.23.140.1.2.1

            Authority Information Access: 
                OCSP - URI:http://ocsp.sca1b.amazontrust.com
                CA Issuers - URI:http://crt.sca1b.amazontrust.com/sca1b.crt

            X509v3 Basic Constraints: critical
                CA:FALSE
            CT Precertificate SCTs: 
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : BB:D9:DF:BC:1F:8A:71:B5:93:94:23:97:AA:92:7B:47:
                                38:57:95:0A:AB:52:E8:1A:90:96:64:36:8E:1E:D1:85
                    Timestamp : Sep 27 01:25:38.815 2019 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:44:02:20:44:73:AB:60:F2:3A:C3:4D:79:78:79:70:
                                7E:75:53:EB:A8:3D:B7:B1:D6:36:78:46:79:8B:2D:4A:
                                A3:D6:E3:70:02:20:6A:81:FB:DC:B2:C3:C0:AF:5A:8A:
                                7C:B2:4B:A1:12:EA:86:A4:8F:56:4B:18:D4:2E:AE:F2:
                                28:FA:19:4F:2A:8A
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : 87:75:BF:E7:59:7C:F8:8C:43:99:5F:BD:F3:6E:FF:56:
                                8D:47:56:36:FF:4A:B5:60:C1:B4:EA:FF:5E:A0:83:0F
                    Timestamp : Sep 27 01:25:38.892 2019 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:44:02:20:3E:76:01:7D:40:89:4D:BD:E7:0C:2F:62:
                                A2:30:A0:30:98:FC:B3:B6:8D:F8:2A:CE:3A:89:CA:78:
                                9A:90:C8:F3:02:20:0D:E0:95:3A:3A:39:EE:3A:60:9C:
                                3B:3B:D6:DF:73:87:5C:0E:EE:56:9E:13:37:2F:57:57:
                                67:C7:BC:22:6A:AC
    Signature Algorithm: sha256WithRSAEncryption
         51:d9:5f:3c:b7:21:38:e9:3c:74:b0:73:e3:ef:60:4e:7d:20:
         e1:71:3d:01:56:f3:fe:09:30:b4:74:5b:1f:e3:6a:96:bb:88:
         2f:0e:a1:c2:6e:18:3f:af:e7:98:d9:1d:f0:d3:63:5c:a1:36:
         11:30:9c:83:21:8f:f9:9b:13:6c:25:9a:8b:2f:07:0e:e7:ae:
         8d:27:38:e5:42:24:75:40:35:a0:da:5e:c0:2a:9f:d3:a2:84:
         0b:c0:ca:67:28:89:11:fb:d5:13:48:d5:7e:81:91:f7:ed:81:
         0b:b8:88:f4:75:dc:08:19:7a:c7:3c:af:a5:53:ec:26:d3:f5:
         40:51:25:a7:0c:86:e1:d6:20:bd:84:05:c9:14:d9:96:b3:6d:
         83:32:d2:5a:ae:bf:5a:cc:37:79:e7:bd:3a:38:e9:42:a0:6d:
         3d:3f:6d:c9:8e:73:5d:10:f6:88:1f:ab:02:aa:34:8a:41:2c:
         e6:00:2e:27:50:7c:e8:29:bc:0f:1e:10:60:2c:4b:7c:96:2f:
         72:60:34:a3:60:3c:1b:a7:1c:82:10:83:b7:d9:6b:5b:17:06:
         4e:fc:7f:47:33:f3:b7:41:a7:94:56:b1:a6:14:33:d6:64:07:
         66:65:71:c8:70:56:24:8b:93:60:ee:12:fe:48:c4:65:35:fb:
         6f:46:44:26
root@4316ae0:~# openssl s_client -showcerts -servername icanhazip.com -connect icanhazip.com:443 | openssl x509 -inform pem -noout -text
depth=2 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO ECC Certification Authority
verify return:1
depth=1 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO ECC Domain Validation Secure Server CA 2
verify return:1
depth=0 OU = Domain Control Validated, OU = PositiveSSL Multi-Domain, CN = ssl514403.cloudflaressl.com
verify return:1
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            bc:40:1e:25:06:3a:7d:34:f5:06:b9:14:13:1d:02:35
        Signature Algorithm: ecdsa-with-SHA256
        Issuer: C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO ECC Domain Validation Secure Server CA 2
        Validity
            Not Before: Aug 24 00:00:00 2019 GMT
            Not After : Mar  1 23:59:59 2020 GMT
        Subject: OU = Domain Control Validated, OU = PositiveSSL Multi-Domain, CN = ssl514403.cloudflaressl.com
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:7e:9d:e6:a2:6f:fd:46:f1:6a:98:3c:1f:89:9c:
                    97:dd:93:75:b0:c6:ac:5a:ed:d0:4b:7a:71:d7:b9:
                    d1:27:02:4f:9c:cb:ae:ec:85:76:b8:30:ae:3b:85:
                    32:fe:17:3b:26:a2:46:b6:28:e5:69:09:03:0c:a3:
                    64:16:34:52:5a
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Authority Key Identifier: 
                keyid:40:09:61:67:F0:BC:83:71:4F:DE:12:08:2C:6F:D4:D4:2B:76:3D:96

            X509v3 Subject Key Identifier: 
                08:73:D1:DF:56:5A:34:C3:BE:9E:C5:97:F2:94:D0:DD:82:DC:99:C8
            X509v3 Key Usage: critical
                Digital Signature
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Certificate Policies: 
                Policy: 1.3.6.1.4.1.6449.1.2.2.7
                  CPS: https://secure.comodo.com/CPS
                Policy: 2.23.140.1.2.1

            X509v3 CRL Distribution Points: 

                Full Name:
                  URI:http://crl.comodoca4.com/COMODOECCDomainValidationSecureServerCA2.crl

            Authority Information Access: 
                CA Issuers - URI:http://crt.comodoca4.com/COMODOECCDomainValidationSecureServerCA2.crt
                OCSP - URI:http://ocsp.comodoca4.com

            X509v3 Subject Alternative Name: 
                DNS:ssl514403.cloudflaressl.com, DNS:*.icanhazip.com, DNS:icanhazip.com
            CT Precertificate SCTs: 
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : B2:1E:05:CC:8B:A2:CD:8A:20:4E:87:66:F9:2B:B9:8A:
                                25:20:67:6B:DA:FA:70:E7:B2:49:53:2D:EF:8B:90:5E
                    Timestamp : Aug 24 01:19:13.193 2019 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:46:02:21:00:B5:37:3E:68:F6:91:22:2D:81:88:86:
                                D6:D0:8D:28:67:FA:F9:8E:38:C6:04:43:82:12:B7:BA:
                                B2:70:01:70:E0:02:21:00:92:16:8C:5C:AD:6C:2B:64:
                                AB:54:4F:44:37:78:07:8E:A0:95:91:64:E6:5F:D4:A2:
                                2D:82:69:A9:54:80:1B:2A
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : 5E:A7:73:F9:DF:56:C0:E7:B5:36:48:7D:D0:49:E0:32:
                                7A:91:9A:0C:84:A1:12:12:84:18:75:96:81:71:45:58
                    Timestamp : Aug 24 01:19:13.255 2019 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:45:02:21:00:CB:FD:27:55:EB:BF:55:BA:62:33:08:
                                DB:45:B8:92:EA:D1:F3:7D:9A:91:81:37:ED:F6:8A:AC:
                                D4:D7:FE:D9:90:02:20:13:73:5B:D8:E4:8A:AC:9E:AB:
                                C1:96:12:71:0C:03:E9:2E:D3:31:16:78:A5:2E:05:CA:
                                7D:25:46:D8:BE:A6:FA
    Signature Algorithm: ecdsa-with-SHA256
         30:45:02:20:6c:74:0b:98:1a:3f:19:fa:29:70:3f:49:e3:79:
         f5:67:7c:f8:ee:5b:4e:10:81:b7:13:a6:c0:b6:6d:73:d0:a6:
         02:21:00:b5:78:46:fd:62:b3:b6:49:4d:83:5c:6a:82:64:ad:
         75:6e:31:2d:ef:ba:ad:41:7b:90:09:b8:6d:1a:0e:75:50

Looks like the clock on the RPi is wrong… Doesn’t ntp pick up the redsocks configuration?

root@4316ae0:~# date
Thu Aug 29 10:46:18 UTC 2019

That explains the certificate error: It’s not yet valid.

FWIW:

root@4316ae0:~# systemctl status chronyd
● chronyd.service - NTP client/server
   Loaded: loaded (/lib/systemd/system/chronyd.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/chronyd.service.d
           └─chronyd.conf
   Active: active (running) since Thu 2019-08-29 10:10:15 UTC; 35min ago
     Docs: man:chronyd(8)
           man:chrony.conf(5)
 Main PID: 847 (chronyd)
   Memory: 1.2M
   CGroup: /system.slice/chronyd.service
           └─847 /usr/sbin/chronyd -s -d

I can see the error is time related here. Can you compare the time reported on your device? It appears there is a time drift, resulting in a certificate is not yet valid error. The cause on such an issue, would probably be an unmet network requirement. More info on our Network Requirements may be found here: https://www.balena.io/docs/reference/OS/network/2.x/#network-requirements.
You can set the time quite close to the correct current time from host OS as a workaround: date -s "$(curl --silent --head --insecure https://api.balena-cloud.com/ping | grep -i Date: | sed 's/[Dd]ate: //')"
But you need to ensure access to 123 UDP is whitelisted For NTP time synchronisation.
Best regards!