Setup resin.io behind corporate proxy/firewall

[SimonTrockel]

Hi everybody,

I’m currently trying to run resin io on a Raspi3 behind our corporate network security infrastructure and can’t seem to get it to work.

What have I done so far:
Download the image from the dashboard: Filename was resin-Monitoring-2.7.8+rev1-dev-v6.4.2.img

Setup system proxy settings with redsocks.conf according to Proxy-Settings for Static IP Configuration and https://docs.resin.io/deployment/network/2.x/
no_proxy:

10.0.0.0/8
172.16.0.0/12
192.168.0.0/16

redsocks.conf:

base {
log_debug = off;
log_info = on;
log= "syslog:local7";
daemon = off;
redirector=iptables;
}
redsocks {
type=http-connect; // I tried socks as well
ip =  proxy.mycompany.com;
port = 80;
login = "username";
password = "password";
local_ip= 127.0.0.1;
local_port = 12345;
}

Additionally I setup the networking with a resin-eth file in system-connections so:

[ipv4]
method=auto
ignore-auto-dns=true
dns=<internal DNS IP>

Also I added the following in config.json:

"ntpServers":"time.mycompany.com",
"dnsServers":"<internal DNS IP>"

Everything else in the image is unchanged.

When I start up I get no connection to the resin io dashboard. resin local ssh <device-ip> works as well as resin local logs <device-ip>
I see 2 kind of errors in the logs. Once this one:

Event: Device bootstrap failed, retrying {"delay":30000, 
"error":{"message":"connect ECONNREFUSED 54.164.253.112:443",
"stack":"Error: connect ECONNREFUSED 54.164.253.112:443
at Object.exports._errnoException (util.js:1026:11)
at exports._exceptionWithHostPort (util.js:1049:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1081:14)"}}

and then recurring this one:

Event: Device bootstrap failed, retrying {"delay":30000, 
"error":{"message":"socket hang up",
"stack":"Error: socket hang up 
at TLSSocket.onHangUp (_tls_wrap.js:1092:19)
at TLSSocket.g (events.js:286:16)
at emitNone (events.js:91:20)
at TLSSocket.emit (events.js:185:7)
at endReadableNT (_stream_readable.js:974:12)
at _combinedTickCallback (internal/process/next_tick.js:74.11)
at process._tickCallback (internal/process/next_tick.js:98:9)"}}

Can anybody help me to get this working?

Additional Info: Despite the above DNS settings I can see in our network monitoring that it is trying to call Google DNS on 8.8.8.8.

Thanks in advance!
Simon

[pcarranzav]

Hi @SimonTrockel!
A few things to try on your configuration:

• For ethernet, I don’t think you need the resin-eth file in system-connections - the OS will handle DNS settings with what you put in config.json. That file would only be needed to set a static IP.
• You don’t need that no_proxy file, since reserved addresses are already filtered by the OS (see https://github.com/resin-os/meta-resin/blob/master/meta-resin-common/recipes-connectivity/resin-proxy-config/resin-proxy-config/resin-proxy-config#L48 ) - the no_proxy is in case there’s any other custom IPs you need to route directly.

Other than that, could you double check if it’s a SOCKS or HTTP proxy, and whether the port is correct? Maybe knowing how you set up other devices/computers on this networks would help.

[SimonTrockel]

Hi @pcarranzav,

thanks for the tips. I removed the resin-eth and no_proxy.

Our proxy is definitely a HTTP proxy and a SOCKS5 proxy. SOCKS5 might require some configuration from network team so I’m going with HTTP. The settings are definitely correct. I can see it connecting to the proxy and receiving an answer in network monitoring.
I now tried http-connect and http-relay. Still the same errors.

Any further ideas?

Thanks

[pcarranzav]

@SimonTrockel maybe we can try a few things on the device to see where the problem lies.
A few things that come to mind are running curl google.com, curl https://google.com, and the same using IPs: curl 54.164.253.112 and curl https://54.164.253.112 - those will give us some idea on whether HTTP / HTTPS / DNS work at all.
Is it possible the proxy also listens on port 443? (Since we’re trying to do HTTPS proxying, it sounds reasonable). If it’s at all possible, I’d also try setting the port option in redsocks.conf to 443.

Also journalctl -u resin-proxy-config and journalctl -u redsocks might give us more information.

[SimonTrockel]

Hi @pcarranzav,

thanks for the tips but I seem to be missing something. When I resin local ssh into the device I only get the supervisor container as a choice and when I try to run the above commands I get:

/usr/src/app # curl google.com
/bin/sh: curl: not found
/usr/src/app # journalctl -u resin-proxy-config
/bin/sh: journalctl: not found

What am I missing? Can I install these?

Sorry for the maybe dumb question.

On the proxy: It definitely only listens on port 80. So all connections even HTTPS or SSH are proxied through port 80.

[Shou]

Hey there, those commands are available on the host OS. You will have to provide the --host flag to resin local ssh as mentioned in the docs1. Hope that helps, let us know!

[SimonTrockel]

Hi again,

so after the weekend I had a chance to test the above commands:

curl google.com works, but google is whitelisted anyway.

curl https://google.com
curl: (35) gnutls_handshake() failed: The TLS connection was non-properly terminated.

Which probably is the firewall denying the direct connection.

journalctl -u resin-proxy-config
resin-proxy-config: Found config.json in /mnt/boot/config.json
iptables v1.6.1 Couldn't load target 'REDSOCKS': No such file or directory

journalctl -u redsocks
HTTP Proxy auth failed: HTTP/1.1 407 Proxy Authentication Required

To me it looks like it can’t find the redsocks config and therefore does not have the proxy authentication info.

Any help would be appreciated!

Regards,
Simon

[SimonTrockel]

Hi everybody,

has anybody an idea? Do you need more information?

Thanks and Regards,
Simon

[jgentes]

I’m in the same boat - I’ve tried all types of configuration options to no avail. As soon as I add the redsocks.conf file, the device never connects to the server. I’ve monitored my proxy traffic and it doesn’t seem to be making a connection (whereas other devices are connecting through it fine) so I think something is up with the code, but I’m not sure how to resolve it.

[pcarranzav]

Hi @SimonTrockel - the fact that the error is “Proxy Authentication Required” means that the device has definitely found the redsocks config file, but for some reason it’s not able to authenticate using those credentials. This makes me think the proxy might not use one of the schemes supported by redsocks (Basic or Digest). Do you know what kind of proxy is used by the network, and what authentication schemes it supports?

@jgentes could you also check the logs to see if you get the same error or something different? (journalctl -u redsocks and journalctl -u resin-proxy-config)

[jgentes]

Thanks @pcarranzav that was very helpful - I found this error when I looked at the log for resin-proxy-config:

Starting Resin proxy configuration service...
resin-proxy-config: Found config.json in /mnt/boot/config.json
iptables v1.6.1: Couldn't load target 'REDSOCKS': no such file or directory
iptables: No chain/target/match by that name.

I believe I’ve got the redsocks file loaded properly:

Although when I look at the /resin-boot directory on the actual resin box after I’ve provisioned it, I do not see the system-proxy folder. Perhaps I’m doing something wrong when modifying the img file? Unfortunately I haven’t been able to get the resin-cli to work while copying files to this directory (see Invalid Volume Signature with resin img) although I can copy over to /splash just fine.

[pcarranzav]

@jgentes indeed your setup looks fine. Those iptables errors are fine too, they don’t mean anything’s wrong. Could you show the output of systemctl status redsocks.service?

Also, there’s a new API endpoint on the supervisor that might help you set up the proxy configuration in an easier way, you just need to interact with these endpoint instead of putting a redsocks.conf: https://github.com/resin-io/resin-supervisor/blob/master/docs/API.md#patch-v1devicehost-config (you’ll need a resinOS version > v2.9.7)

[jgentes]

Sorry for the delay, here’s the result of the systemctl status redsocks.service

[jgentes]

I like the idea of setting the proxy settings programmatically, but I’m concerned that by pushing proxy settings over the wire, if it doesn’t work, the device will be bricked, since it can no longer communicate to the resin servers, so at that point I wouldn’t be able to reverse the process.

Is that true? If so maybe a simple test script with an undo operation would be able to reverse the change if it can’t hit the resin server or something.

[jgentes]

Hi @pcarranzav, I was able to test the API approach, but still no luck. The device does provision properly since I no longer have a redsocks.conf file, and the API update succeeds, but it doesn’t seem to be using the proxy.

When I run systemctl status redsocks.service I see this:

   Loaded: loaded (/lib/systemd/system/redsocks.service; enabled; vendor preset: enabled)
   Active: inactive (dead)
Condition: start [[0;1;33mcondition failed[[0m at Mon 2018-02-26 20:58:54 UTC; 7min ago
           └─ ConditionPathExists=/mnt/boot/system-proxy/redsocks.conf was not met

[jgentes]

Here’s my code for reference:

if (process.env.HTTP_PROXY_HOST) {
        const proxy = {
          type: process.env.HTTP_PROXY_TYPE || 'socks5',
          ip: process.env.HTTP_PROXY_HOST,
          port: process.env.HTTP_PROXY_PORT || 8123
        };
        
        if (process.env.HTTP_PROXY_USERNAME) {
          proxy.login = process.env.HTTP_PROXY_USERNAME;
          proxy.password = process.env.HTTP_PROXY_PASSWORD;
        }
        
        try {
          request({
            uri: process.env.RESIN_SUPERVISOR_ADDRESS + '/v1/device/host-config?apikey=' + process.env.RESIN_SUPERVISOR_API_KEY,
            method: "PATCH",
            headers: {
              'Content-Type': 'application/json'
            },
            data: {
              network: {proxy}
            }
          }, function(e, r, body) {
            if (body) {
              log.info('Proxy Configuration Successful: ', process.env.HTTP_PROXY_HOST);
            } else log.warn('Resin request failed: ', e);
          });
        } catch (e) {
          log.warn('Resin request failed: ', e);
        } 
      }

[jgentes]

Btw here’s the redsocks log and resin-supervisor log as well:

[jgentes]

SOLVED:

Ok, so after all that, it came down to a damn syntax issue. The redsocks.conf file must end with an extra line.

We identified the issue by running these commands on the device:

systemctl stop redsocks
redsocks -c /mnt/boot/system-proxy/redsocks.conf

That second command resulted in an unclosed section error.

Here’s the final redsocks file that worked… note there is an extra line at the end:

base {
log_debug = off;
log_info = on;
log = "stderr";
daemon = off;
redirector = iptables;
}

redsocks {
type = http-connect;
ip = 192.168.1.65;
port = 1080;
local_ip = 127.0.0.1;
local_port = 12345;
}

Still working out why the API approach didn’t work.

[jgentes]

API SOLVED: The issue was with sending the object via the request module. I was using data:

data: {
  network: {proxy}
}

Instead I needed to use json:

json: {
  network: {proxy}
}

It’s unfortunate the API didn’t kick back an error, because the previous call resulted in an ‘OK’ response

[cgavriluta]

I am having the same problem. I installed the development version of resin OS and I configured the proxy through redsocks as indicated.
After booting, I get the “Booted - Check your resin.io dashboard” message, but regardless of what I tried, my device doesn’t show in the dashboard.
From my local network i can access the device through the resin CLI:
sudo resin local ssh 00bc0c9.local --host
Then I can check if the redsocks configuration is correct and the status of the service:

root@00bc0c9:~# cat /mnt/boot/system-proxy/redsocks.conf 
base {
log_debug = off;
log_info = on;
log = "stderr";
daemon = off;
redirector = iptables;
}

redsocks {
type = socks5;
ip = 192.168.1.102;
port = 4192;
local_ip = 127.0.0.1;
local_port = 12345;
}

root@00bc0c9:~# systemctl status redsocks.service             
● redsocks.service - redsocks transparent proxy redirector
   Loaded: loaded (/lib/systemd/system/redsocks.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2018-02-28 06:08:27 UTC; 29min ago
 Main PID: 749 (redsocks)
   CGroup: /system.slice/redsocks.service
           └─749 /usr/bin/redsocks -c /mnt/boot/system-proxy/redsocks.conf

Feb 28 06:08:27 00bc0c9 systemd[1]: Started redsocks transparent proxy redirector.
Feb 28 06:08:27 00bc0c9 redsocks[749]: 1519798107.437677 notice main.c:165 main(...) redsocks started, conn_max=128

However, when trying to run
root@00bc0c9:~# curl https://api.resin.io
it doesn’t do anything.If I add the proxy as an env variable:

root@00bc0c9:~# export http_proxy=192.168.1.102:4192
root@00bc0c9:~# export https_proxy=192.168.1.102:4192
root@00bc0c9:~# curl https://api.resin.io
returns

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>Cannot GET /</pre>
</body>
</html>

which seems ok. Also, I can see also some activity on the proxy server:
sudo tail -f /var/log/squid/access.log
returns
1520348904.067 948 192.168.1.106 TCP_TUNNEL/200 5486 CONNECT api.resin.io:443 - HIER_DIRECT/52.86.191.185 -

Any ideas as of why the device doesn’t show on the dashboard?

Thanks!

Other outputs:

root@00bc0c9:~# journalctl -u redsocks 
-- Logs begin at Wed 2018-02-28 06:08:21 UTC, end at Wed 2018-02-28 06:44:03 UTC. --
Feb 28 06:08:27 00bc0c9 systemd[1]: Started redsocks transparent proxy redirector.
Feb 28 06:08:27 00bc0c9 redsocks[749]: 1519798107.437677 notice main.c:165 main(...) redsocks started, conn_max=128

root@00bc0c9:~# journalctl -u resin-proxy-config
-- Logs begin at Wed 2018-02-28 06:08:21 UTC, end at Wed 2018-02-28 06:45:23 UTC. --
Feb 28 06:08:26 00bc0c9 systemd[1]: Starting Resin proxy configuration service...
Feb 28 06:08:26 00bc0c9 sh[704]: resin-proxy-config: Found config.json in /mnt/boot/config.json .
Feb 28 06:08:27 00bc0c9 sh[704]: iptables v1.6.1: Couldn't load target `REDSOCKS':No such file or directory
Feb 28 06:08:27 00bc0c9 sh[704]: Try `iptables -h' or 'iptables --help' for more information.
Feb 28 06:08:27 00bc0c9 sh[704]: iptables: No chain/target/match by that name.
Feb 28 06:08:27 00bc0c9 sh[704]: iptables: No chain/target/match by that name.
Feb 28 06:08:27 00bc0c9 systemd[1]: Started Resin proxy configuration service.