ProxyTunnel 407 Error when using ACME certificate

Hello,

I followed the getting started guide to build an openBalena server at aws using an “off the shelf” Ubuntu image. I successfully managed to setup proxytunnel on my machine to get ssh access to my devices both for development and production balenaOS images. Happy Dayz!

Given this success I thought I’d have a crack at setting up a service that was a bit more robust and in line with my estate policies so I pulled the test environment down and started again with my own Centos7 image. With a bit of trial and error I’ve got to a point where the service is up and running, this time using the ACME cert provider service. I can connect and query devices using the CLI so this is all good.

The only thing I can’t get working is the ssh tunnelling (either to a dev or a prod balenaOS image). Specifically, when I try “ssh root@.balena” I get the standard “Via vpn.<mydomain.com>:3128 -> .balena:22222” message and then it throws:

“HTTP return code: 407 Proxy Authorization Required”

Looking in the vpn service logs on the openBalena server side it’s saying the device is unreachable but I can see earlier in the log the device connected and authenticated successfully. I’ve tried turning off firewalls opening up security groups to no avail.

Can anyone give me any pointers for where to look next?

Hi, can you check if the device is online via api request or cli?

Thanks @karaxuna,

Yes, I’ve got 3 registered devices on the server at the moment and they all show as online through the CLI. One of them has a dev image on it so I can ssh locally and when I block the openBalena ports on the server it drops connection, when I open the ports again they all reconnect as expected.

Hi @bighair

Great, now can you please try the following on the command line:

balena tunnel {uuid} -p 22222:22222

replacing {uuid} with the UUID of your device, to make your localhost:22222 tunnel through to the device’s port 22222 (this is the port SSH runs on). Then, try

ssh -p 22222 -l root 127.0.0.1

to see if SSH connects. Please let us know if it works!

Kind regards
Alida

Thanks Alida,

I think we’re getting somewhere. Across all three devices (2 prod, 1 dev) I get the following error when trying the tunnel steps you’ve suggested:

[Error] 127.0.0.1:63888 => 127.0.0.1:22222 ===> 10.240.0.3:22222 :: Device is not listening on port 22222

Does this suggest I need to do something with the image before flashing it to the SD card?

I took this error and have tested out a few more things:

  • netstat -ant | grep 22222 from one balena device shows:

tcp 0 0 :::22222 :::* LISTEN
tcp 0 0 ::ffff:192.168.3.23:22222 ::ffff:192.168.5.7:65370 ESTABLISHED

  • telnet <mydomain.com> 3128 from my laptop is successful (i.e. it connects and I get a telnet terminal prompt)

  • telnet <mydomain.com> 3128 from one of my balena devices doesn’t seem to work at all (i.e. it doesn’t connect and I don’t get a telnet terminal prompt)

This suggest to me that my devices are listening on port 22222 but for some reason they can’t access port 3128 on my server. Although my laptop can even when I have it on the same network as the devices. The only wrinkle is that I can’t see any errors when I query journalctl:

  • journalctl -au openvpn (all looks good and certainly the data channels appear to be up)
  • journalctl -au resin-supervisor (again, no issues)

OK, so I’ve done more digging and it turns out that certainly one of my problems was the firewall on my server host machine. I could ping www.google.com from the host but not from any of the openBalena containers.

Turning the firewall off systemctl stop firewalld on the host server and restarting Docker and openBalena proved it. I then ran the below to make the changes to the host server firewall that allows it to run whilst still letting the container traffic out:

  • firewall-cmd --permanent --zone=trusted --change-interface=docker0

I did need to restart the whole server again after the above command, but I’m not sure that’s fully necessary. You would definitely need to restart Docker and the openBalena containers though.

After making the above change I could successfully tunnel using the balena tunnel {uuid} -p 22222:22222 command. However, the proxytunnel approach still doesn’t work and I get the same 407 error as in my first post?

Can you try the tunnel approach with the dev device first? I suspect the production ones won’t let you connect unless you add a key to them.

Hi there,

This was completely my bad! I’d setup proxytunnel as per the instructions here HowTo: SSH into host device for my proof of concept host server / service. I hadn’t re-run the balena api-key generate proxytunnel command and so I was still using the old credentials in my ~/.ssh/balena-ssh file.

I re-ran the api-key generate command, applied the new key and I’m back in business. Thanks so much for all your help.

Thanks for getting back to us, and we’re glad to hear you’ve sorted it out!

1 Like