I have a balena device connected to some TCP server on port 5555 which I do not want to expose. However, for debugging purposes it would be helpful to temporarily forward this port via ssh, e.g.
ssh -L 5555:<ip-of-device>:5555 gh_myuser@ssh.balena-devices.com host <device-id>
but that does not seem to work. Am I doing it wrong or is that not supported?
Please note that this is a binary TCP protocol. Not HTTP.
How can I achieve this?
balena tunnel seems to do the trick
Not really since it only allows to tunnel to ports listening on the balena device itself but not to devices connected to or reachable by it.
Usage: tunnel <uuid>
Use this command to open local ports which tunnel to listening ports on your balenaOS device.
For example, you could open port 8080 on your local machine to connect to your managed balenaOS
device running a web server listening on port 3000.
You can tunnel multiple ports at any given time.
Examples:
# map remote port 22222 to localhost:22222
$ balena tunnel abcde12345 -p 22222
# map remote port 22222 to localhost:222
$ balena tunnel abcde12345 -p 22222:222
# map remote port 22222 to any address on your host machine, port 22222
$ balena tunnel abcde12345 -p 22222:0.0.0.0
# map remote port 22222 to any address on your host machine, port 222
$ balena tunnel abcde12345 -p 22222:0.0.0.0:222
# multiple port tunnels can be specified at any one time
$ balena tunnel abcde12345 -p 8080:3000 -p 8081:9000
Options:
--port, -p <port> The mapping of remote to local ports.
You are right, we donāt support port forwarding via proxy. My teammates suggest the following workarounds.
If you are running a development image, you can ssh -L ... directly to the device. Then you can add custom authorized_keys which would also enable direct SSH or SSH onto the device using normal means (balena ssh ā¦) and then use ssh -R to reverse-forward the port back to your machine.
Or if the device is not local to you, you can use balena tunnel to get port 22222 and then SSH tunnel over that (depending on SSH access via key/password)
Adding to @gelbalās answer, perhaps covering a slightly different use case that may be useful to other users reading this thread:
The standard ssh and scp tools can be used with balenaOS devices, including standard ssh port forwarding. It works with both development and production balenaOS images. Production images require a ssh key to be added to the config.json file ā see sshKeys section of the meta-balena README file. Development images do not require ssh keys to be added.
The ssh server on a device listens on TCP port number 22222. This port is not blocked by the device host OS, not even in production. If this port is blocked by a firewall or router on the deviceās local network, the balena tunnel command can be used as in the examples below.
Example 1 (port 22222 not blocked in the deviceās local network)
# all commands below to be run your laptop/desktop computer
ssh -L 1111:www.google.com:443 -p 22222 root@<deviceIpAddressOrHostname>
curl -ik https://127.0.0.1:1111
In the example above, the ssh command connects to the balenaOS device (ssh server on port 22222). It also opens a listening socket on port 1111 on your laptop/desktop computer. When curl opens a connection to 127.0.0.1:1111, the data is tunnelled to the balenaOS device. The device opens a connection to www.google.com:443 āon demandā, and traffic is forwarded between Google and curl on your laptop/desktop via the balenaOS device.
Example 2 (port 22222 is blocked in the deviceās local network)
# all commands below to be run your laptop/desktop computer
balena tunnel <deviceUUID> -p 22222:4321
ssh -L 1111:www.google.com:443 -p 4321 root@127.0.0.1
curl -ik https://127.0.0.1:1111
The balena tunnel command above is used to create a tunnel between port 22222 on the balenaOS device (the port that the ssh server listens on) and port 4321 on your laptop/desktop computer. What follows is similar to example 1, except that, instead of ssh connecting directly to the deviceās IP address or hostname, ssh connects to port 4321 on the laptop/desktop, which is tunnelled to the deviceās ssh server. The rest is the same as in the first example.
I tried to use this technique and got an error. In one terminal, I ran:
$ balena tunnel <deviceUUID> -p 22222:22222
And then in the second I ran:
$ ssh -p 22222 root@127.0.0.1
ssh_exchange_identification: Connection closed by remote host
And back in the first terminal I see the error:
[Info] Tunnel to <deviceUUID>
[Info] - tunnelling localhost:22222 to device:22222
[Info] Waiting for connections...
[Error] 127.0.0.1:55962 => 127.0.0.1:22222 ===> 10.240.36.7:22222 :: Unable to connect: 404 Not Found
Running balena ssh <deviceUUID> works as expected.
I also tried tunneling port 80 (which does work over the āpublic device URLā) and got the same error.
What am I doing wrong?
OK, I figured that one out! balena ssh works fine with an abbreviated device UUID, but tunnel requires the full UUID. I can now tunnel ports 22222 and 80.
I have an ssh server running on port 33333 inside my application container. The following sequence works:
$ balena tunnel <full deviceUUID> -p 22222:2222 &
$ ssh -p 2222 root@127.0.0.1
root@<abbreviated UUID>:~# ssh -p 33333 user@localhost
user@<abbreviated UUID>:~$ # This is inside my container!
But, trying to connect directly does not work:
$ balena tunnel <full deviceUUID> -p 33333:2222
In a second terminal:
$ ssh -p 2222 user@127.0.0.1
back in the first terminal, I get
[Info] Tunnel to 99f737f27a3b2a01ff93ec0f62ad7a3f
[Info] - tunnelling localhost:2222 to device:33333
[Info] Waiting for connections...
[Error] 127.0.0.1:52596 => 127.0.0.1:2222 ===> 10.240.19.98:33333 :: Unable to connect: 407 Proxy Authorization Required
Is this one just as silly an error?
@russel, thanks for reporting these errors, and sharing the reason you found for the ā404 Not Foundā error. Itās certainly not intuitive that it means āuse the full UUIDā. 
About your ssh server running on port 33333 in an app container, the first thing that comes to my mind is whether you have a multicontainer app with a docker-compose file. If you do, double check whether the docker-compose file has an expose or ports instruction covering port 33333. Check this sample file and the docker-compose reference for expose and ports. (Unless your network_mode is host, then expose/ports are not needed.)
Something else I thought I would mention, just in caseā¦ If the objective was simply to open a ssh shell in the app container, the following commands would do it, without the need of running a ssh server in the app container:
# both commands to be run on your laptop/desktop
$ balena tunnel <long_UUID> -p 22222:4321
$ ssh -tp 4321 root@127.0.0.1 balena-engine exec -it main_1_1 /bin/sh
Replacing āmain_1_1ā with the name of the app container you want to connect to. Note how the ssh command options above include -t, which tells ssh to āallocate a pseudo-terminalā. The "balena-engine exec -it main_1_1 /bin/sh" part is a command that ssh will execute on the host OS. That command opens a shell on a running container via balenaEngine (like Docker does), using simple stdin/stdout pipes rather than āssh over HTTPā.
I am not using a multicontainer app so I do not have a docker-compose file. I think this means all ports are exposed by default. When I am on the same network as the devices, I can ssh -p 3333 user@deviceIP so I believe the connection through the tunnel should be possible.
The balena-engine trick is neat, but what I really want to be able to do is use scp. Using balena tunnel to get to the Host OS:
$ balena tunnel <long_UUID> -p 22222:2222
and then adding these lines to my .ssh/config works:
Host balena-device
User user
HostName localhost
ProxyCommand ssh -p 2222 root@127.0.0.1 nc %h %p
But if I could tunnel the local 2222 directly to the device 33333 then the ProxyCommand would not be necessary.
@russel, probably just a typo, but Iāve noticed you wrote that ssh -p 3333 user@deviceIP works when on the same network as the device ā that port number has 4 digits, whereas the previous balena tunnel command was forwarding port 33333 (5 digits, an extra ā3ā).
I have another theory of what could be going wrong. The balena tunnel output shows that, on the host OS, a connection is attempted to 10.240.19.98:33333. If you run netstat -ant on the host OS (and in your app container), what is the IP address that the 33333 ssh server is bound to / listening on? If it is something like āx.x.x.x:33333ā, where āx.x.x.xā is neither ā0.0.0.0ā nor ā10.240.19.98ā, then that may be the problem. If thatās the problem, then configuring the ssh server to bind to ā0.0.0.0ā (instead of perhaps 127.0.0.1) should solve it.
About scp: if rsync was an acceptable alternative to scp (it has a lot more features!), it possible to use rsync to transfer files to/from an app container without installing a ssh server in the container. rsync needs to be installed in both the source and destination containers/machines, so both your laptop and the app container. On the app container, itās usually as simple as apt-get update; apt-get install rsync. On MacOS, I think it comes with Xcode, and if not then homebrew. On Windows thereās Cygwin.
It is not necessary to setup rsync as a daemon/server, although rsync offers that possibility. Instead, I suggest using rsync over ssh, which is the most common setup, in which case rsync has a similar basic command line as scp:
rsync source_folder root@deviceIp:destination_folder
The trick to avoid the extra ssh server is to get rsync to use balena-engine exec behind the scenes, and for that I have small shell script.* If that script was saved to, say, /home/user/rsync-shell.sh, then usage goes along the lines of:
# on a laptop terminal:
balena tunnel <long_UUID> -p 22222:4321
# on another laptop terminal:
export RSYNC_RSH="/home/user/rsync-shell.sh" && chmod +x "$RSYNC_RSH"
rsync source_folder main:destination_folder
Where 'main' is any name chosen for the device or app container.
*The rsync-shell.sh script, with hardcoded values for port numbers and container names:
#!/bin/bash
new_args=()
for arg in "$@"; do
if [ "$arg" = "main" ]; then
new_args+=(-p 4321 root@127.0.0.1 balena-engine exec -i main_1_1)
else
new_args+=("$arg")
fi
done
echo original command: ssh "$@" >&2
echo modified command: ssh ${new_args[@]} >&2
ssh ${new_args[@]}
As a matter of fact, a similar script can be used with the scp -S command-line option, in which case itās possible to use scp without installing a ssh server in an app container. But I find it less attractive than rsync, because of all of rsync's additional features ā and the script would need some changes. 
Thank you, this is helpful. The server is bound to 0.0.0.0, although the IP of the server is not 10.240.19.98, that is the IP of the NAT box, so now I am less sure of how any of this is working. That IP in the log messages seems to be the same in the cases where tunneling works and where it doesnāt work.
Just for fun I tried moving the internal ssh server back to port 22, I get the same results there as I do when it is on 33333. I can successfully tunnel 22222, 8080, and 80, but 33333 and 22 both fail, I
The notes about other ways to solve this problem are helpful, I have enough to make it work now and hopefully I will be able to collect enough data to ask a better question about the simple solution along the way.
Just to reiterate - the only way of ssh port forwarding is using balena tunnel, correct?
I am no longer with balena so donāt treat my reply as anything official, 
but have a look at the ssh-uuid project that I put together back then: GitHub - pdcastro/ssh-uuid: Thin ssh / scp wrapper - proof of concept alternative to 'balena ssh'