HowTo: SSH into host device

holg · February 25, 2021, 3:36pm

Can anybody explain how this is done in future? Since v3 port 3128 is closed and I think VPN Connection is done through HTTPS? So how should I configure my proxytunnel for Version >= 3?

klutchell · February 26, 2021, 7:53pm

Can anybody explain how this is done in future? Since v3 port 3128 is closed and I think VPN Connection is done through HTTPS? So how should I configure my proxytunnel for Version >= 3?

Hey just catching up on this old thread. Can you clarify which v3 release you are referring to? You can still use our balena CLI to initiate a tunnel as outlined in edorgeville’s Jul 2019 post above.

isaacchamp · June 4, 2021, 10:09am

I rerun balena api-key generate proxytunnel and update proxy_passwd in the balena-ssh.
Then while I ssh to the device, I got the following message,

error: connect() failed: [111] Connection refused
ssh_exchange_identification: Connection closed by remote host

What should I do?

Thanks
Isaac

isaacchamp · June 8, 2021, 2:59am

Since I pulled the latest version and port 3128 did not expose in compose/services.yml. After modifying this part and restart the HAProxy service, everything works perfectly.

Cheers
Isaac

sguzzo · March 25, 2022, 10:52am

Hey, thanks for the useful guide!

When I send the command ssh root@deviceuuid.balena, I am prompted with Enter local proxy password for user root. What should my password be in this case?

On a secondary note: I have my openBalena server in a closed infrastructure and I need to access it by first accessing to a bastion. I usually do this by configuring my .ssh/config file as

Host my-balena-server
  HostName my.domain.somewhere
  User balena
  ProxyJump my-bastion-host

Do you think I should just add the line ProxyJump my-bastion-host to the balena ssh host’s config?

Thanks a ton!!

bkaindl · September 15, 2022, 6:36pm

Updated Info: Setup for baleanaOS Device SSH access

This is an update for the guide for BalenaOS SSH access without “balena ssh”:

What is changed since the post @ the start of this thead:

@richbayliss missed to add :22222 for SSH instead of the default SSH port 22.
Port 3128 of “tunnel.$BALENARC_BALENA_URL” was closed: Use TLS on port 443:
proxytunnel --encrypt-proxy --proxy tunnel.$BALENARC_BALENA_URL:443 -d device
The configuration can be simplified a bit by using environment variables instead of a file.
-oHostname=$DEVICE_ADDRESS can be used to make SSH KnownHosts checking work.

Putting it all together:

SSH to Balena HostOSes on openBalena servers (without needing to create any files for the authentication anywhere) can be done this way:

On MacOS, you can use MacPorts or compile it: Install proxytunnel on macOS with MacPorts
On Linux or with Homebrew (other Linux distros and MacOS) use:

sudo apt install proxytunnel || sudo yum install proxytunnel || brew proxytunnel

# You'll use this already for the balena CLI to use your openBalena server,
# For the Balena Cloud, don't set it (balena-cloud.com will be used instead):
BALENARC_BALENA_URL="<your-openbalena-domain.org>"
# And set this:
FULL_DEVICE_UUID=<the long UUID of the device to connect to>

PROXYUSER and PROXYPASS for openBalena users. For Balena Cloud, use your user credentials or a dedicated API key for proxytunnel as mentioned in the thread before:

# For authenticating with tunnel.$BALENARC_BALENA_URL:443 (for proxytunnel):
export PROXYUSER="admin"
export PROXYPASS="$BALENA_API_KEY"  # The API key of the openBalena admin user

Then, the variables and the command for everyone:

# Construct the ProxyCommand which ssh will execute to connect to the Proxy:
PROXY_ADDRESS=tunnel.${BALENARC_BALENA_URL:-balena-cloud.com}:443
DEVICE_ADDRESS="$FULL_DEVICE_UUID.balena:22222"
PROXY_TUNNEL="proxytunnel -z -q -E -p $PROXY_ADDRESS -d $DEVICE_ADDRESS"
ssh -oProxyCommand="$PROXY_TUNNEL" -oHostname=$DEVICE_ADDRESS root@localhost
# -oHostname=$DEVICE_ADDRESS makes ssh use the full destination address for the
# KnownHosts verification instead of localhost which would be the same for all devices.

Edit/Update: Note, I added -z to the flags for proxytunnel to not fail with unable to get local issuer certificate when the openbalena proxy uses a self-signed certificate. See my answer to the next post for more information.

Additional options for specific features:
 -z, --no-check-certificate Don't verify server SSL certificate

It can be converted into code and an alternative “balena ssh” tool can be created based on that.
To be continued…

ahmadalsajid · September 29, 2022, 2:29pm

@bkaindl thanks for the update.

While following your suggestion, I got another error.

Certificate verification failed (unable to get local issuer certificate)
kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535

Any idea how can I solve this? Thanks in advance.

bkaindl · September 29, 2022, 3:08pm

Does “Certificate verification failed (unable to get local issuer certificate)” happen with an openBalena server? I ask because your errors would be consistent with trying to connect to an openBalena instance whose certificate chain could not be fully verified on the host where you started the proxytunnel tool as part of this configuration.

Actually this error is normal (sigh!) when the openBalena instance was installed using the officially documented getting started guide on openBalena - Home where a certificate generated as part of the installation is used which must also be added to the list of trusted certificates on all machines that talk to this openBalena instance.

These are the instruction which mention to add use:
export NODE_EXTRA_CA_CERTS='/path/to/ca.crt' to trust the CA (also generated) of the self-signed certificate:

Did you install this ca.crt file and export the matching NODE_EXTRA_CA_CERTS value to add this CA to be trusted by your client?

Update: Alternatively, you can add -z to the flags for proxytunnel (Edited above with an Edit note).

It would be better if the official quickstart (and openBalena itself) would be updated to use an ACME-enabled reverse proxy like caddy (optionally as a standalone container as a proxy, see caddy-docker-proxy - which gets an official certificate from LetsEncrypt.org or ZeroSSL and have the SSL certificate verification work just out of the box on all clients).

Then, open-baleana should also not have generate put the self-signed certificate into the as rootCA into the config.json files of the balenaOS devices connecting to it, even when it does, it’s not an issue as balenaOS should still trust the public CAs and thus the letsencrypt.org CA.

I use caddy-docker-proxy successfully already to proxy a single 443 SSL port of one server to several independent docker-compose deployments on the same machine for which the caddy proxy gets, provides and renews all SSL certificates officially with letsencypt.org.

ahmadalsajid · October 3, 2022, 7:26am

Hi @bkaindl, you are right. I am pretty new to Balena/OpenBalena and of course to the forum. This issue is driving me crazy. I followed your update and resulted in a new error

Now it says

The authenticity of host 'xxxxxxxxxx.balena:22222 (<no hostip for proxy command>)' can't be established.
ED25519 key fingerprint is SHA256:xxxxx/xxxxxxxxxx.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'xxxxxxxxxx.balena:22222' (ED25519) to the list of known hosts.
root@xxxxxxxxxx.balena:22222: Permission denied (publickey).
select error: Interrupted system call

BTW, I followed Cert Provider - Cannot issue a production certificate · Issue #108 · balena-io/open-balena · GitHub to create the certificate instead of the way it was described in the getting started guide.
Is there anything I am missing in setting up the public key?

bkaindl · October 3, 2022, 9:28am

Hi @ahmadalsajid. the new message show that your openssh client is correctly talking to another openssh server, presumalby your balenaOS device. The line

root@xxxxxxxxxx.balena:22222: Permission denied (publickey).

shows that the attempt to authorize your openssh client using the publickey method was denied by the opensshserver.

If the openssh server is would allow also the password authentication method, the openssh client would have prompted you with Password: before this error. Since it didn’t that means that PasswordAuthentication is turned off on the server (or your openssh client’s config).

You could get more insight what the openssh client and server are negotiating. For this, you could add the flag -vvv to the command line flags you use to run the openssh client. Only the last couple of lines before the Permission denied would be sufficent to take a look at.

As the server in this case can only be the openssh server of one of your openBalena-connected balenaOS devices, that balenaOS device must have have been in production mode. It is the default mode unless explicitly enabled. BalenaOS disables PasswordAuthentication also when ssh key are defined in config.json in order to require key based authentication, of which publickey would be the most common method.

Another way by which it disables PasswordAuthentication besides developmentMode not being true (configured in /mnt/boot/config.json) would be if one ore more ssh keys are defined in in /mnt/boot/config.json in a string array below .os.sshKey. You can check both using:

jq .developmentMode /mnt/boot/config.json
jq .os.sshKeys /mnt/boot/config.json

If you have no login on the device at all, you can try to get offline access to the config.json by mounting the boot partition of the device (e.g. on a Rasperry Pi CM4 based device, you’d use usboot/rpiboot to make the CM4 to become a USB mass storage device and then mount the partition).

When you want to deploy the device into production and want the root login without password, then you’d need to add your ssh public key to the .os.sshKeys array and then use the ssh -i private-key option to use the corresponding private key to login over ssh to your device.

For development, just add developmentMode: true (this also opens e.g. the docker port for localMode development) and remove any .os.sshKeys in your config.json. That should enable PasswordAuthentcation password-less login with an empty root password for development and debugging.

ahmadalsajid · October 3, 2022, 10:06am

Hi @bkaindl , Again thanks for your support.

Finally, I succeeded in accessing my device. What I found, the newer BalenaOS >=v2.85 does not come in two flavors, i.e. development and production, instead, production is the default mode. I was using raspberrypi3-2.105.1+rev1-v14.2.0.img and while using sudo balena scan I saw it was in production mode. That’s why while configuring the image, I used

balena os configure ~/Downloads/raspberrypi3-2.105.1+rev1-v14.2.0.img --fleet myApp --dev

Also, I made a small adjustment in the original answer by @richbayliss from

Host *.balena
  ProxyCommand proxytunnel -p vpn.<your openBalena domain>:3128 -d %h:22222 -F ~/.ssh/balena-ssh
  ServerAliveInterval 30

To

Host *.balena
  ProxyCommand proxytunnel -z -q -E -p tunnel.<your openBalena domain>:443 -d %h:22222 -F ~/.ssh/balena-ssh
  ServerAliveInterval 30

Finally, could access my device by

$ ssh root@<full UUID>.balena

Again, thanks for your help. I really appreciate it.