Balena ssh fails with Permission denied (publickey)

open-balena is used in the backend.
I can login and get information about a specific devices using balena-cli :
$ balena device 79a1dd9
== HIDDEN SKY
ID: 8
DEVICE TYPE: imx6ul-var-dart
STATUS: idle
IS ONLINE: true
APPLICATION NAME: imx6ulApp
UUID: 79a1dd9c1bacdc79661d1afd31e7c52b
COMMIT: fc2e2716d8c8913ae434a0f05f9ed713
SUPERVISOR VERSION: 8.6.3
OS VERSION: balenaOS 2.28.0+rev2
DASHBOARD URL: https://dashboard.mydomain.com/devices/79a1dd9c1bacdc79661d1afd31e7c52b/summary

But when try to ssh to host BalenaOS :
$ balena ssh 79a1dd9 -s
Connecting to: 79a1dd9
root@ssh.devices.mydomain.com: Permission denied (publickey).

Hi @rlev

Use the development version, login and take a look at authorized keys on device:

$ vi ~/.ssh/authorized_keys_remote

It will be empty I’m pretty sure. I have this exact same issue and am trying to work what key should be included and at what point it should be created on the open-balena host, and how it is added to device.

I suspect the authorised pub key that needs to be added is from the open balena host but it is unclear if this needs to be generated manually or exists after executing setup script to create certificates etc.

Hopefully open balena team will be back from holidays soon and be able to shed more light on this.

Cheers
Chris

Just a little further background on this. I have tried adding SSH key with:

$ balena key add Main ~/.ssh/id_rsa.pub
BalenaRequestError: Request error: Unauthorized

Also listing keys doesn’t work with:

$ balena keys
BalenaRequestError: Request error: Internal Server Error

Other cli commands for listing devices, viewing logs, and modifying applications are working with this openBalena instance.

This might be a bigger issue than I thought. Can probably work out how to manually add a key to the right spot, will let you know if I work it out @rlev .

In the meantime this will probably have to wait for a patch along with all of the other cli issues when using openBalena instance.

Cheers
Chris

Thanks for your insights @dash.

I just wanted to say that the following command worked for me on a Mac.

balena key add Main ~/.ssh/id_rsa.pub

1 Like

if you use the production version, you can mount the balena.img with losetup -P
the mount rw the resin-boot volume, add your key to the file config.json.

umount, flash your sd card : profit.

It took me a short while to figure this out.

Having seen activity in this thread… I understand that:

  • The approach described by Pietro (adding a ssh key to the config.json file) would allow “standard ssh” (rather than “balena ssh”) to work. The sshKeys entry in config.json is described on this page: https://github.com/balena-os/meta-balena/#sshkeys
  • The "balena key", "balena keys" and "balena ssh" commands don’t currently work with openBalena. See this other forum thread for further advice on how to ssh to devices on openBalena: HowTo: SSH into host device

all these things are bullshit.
logged in and added keys.
noting works

1 Like

Three things to check if SSH is not working:

  1. Check that the SSH key has been uploaded to balena (Preferences -> SSH keys). Take a copy of the key.
  2. Check that the public key file ~/.ssh/id_rsa.pub exists, and has the same contents as the uploaded key from check 1.
  3. Check that you have a private key file ~/.ssh/id_rsa which corresponds to the public key you checked in check 2.
    This got balena ssh working for me on MacOS. (The key creation command is ssh-keygen -t rsa)
1 Like

Hi,
I trying balena-io-playground/ssh-key-insert: Tooling to insert the relevant SSH keys into balena devices’ configuration. (github.com) but it is driving me nuts.
What key shall be added to

I reckon the public one.

I still get

$ bash -x ./run.sh
+ cat batch
+ stdbuf -oL xargs '-I{}' -P 10 /bin/sh -c 'grep -a -q '\''{} : DONE'\'' sshkey.log || (cat add-ssh-key.sh | balena ssh {} | sed '\''s/^/{} : /'\'' | tee --append sshkey.log)'
...@ssh.balena-devices.com: Permission denied (publickey).
Warning: ssh process exited with non-zero code "255"

Are the SSH keys correctly configured in balenaCloud? See:
https://www.balena.io/docs/learn/manage/ssh-access/#add-an-ssh-key-to-balenacloud
Are you accidentally using `sudo`?

I reckon the public one

@PackElend, the public key, yes. Balena only ever needs to have the public key. The private key stays in your workstation, ~/.ssh/ directory.

I trying balena-io-playground/ssh-key-insert

To help diagnose the issue, try enabling ssh’s verbose output. Edit the run.sh script: Where it reads “balena ssh”, replace it with “balena ssh -v”.

It’s a long shot, but if the verbose output prints “no mutual signature algorithm”, check this balena CLI issue: balena ssh fails with error: "no mutual signature algorithm" · Issue #2373 · balena-io/balena-cli · GitHub

Another thing you may want to double check is that you are using the same balenaCloud account (same username) when logging in to the web dashboard (through which you added the public SSH key(s)) and when logging in to the balena CLI. Check that the output of the balena whoami command matches the balenaCloud web dashboard username: On the web dashboard, click your name at the top-right corner of the window to raise a pop-up menu that shows your username.

Note that it is not necessary to use the ssh-key-insert script and it is not necessary to add ssh keys to the config.json file in order to use the balena ssh <device-UUID> command, regardless of whether the device is running a production or development variant of balenaOS.

Additional, custom ssh keys added to the config.json file through the ssh-key-insert script (or by manually editing it) allow the balenaCloud proxy backend to be bypassed and a standalone ssh client to be used (instead of the balena CLI) to establish a direct ssh connection from a user’s workstation to a device running a production variant of balenaOS, using the device’s IP address or hostname as the ssh target and ‘root’ as the ssh username. This may be useful, for example, if the device loses access to the internet (perhaps on a schedule) but a workstation is on the same local network as the device (same WiFi access point or Ethernet switch).

Note: this answer was edited to clarify the scenario in which the ssh-key-insert may be useful.

1 Like

For information, balena CLI v13.3.0 or later also allows balena ssh <device-ip-address> to be used with production variants of balenaOS without the need of adding a ssh key to the config.json file. This requires the device to be running balenaOS v2.44.0 or later. The combinations are:

Command balenaOS (dev) balenaOS (prod) < v2.44.0 balenaOS (prod) >= v2.44.0
balena ssh <device-UUID> OK :heavy_check_mark: OK :heavy_check_mark: OK :heavy_check_mark:
balena ssh <device-ip-address> OK :heavy_check_mark: SSH key needs to be added to config.json OK :heavy_check_mark: (CLI v13.3.0 or later)

Also for information, on Windows, note that the balena CLI makes use of the Microsoft’s built-in ssh client that ships with Windows 10 or later. It is not compatible with PuTTY. Check the following answer for more details: add SSH key after provisioning to production device? - #10 by pdcastro

1 Like
PS C:\Users\Den> ssh -p 22222 10.91.102.253 -v
OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2
debug1: Reading configuration data C:\\Users\\Den/.ssh/config
debug1: Connecting to 10.91.102.253 [10.91.102.253] port 22222.
debug1: Connection established.
debug1: identity file C:\\Users\\Den/.ssh/id_rsa type 0
debug1: identity file C:\\Users\\Den/.ssh/id_rsa-cert type -1
debug1: identity file C:\\Users\\Den/.ssh/id_dsa type -1
debug1: identity file C:\\Users\\Den/.ssh/id_dsa-cert type -1
debug1: identity file C:\\Users\\Den/.ssh/id_ecdsa type -1
debug1: identity file C:\\Users\\Den/.ssh/id_ecdsa-cert type -1
debug1: identity file C:\\Users\\Den/.ssh/id_ed25519 type -1
debug1: identity file C:\\Users\\Den/.ssh/id_ed25519-cert type -1
debug1: identity file C:\\Users\\Den/.ssh/id_xmss type -1
debug1: identity file C:\\Users\\Den/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_for_Windows_8.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.9
debug1: match: OpenSSH_8.9 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 10.91.102.253:22222 as 'den'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:qwbLBNYPdxNW7shUuf40T+nbopajc2bTMme9cKoSKl8
debug1: Host '[10.91.102.253]:22222' is known and matches the ECDSA host key.
debug1: Found key in C:\\Users\\Den/.ssh/known_hosts:6
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: pubkey_prepare: ssh_get_authentication_socket: No such file or directory
debug1: Will attempt key: C:\\Users\\Den/.ssh/id_rsa RSA SHA256:B2EX6/h7O1g7B5B8r2nFd/glARxOgtqn/UMksMVR7Dg
debug1: Will attempt key: C:\\Users\\Den/.ssh/id_dsa
debug1: Will attempt key: C:\\Users\\Den/.ssh/id_ecdsa
debug1: Will attempt key: C:\\Users\\Den/.ssh/id_ed25519
debug1: Will attempt key: C:\\Users\\Den/.ssh/id_xmss
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,webauthn-sk-ecdsa-sha2-nistp256@openssh.com>
debug1: kex_input_ext_info: publickey-hostbound@openssh.com (unrecognised)
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: C:\\Users\\Den/.ssh/id_rsa RSA SHA256:B2EX6/h7O1g7B5B8r2nFd/glARxOgtqn/UMksMVR7Dg
debug1: Authentications that can continue: publickey
debug1: Trying private key: C:\\Users\\Den/.ssh/id_dsa
debug1: Trying private key: C:\\Users\\Den/.ssh/id_ecdsa
debug1: Trying private key: C:\\Users\\Den/.ssh/id_ed25519
debug1: Trying private key: C:\\Users\\Den/.ssh/id_xmss
debug1: No more authentication methods to try.
den@10.91.102.253: Permission denied (publickey).

Still testing direct SSH on production image. Testet config.json | jq and all seem correct after adding the ssh-keys.
Still getting permission denied

Any clue for the verbose output above?

I’m able to balena ssh but this is still way too slow, so I’m still pushing to SSH directly through local connection.

Hi,

Assuming you are using a production image, you will first need to copy the ssh public key into config.json as detailed in meta-balena/README.md at master · balena-os/meta-balena · GitHub.

Then you can try to run ssh by specifying both the user and the key:
ssh -p 22222 -i </path/to/private/key> root@10.91.102.253 -v