add SSH key after provisioning to production device?

Hello,
I’m really lost and doubt that it is even possible at all.

I have already read through Adding SSH key after flashing - Product support / balenaOS - balenaForums and followed Unable to add SSH key - Product support - balenaForums but I have not been able to ssh into my device.

Is any public added to pushed to the devices automatically or is any action of the admin necessary?

Where does balena-cli\bin>balena ssh <uuid> expect the PuTTY key > OpenSSH key to be stored as the -i flag is not working in the CLI?
I run --verbose what reveals that only %USERPROFILE%/.ssh/ is checked for the identity file.
Moreover, I see

debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts2 error:2

My CLI runs on Windows 10.

Could anyone tell me how to proceed?
I read:

  1. balena-io/balena-cli-masterclass: A guide to getting started with the balena CLI (github.com) which does not tell how the key finds it way into the device
  2. balena-os/configizer: Safe(r) balenaOS config.json updates remotely (github.com) which seems only possible with existing ssh connection.
    1. Moreover, I have to use push to get the files into the device, don’t I?
  3. balena CLI Documentation - Balena Documentation but could not find an option to modify the config of a running device config works against an image only (seems to me so).
  4. balena-io-playground/ssh-key-insert: Tooling to insert the relevant SSH keys into balena devices’ configuration. (github.com) leaves me clueless that might be the solution but it cannot only be executed on Linux (or Windows Subsystem for Linux), can it?

In the end I want to finish how to import proxy server certificates into CLI and OS? - Product support / balenaOS - balenaForums

thank you for enlightening me in my desperation
stefan

Hello there,

SSH and balenaOS works like this:

  • balenaOS trust one SSH key by default: The key from balena cloud
    • when you connect via the web-dashboard, this is used to connect
    • when you use balena ssh DEVICE_UUID your own ssh key is used to connect to the cloud and the cloud uses its key to connect to the device
  • when you use balena ssh LOCAL_IP_OF_DEVICE we assume it’s in development mode (actually not 100% sure on that, need to re-check) which means no key is needed at all
  • you can add an extra trusted ssh certificate in config.json, this will allow you to connect via ssh -p 22222 root@LOCAL_IP_OF_DEVICE, this is mostly useful of you need offline access
  • you can ssh to a device, which is online via ssh -p 22222 YOUR_CLOUD_USERNAME@LOCAL_IP_OF_DEVICE
    • here the device temporarily downloads ssh key that you have configured in balenaCloud and checks your local key against it

Neither of these has anything to do with the balenaRootCA key in config.json which you mention in your other forum post. That one changes which certificates we trust when doing HTTPS (TLS) communication with balenaCloud and is most commonly changed when you have an intercepting (corporate) proxy in the middle.

Hope this clarifies things a bit.

Cheers,
Martin

PS: Windows 10 has the original ssh included nowadays. If you have issues with Putty, you can use that

thx for the update

it is related in such a sense, that I have to ssh into the image to install the CA, that is the background why I started doing it.

it helps :slight_smile: a bit

does the above mentioned apply to the productive image as well as it is stated in What is balenaOS? - Balena Documentation

Production images disable passwordless root access, and an SSH key must be added to config.json to access a production image using a direct SSH connection.
I’m wondering how to do this.
Currently I see only two options:

  1. balena-io-playground/ssh-key-insert: Tooling to insert the relevant SSH keys into balena devices’ configuration. (github.com) but currently battling WSL to make it work with the proxy (in the end it is a repeat of what has to be done to make balena-cli work with the proxy)
  2. doing all through the web terminal (which prone to errors)

I know that you can add public SSH key to the cloud but no clue how that makes its way to the device :question:

@Hades32 I got WSL2 up and balena-cli running in it, see How-To: How install balena-cli in WSL2 if behind Proxy like Zscaler - Product support / balenaOS - balenaForums.
I tried now balena-io-playground/ssh-key-insert: Tooling to insert the relevant SSH keys into balena devices’ configuration. (github.com) but I get

:~/balena.io/ssh-key-insert-master$ bash -x ./run.sh
+ cat batch
+ stdbuf -oL xargs '-I{}' -P 10 /bin/sh -c 'grep -a -q '\''{} : DONE'\'' sshkey.log || (cat add-ssh-key.sh | balena ssh {} | sed '\''s/^/{} : /'\'' | tee --append sshkey.log)'
<usedr-name>@ssh.balena-devices.com: Permission denied (publickey).
Warning: ssh process exited with non-zero code "255"

Are the SSH keys correctly configured in balenaCloud? See:
https://www.balena.io/docs/learn/manage/ssh-access/#add-an-ssh-key-to-balenacloud
Are you accidentally using `sudo`?

I’m still clueless about which key is expected for what action and how keys are distributed…?

You don’t have to SSH into the device to install the CA. You just add it into config.json as part of provisioning: meta-balena/README.md

And if you really have to add ssh keys - which I don’t think you need - the same docs show you how: meta-balena/README.md

but what if I forgot to do so?

that is what I’m trying using balena-io-playground/ssh-key-insert: Tooling to insert the relevant SSH keys into balena devices’ configuration. (github.com) but I’m bit clueless here how is the SSH key added to balenaCloud used? - Product support - balenaForums :slight_smile:

@PackElend, I can confirm that balena ssh <device-UUID> does not require adding a public ssh key to a device’s config.json file (regardless of whether it is running a production or development image of balenaOS), as long as the device can access balenaCloud (i.e. the device has access to the internet and no firewall is blocking its access to *.balena-cloud.com).

You should not need to use the ssh-key-insert script in order to use balena ssh <device-UUID>.

Where does balena-cli\bin>balena ssh <uuid> expect the PuTTY key > OpenSSH key to be stored as the -i flag is not working in the CLI?

I understand that PuTTY store SSH keys in the Windows Registry, but the balena CLI does not look for keys in the Windows Registry, and in this sense it is not compatible with PuTTY. (There may be a way of making it work with PuTTY, but it is not a documented/supported scenario.)

The balena CLI assumes the use of Microsoft’s built-in ssh client that ships with Windows 10 or later. (Microsoft introduced the built-in ssh client through a Windows 10 maintenance update in year 2018.)

I run --verbose what reveals that only %USERPROFILE%/.ssh/ is checked for the identity file .

Exactly. Your public and private ssh keys should be in that directory.

Here’s an example of how to generate the keys using PowerShell.

Confirm that ssh, ssh-keygen, ssh-add and ssh-agent are the Microsoft built-ins, rather than PuTTY’s:

PS C:\Users\paulo> Get-Command ssh | Format-Table -AutoSize
PS C:\Users\paulo> Get-Command ssh-add | Format-Table -AutoSize
PS C:\Users\paulo> Get-Command ssh-agent | Format-Table -AutoSize
PS C:\Users\paulo> Get-Command ssh-keygen | Format-Table -AutoSize

CommandType Name           Version Source
----------- ----           ------- ------
Application ssh.exe        8.1.0.1 C:\WINDOWS\System32\OpenSSH\ssh.exe
Application ssh-add.exe    8.1.0.1 C:\WINDOWS\System32\OpenSSH\ssh-add.exe
Application ssh-agent.exe  8.1.0.1 C:\WINDOWS\System32\OpenSSH\ssh-agent.exe
Application ssh-keygen.exe 8.1.0.1 C:\WINDOWS\System32\OpenSSH\ssh-keygen.exe

Generate a public/private key pair as follows. The most widely recommended key type nowadays appears to be Ed25519 (e.g. Microsoft guide and GitHub guide) as RSA is increasingly considered less secure, so:

PS C:\Users\paulo> ssh-keygen -t ed25519
Generating public/private ed25519 key pair.
Enter file in which to save the key (C:\Users\paulo/.ssh/id_ed25519):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in C:\Users\paulo/.ssh/id_ed25519.
Your public key has been saved in C:\Users\paulo/.ssh/id_ed25519.pub.
The key fingerprint is: ...

PS C:\Users\paulo> dir C:\Users\paulo\.ssh\
Mode      LastWriteTime       Length    Name
----      -------------       ------    ----
-a----    28/01/2022 16:39    464       id_ed25519
-a----    28/01/2022 16:39    104       id_ed25519.pub

PS C:\Users\paulo> type .\.ssh\id_ed25519.pub
ssh-ed25519 AAA... paulo@DESKTOP-3021L0B

Now copy the public key component (contents of the id_ed25519.pub file – output of the type command above) to the balenaCloud web dashboard: Preferences → SSH Keys → Add a new SSH key → Add key manually. (The contents of the Title field are just for your own reference and not used by the balena CLI.)

Something to double check: Ensure that you are using the same balenaCloud account (same username) when logging in to the web dashboard and when logging in to the balena CLI. Find the web dashboard username by clicking on your name at the top-right corner of the window – this raises a pop-up menu that shows the username. Check that the username matches against the output of the CLI’s balena whoami command.

Note re private key passphrase:
When ssh-keygen prompts you for a passphrase, it is good security practice to set one (not leaving it blank). Then, for convenience, to avoid having to type the private key’s passphrase every time you run balena ssh, run the ssh-agent service as per Microsoft’s guide. That is:

On a PowerShell prompt opened as Administrator:

PS C:\WINDOWS\system32> Get-Service ssh-agent | Set-Service -StartupType Manual
PS C:\WINDOWS\system32> Start-Service ssh-agent
PS C:\WINDOWS\system32> Get-Service ssh-agent

Status   Name               DisplayName
------   ----               -----------
Running  ssh-agent          OpenSSH Authentication Agent

On a PowerShell prompt opened as regular user:

PS C:\Users\paulo> ssh-add
Enter passphrase for C:\Users\paulo/.ssh/id_ed25519:
Identity added: C:\Users\paulo/.ssh/id_ed25519 (paulo@DESKTOP-3021L0B)

PS C:\Users\paulo> balena ssh aab9ad....
=============================================================
    Welcome to balenaOS
=============================================================
root@aab9ad3:~#

You should then only need to re-enter the passphrase when the workstation is rebooted.

Finally, regarding the following debugging output:

debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts2 error:2

It is normal, not a problem. ssh’s debugging output seems to use the word “failed” whenever some configuration file doesn’t exist, even when harmless.

Wow that is comprehensive answer :slightly_smiling_face:
I’m currently away, will check as soon as I am back

@PackElend, were you able to get balena ssh to work? For your information, balena ssh improvements were made in balena CLI version 13.3.0 or later, such that even when using the IP address of a production balenaOS device as the ssh target, it is not necessary to add a ssh key to the config.json file. The ssh documentation on balena’s website was updated as well: SSH access - Balena Documentation

PuTTY is still not supported though, and the instructions in my previous answer still apply.

to be honest I had to pause this as I was forced to change priorities.

it is still on my agenda. I hope I can make up some hours in April as I would like to use it for testing containerized SysLog-Application locally