Production Image SSH Access

Could someone explain to me (conceptually) how balena cloud accesses production devices via the web terminal? I am looking to build this functionality into open-balena-admin, currently web ssh functionality in open-balena-admin works for development images only because they do not require an SSH key, making the proxying simpler - but obviously not suitable for a production environment.

My current plan to make this work for production images is to initially provision all devices with an SSH key that has its private pair stored on the open-balena-remote server, but this is not ideal because it should really be rotated, and would create issues if ever compromised.

In looking through the meta-balena recipes, I don’t see any specific backdoor that allows balena cloud ssh access, nor do I see any ssh keys provisioned with balena cloud production devices. So how is balena cloud accessing production devices via the web terminal?

UPDATE It looks like this functionality may be in the cloud-public-sshkeys script in recipes-connectivity/openssh which appears to take ssh keys stored in open-balena-db (accessed via api) and merge them with any custom keys from config.json. So as long as there is a public key associated with a user in open-balena-db, and you ssh into the device using that user (with the associated private key) it should work. Now where to store the private key is another issue…

For the benefit of anyone who might be interested, I’m happy to report that remote SSH access to production devices has been added to open-balena-admin, with commits pushed to open-balena-ui and open-balena-remote and respective docker images updated. Again, huge kudos to the Balena team for even making this possible, all I am doing is leveraging the amazing infrastructure that they have built.

The way it works is as follows:

  • When a user initiates a SSH connection to a device via the “connect” button on either the main dashboard or device dashboard, open-balena-ui automatically generates a SSH RSA key pair
  • The public key is saved as a new “SSH Key” for the user, with the name open-balena-remote; if a key already exists with this name it is updated
  • The private key is passed to open-baena-remote, and discarded by open-balena-ui. It is only saved in temporary storage (the session folder) on open-balena-remote, and automatically destroyed when the session is over, after time expiry or the user manually closing the SSH session - so the key is effectively a one time use item.
  • open-balena-remote initiates the SSH conenction to the remote device using the username of the initiator and the private key; the device then automatically pulls the matching public key from that user’s SSH key storage via the cloud-public-sshkeys script described above; this is the magic of Balena at work
  • You are now securely connected!

This method is used regardless of whether an OS is configured in development mode or not, as it works in either case.

Hope someone finds value in this - and any feedback is always welcomed.

This is great stuff! Thanks for the contributions, and the write up above on how to go about this!