TL;DR: You can add an entry to config.json which will trust any key signed by your SSH certificate authority: Add a key like cert-authority $CA_PUBLIC_KEY to your config.json.
I thought I’d write up something we implemented earlier this week, making use of SSH’s support for certificate based authentication to allow SSHing to devices without having to place specific public keys on the device which would then need rotating as people join and leave the company.
If you don’t have any public keys in a device’s config.json the only way to authenticate is by the device sending your public key and username to Balena’s API to validate you’re allowed to connect. This is usually fine, and means that access can be revoked immediately if needed, but becomes problematic if the device has no connectivity to the Balena API. We had this happen recently where we could get to the device via an alternative route, but were unable to SSH in to fix things because auth failed.
We could pre-populate with one or more public keys attached to staff members, or a shared public key with the private key saved in 1Password to provide access to people, but then we’d need to rotate those keys on a regular basis to remain in compliance with security policies. Thankfully OpenSSH supports the used of signed keys, which allow a user’s public key to be cryptographically signed by a certificate authority, with devices permitting any key signed by that CA to authenticate. Signed keys can also have limitations attached around which users they’re valid for, expiry dates to be attached, and source IPs to be limited. You can find some background at OpenSSH/Cookbook/Certificate-based Authentication - Wikibooks, open books for an open world.
The missing piece for us was the need to configure Balena devices to accept our certificate authority’s signatures. Most of the tutorials out in the wild state that you need to update sshd_config for this, which would have required us forking balenaOS and managing our own images, however OpenSSH’s documentation mentions (buried deep in some options) that you can also do this via an authorized_keys file… the very same file that gets populated by adding keys to config.json.
The final step was getting that rolled out to our existing fleet of devices, which is very handily done using the script published at GitHub - balena-io-experimental/ssh-key-insert: Tooling to insert the relevant SSH keys into balena devices' configuration. to SSH into each device in turn, then add the CA’s public key to them.