Problem with SSH key on Production

We have started trying out production images, and have run into an issue with using SSH keys. We first flash the image to an eMMC, then edit the config.json to add the key. The reason we add the key before provisioning the device is because we’ve setup a way to encrypt the data partitions, so someone can’t just pop out the eMMC and look at our stuff. With the ssh key added, we can copy over the keys/scripts to encrypt the device, then add it to an application. We can’t add it to an app first because then services will be running and using those data partitions, at which point we can’t encrypt them.
Once the key is added, we can ssh into the device just fine, giving the key with the -i flag. However, when we try to balena ssh or balena join the device, we are prompted for root@<ip>'s password. We do not set this password, so there’s no way to login. Since there’s no way to give the ssh key to these commands, they appear to be stuck here. We are able to do all of this just fine on dev devices, though with those we don’t need to add an ssh key.
The kicker here (and what we think may be the main issue) is that we’re working on unsupported boards (the Odroid-C2). We use a custom built OS, and have to manually add devices to applications. So since the devices don’t natively have an API key installed (like what you’d get on a supported board image when downloading it from balena for an application), it appear that balena doesn’t know to trust it (or something).
We think this may be the sticking point, but just wanted to check if there was perhaps another issue at play.

After trying out a production image without adding an ssh key, I have found that I still can’t add the device to an application.

Hello,

Just so I understand this correctly, what you’re doing is:

  1. Flash the eMMC on the device with the unconfigured OS image, editing the SSH key to the config.json using the OS section, like: "os": { "sshKeys": [ "...." ] }
  2. SSH into the device once booted using the SSH key and then copy the rest of the required keys to encrypt the device
  3. Add the device to a balenaCloud application

If you’ve added the SSH keys as described above, then as long as your custom OS build is using the meta-balena Yocto layers, then this should work (with a call to both balena ssh and ssh -p 22222 root@<deviceIp>. I’ve just tried this with an unconfigured production device image for 2.47.0+rev1 and then inject a config.json that had no application details or keys, and was able to SSH into the device correctly. As long as those keys are added to the config.json then this should work fine.

As you say, the only thing I can think of here is that for some reason the custom OS build isn’t including the functionality to pick up the keys in the configuration file?

Best regards,

Heds

Yes, that is what I’m doing. I can regular ssh into the device just fine, but I can’t balena ssh or balena join the device, because it asks for the root password. The fact that I can ssh into the device with the key means that the OS on the device can see the key (I believe). Does the key need to be in a specific spot in order for the balena commands to see it, as opposed to specifying it with the -i flag (which balena doesn’t have)? The key is just in the local directory where I’m trying to run these commands.

Hi again @MasterSpecter,

Right, I understand the confusion now. So balena ssh runs the standard ssh command, and as you say it has no -i flag for passing a keyfile. For this reason, you’ll either need to add the key to the SSH agent running (via ssh-add) or you’ll need to put the key in the .ssh directory for the user running the command. SSH will then do the usual key-exchange with the device to send the right key.

Let me know if this still doesn’t work, and I’ll look into it more.

Best regards,

Heds

That did it. Thank you very much!

You’re most welcome, glad I could help. :slight_smile:

Best regards,

Heds