We have a fleet of micro services devices on Balena. I’m working on a project to upgrade the security between our devices an our back-end systems. We would like our devices to authenticate themselves against our back-end using mutual TLS.
At the back-end I intend to use cert-manager in Kubernetes to manage all the certificates for me. But I can’t quite envision how I can distribute the client certificates to each of our devices. Each device should get a ‘personal’ client certificate. And I would like to automate this process so that the certificates can be rotated every couple of months.
The only thing I can come up with is; To put the certificates in service variables for each device. I could make a (small?) service in the back-end that uses the Balena API to receive the certificates from Kubernetes and set these on the correct devices periodically.
But I’m wondering, is that like a good idea? I’m not thrilled by the idea that the certificates will show up in the dashboard is there a way around that? Has anyone else experience with something like this?
Our AWS IoT integration does something like that - AWS IoT integration - Balena Documentation
It uses device service variables to share the certificates and keys with each device. Something like that would work.
If you don’t want to use Balena’s servers to distribute the certificates, I think you are going to have to spin your own service which talks to your backend services. The concern with that could be the same - that service is going to need some credentials to talk to your service.
One other way to do this would be to put the certs in hardware. Check out Microchip’s ATECC608A for example. But with this approach IIRC it’s not possible to cycle the certificates. As the certificates never leave the hardware. You can only run operations using those certificates, but you can never read/write them.
Thank you for the extra insights. This helps me a lot. So I’m going to go ahead by putting the certificates in variables like the AWS IoT integration documentation suggests.
With regards to the variables being visible. Can I put in a feature request for the dashboard that will allow us to mark environment variables as secure?
They do this with Bitbucket Pipelines (CI) for example. There you can mark variables for you pipeline as secure. When a variable is marked secure the value is never shown in the interface and it can’t be copied from there either. It will make sharing secrets with devices that much more secure.
Erik hi! We recieved your feedback and I already started the internal process to be able to discuss it further. Cant say if and when its going to be implemented, but we’ll do our best to let you know. Please let us know how it goes with the certificates in variables and let us know of any obstacle you may encounter so we can help. Thanks!
I ended up making a batch job using the Balena SDK. I mount all the required certificates an keys from Kubernetes in the Pod. It runs by all the devices and sets the appropriate service variables with the base64 encoded contents of the TLS files.
I run it once every 15 minutes so that when certificates are rotated the device configuration receives the updated certificates. In the container on the device I made a small script that restarts the container when one of the environment variables has changed so that new values are picked up.