I was thinking that in stead of mTLS it is also possible to “just” store a JWT token in the environment of the client. With the proper fields in there, it is also possible to revoke the token from the server side without the need to erase it from the device as well. We could still push the JWT token from the server to the Balena Cloud using the API to make it dynamic and do auto-rotation of the keys (depending on how often the devices are expected to check-in with the server) but we would not need the mess of storing a mTLS key-pair in the device environment using base64 encoding etc.
Perfect timing, I was just sitting here thinking through some options.
I am certainly onboard with adopting one of these methods we are discussing, I think your idea on having the MTLS keys in the env variables has been the one that seems to keep surfacing to the top the most.
in stead of mTLS it is also possible to “just” store a JWT token in the environment of the client
This is another one I will have to throw in the pot. First thoughts are that I’m not sure if we would want to use it as a substitute for MTLS/TLS, I think a JWT token exchange would be sending the key through an insecure communication channel (like sending your password over http rather than https). MTLS secures the traffic in transit. Unless the proposal is to move everything to the Cloud, but then we have the vulnerability of the Cloud API keys on the client, which although would allow revoking wouldn’t have the ability for the key to be withdrawn by default.
So many moving parts, I will definitely be pondering the JWT token idea though. My unfinished thought when I read your post was whether authentication could be done via a third party platform like Auth0 (which I’m not particularly familiar with but was the first one that sprung to mind). Even if the JWT approach had to be done with MTLS too, at least we could make the MTLS process invisible by having it auto configure, and only expose the JWT key management options to the user which would be far less friction.
It’s quite complex visualising all the moving parts, I’m thinking we may need a diagram because I am staring at the words but the moving parts are not falling in to place.
What I was thinking about is that we might be able to use the Balena VPN as the secure environment for the secrets store. If we can somehow validate the identity of the requester through the Balena API / VPN we can ensure that we only deliver the keys to devices that are in a particular fleet.
@amrishhpuri, I think you are right, it is similar to what @cees.koolen mentioned above. It seems the most likely scenario right now is to allow the MTLS keys to be in the environment variables in the Cloud then only devices that have access to the Cloud will be able to decrypt.
That is basically there already, it would just need to convert the key from base64, prioritise keys over files, and update the docs: secure-store/mtls.go at 3558f947fe3bd3416d3ebaec2a94aefd53688f6b · maggie0002/secure-store · GitHub
We could put the decryption password on the Cloud too, rather than on another piece of hardware, but it wouldn’t provide as good security because we wouldn’t be able to remove the Cloud from the equation. For example, right now the server is kept on separate hardware which you can protect or remove from your network, if we move all that functionality to the Cloud then there is no way to stop exposing it. Effectively it would mean someone wouldn’t be able to read the SD card and copy the content off it because it would be encrypted, but instead they could copy the Balena Cloud API keys from it, use the key to query the Cloud for the password, then decrypt the content with the password.
There are other options too, I thought about notifications on a dashboard or your phone requesting whether a device can be decrypted, and if you click yes then it passes the decryption key.
@cees.koolen, someone has also queried whether it would be worth using the TPM for Secure Store. We tend to think about TPM for full disk encryption, secure boot etc, and that is being worked on, but you can also use TPM to generate keys for other things. It could be used from inside a container to generate the key that decrypts the Secure Store content, instead of using the environment variables in the Cloud. It has some promise, but without secure boot, someone could potentially put their own software on the hardware, extract the key from TPM, then decrypt the content with that key.
A quick update. A few things in the PR here to be merged (Updates by maggie0002 · Pull Request #1 · balena-labs-research/secure-store · GitHub), any thoughts or testing welcomed and appreciated:
- Allow adding the MTLS certificates as environment variables instead of files and prioritise environment variables over files. This creates a dependency on access to the Balena Cloud for the decryption to function.
- Allow creation of the encrypted mount locally by passing in a password. In other words, running without the need for the server, but the user will need to implement their own steps to protect the password stored on the device. This is helpful for those who want to fork the project, and may want to embed alternative security measures in to the binary.
I think this covers the low hanging fruit from above, and will keep exploring other options, happy to hear any thoughts.
For the benefit of the encrypted mount we need to look at the idea that it can hold many secrets and we only need to unlock it once where as we can also use that same password to encrypt all the individual secrets but then we need the password much longer.