Build time secrets / best practices for SSH keys

Hi there,

I’m building an app that pulls dependencies from private git repos during the docker build phase.

I’m planning to do a multi-stage build and only provide the necessary git ssh keys to the “builder” stage of the build.

I’ve started by adding an ssh key as a secret using .balena secrets in balena.yml:

build-secrets:
    global:
        - source: id_rsa
          dest: id_rsa

I’m then using the ssh key secret in my Dockerfile like this:

FROM wlisac/raspberrypi3-swift:5.0 as builder

# Add credentials and known hosts to builder
RUN mkdir /root/.ssh/ \
    && cp /run/secrets/id_rsa /root/.ssh/id_rsa \
    && chmod 400 /root/.ssh/id_rsa \
    && touch /root/.ssh/known_hosts \
    && ssh-keyscan github.com >> /root/.ssh/known_hosts

# Do stuff that uses a private github.com repo

FROM wlisac/raspberrypi3-swift:5.0

# Copy build artifacts from builder

CMD ["./start.sh"]

This is working with balena push, but it would be nice to find a solution that also works with balena build. A best case scenario would be a “fully portable” Dockerfile that would even work with raw docker build command, too.

Any suggestions on a “portable” approach that works with balena push, balena build, and maybe even docker build?

I’m considering using ARG instead of a mounted secret to allow the Dockerfile to be more portable. This should allow me to pass the ssh key as a build arg to balena build or docker build commands. Although balena push looks like it still requires build args to be defined in the balena.yml file?

The documentation says that secrets are more secure than build args, but I’m not sure it makes a difference in this case since I need(?) to copy the ssh key to the builder image either way (I can’t chmod 400 the ssh key on the secrets mount since it’s a read-only file system).

Any suggestions or best practices folks could share would be very helpful.

The balena secrets looks like a great new feature – hoping I can figure out how to use it correctly :sweat_smile:

Cheers,
Will

I’m considering using ARG instead of a mounted secret […] The documentation says that secrets are more secure than build args, but I’m not sure it makes a difference in this case since I need(?) to copy the ssh key to the builder image either way

Yeah, I was wondering about that… Docker’s documentation on the ARG instruction reads:

Warning: It is not recommended to use build-time variables for passing secrets like github keys, user credentials etc. Build-time variable values are visible to any user of the image with the docker history command.

But if you want / intend / need to save your secrets on the final image anyway (at /root/.ssh/id_rsa), then I think the secrets feature doesn’t make a difference. You might as well just use plain COPY or ADD in the Dockerfile:

COPY id_rsa /root/.ssh/id_rsa

… which is portable across all Balena and Docker commands. :slight_smile: I think the main benefit of the secrets feature is to avoid saving secrets to the image, while allowing them to be used during the build.

But if you want / intend / need to save your secrets on the final image anyway (at /root/.ssh/id_rsa ), then I think the secrets feature doesn’t make a difference. You might as well just use plain COPY or ADD in the Dockerfile:

The ssh key won’t be saved in the final image, only in builder image. I think this helps mitigate risk, but I assume balena still caches the builder image.

Aside from the portability challenges, I’d love to use balena secrets for the git ssh key. However, the permission of the file is too permissive for use with ssh.

Here’s an example Dockerfile that uses the ssh key in the secrets directly without copying to the builder image:

FROM wlisac/raspberrypi3-swift:5.0 as builder

# Add credentials and known hosts to builder
RUN mkdir /root/.ssh/ \
    && echo "Host github.com\n  User git\n  IdentityFile /run/secrets/id_rsa" > /root/.ssh/config \
    && touch /root/.ssh/known_hosts \
    && ssh-keyscan github.com >> /root/.ssh/known_hosts

# Do stuff that uses a private github.com repo

FROM wlisac/raspberrypi3-swift:5.0

# Copy build artifacts from builder

CMD ["./start.sh"]

Here’s the corresponding error from the build output:

[main]     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[main]     @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
[main]     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[main]     Permissions 0644 for '/run/secrets/id_rsa' are too open.
[main]     It is required that your private key files are NOT accessible by others.
[main]     This private key will be ignored.
[main]     Load key "/run/secrets/id_rsa": bad permissions
[main]     git@github.com: Permission denied (publickey).
[main]     fatal: Could not read from remote repository.
[main]     Please make sure you have the correct access rights
[main]     and the repository exists.

I suspect I’m just missing something. My use case is closely aligned with the original feature description on Trello :slight_smile:

Allow users to specify secret files and variables that can be used during build time to authenticate with external services like github, npm, etc. This allows user to build code from private repositories, etc without having these secret keys ending up on the end devices or commited into their git version control

Hi @wlisac - I think you can sort that part of the problem by copying the key to some temporary location, using it, and deleting it within the same RUN step. That way it won’t be stored in any layer of the build image and you can change the permissions of the copied file.

It only solves half of the problem though, cause it’s still not a “portable” solution and it’d only work with balena push. We’ve been discussing making balena secrets compatible with docker’s new build secrets, though they still seem to be rather experimental, and it’d take some time for us to actually start working on this.

1 Like

Thanks – all of that makes sense. I’ll experiment more with this and report back.

Glad to hear that secret support for balena build is being considered.

In the spirit of unifying the use of balena build and balena push with secrets and build args, it might be nice for balena push to allow build args (and secrets) to be specified as command line options and not require the use of balena.yml. Is that something that has been considered, too?

One benefit of allowing build args or secrets as command line options is we wouldn’t have to duplicate the secret in the .balena/secrets folder.

For example, you could cat the ssh key into a build arg or secret option. I think the build arg approach is already possible today with balena build:

balena build --application MyApp --build-arg SSH_KEY="$(cat ~/.ssh/id_rsa)"

Indeed, being able to specify the secrets in the command line would be nice. There’s some ongoing discussion about this in two GitHub issues on our CLI repo:

This involves some challenges, hopefully there’ll be progress on these with time. And I hope for now you can work it out with the cp, chmod and rm within the same RUN command. But let us know if anything else comes up!

1 Like

Thanks @pcarranzav.

A related issue I ran across is https://github.com/balena-io/balena-cli/issues/1164.

It’s marked as closed, but it’s unclear to me if build time secrets work with balena push <deviceIP>.

It looks like the merged PR is for registry secrets only? I’m asking because I haven’t been able to get balena push <deviceIP> to work with build time secrets.

Thanks again for all of the help.

Good question, I’ve asked internally and we’ll report back

Hey @wlisac

Apologies for the confusion, it looks like I accidentally referenced the wrong issue in my commit. Secrets are still on the roadmap for local mode, I unfortunately don’t have a timeframe right now though.

1 Like

Thanks for the update @CameronDiver.

Are there any workarounds for adding ssh keys via balena push <deviceIP>?

Here are the ideas I’ve considered, but don’t appear to work yet:

  • Use build time secrets or args via balena.yml (Issue 1164)

  • Use balena build and provide the ssh key via a --build-arg, then push the built image to a local device via balena push <deviceIP> (Issue 613)

  • Add the ssh key via a normal ADD or COPY command, while keeping the ssh key ignored by git (Issue 1032)

We use balena push <deviceIP> pretty heavily during development, so open to other ideas. :slight_smile:

cc @pdcastro

Cheers,
Will

@pdcastro @CameronDiver I found a bit of a surprising workaround for using an ssh key stored in the ./balena/secrets folder with balena push <deviceIP>.

It turns out you can COPY the .balena/secrets folder into a /run/secrets folder when using balena push <deviceIP> even though the .balena/secrets folder is ignored with .gitignore.

When using the cloud builders with balena push <MyApp>, the .balena folder is not available to COPY, so we need to use a little workaround to conditionally copy the secrets folder.

The end result is a /run/secrets folder when using balena push <deviceIP> and access to the actual balena secrets mount when using the cloud builders.

FROM wlisac/raspberrypi3-swift:5.0 as builder

# Temporary workaround for local push secrets https://github.com/balena-io/balena-cli/issues/1164
# Note that the .balena/secrets are only copied during a local push; not during a cloud push
COPY README.md .balen[a] /run/

# Add credentials, fetch dependencies, remove credentials
RUN mkdir /root/.ssh/ \
    && cp /run/secrets/id_rsa /root/.ssh/id_rsa \
    && chmod 400 /root/.ssh/id_rsa \
    && touch /root/.ssh/known_hosts \
    && ssh-keyscan github.com >> /root/.ssh/known_hosts \
    && clone_private_repo_here \
    && rm /root/.ssh/id_rsa

FROM wlisac/raspberrypi3-swift:5.0

# Copy build artifacts from builder

CMD ["./start.sh"]

I’d love to hear any thoughts on this workaround :slight_smile:

Cheers,
Will

Thanks for the feedback, very interesting, we are checking with the team and will keep you posted!

1 Like

Just to keep you informed on our progress, I’ve implemented the work that is required in one of our satellite modules here: https://github.com/balena-io-modules/resin-multibuild/pull/53. This now just needs implemented into the cli, which one of my colleagues is going to tackle in the coming week or so.

It looks like secrets now work with balena push <deviceIP> in balena CLI v11.7.0.

So awesome, thank you! :raised_hands:

Great to hear! Feel free to open a ticket any time you run into trouble in future, we’ll be glad to assist.

1 Like