I’m facing an issue that is probably something some of you already stumbled on before.
I always try to run all my containers on scratch images and with a user different than root. My last stage when building the images is something like this:
# create the final image
FROM resin/scratch as final
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=builder /app /app
USER nobody:nobody
ENTRYPOINT ["/app"]
This works fine unless the application needs to create/write files on the persistence folder (using named volumes for files on /var/lib/docker/volumes/<APP ID>_resin-data/_data/).
I have tried a few things but without success, like changing the ownership of the named volume folder. That works when running the container on my local machine, but not when running on balenaOS.
Thanks for the question. I notice you’re using resin/scratch as a base image, here. This image is incredibly out of date. You should just be able to use FROM scratch (as per normal Docker usage) to create a bare starting image. Out of interest, could you please let us know where you instructions to use this, as we’d like to ensure they’re removed?
I don’t believe there should be any issue with using non-root user to store persistent data for your application, so I’d like to look into this a bit more. Could you possibly give me a cut-down example Dockerfile (and docker-compose.yml if you’re using multi-container applications) that exhibit this problem? It would help us look into this in more detail.
Apologies, as a follow up, I do know what’s going on. Because the permanent data partitions are set by the host OS, you’ll need to ensure that permissions are setup for the /data directory (or equivalent volume if you’re using multicontainer). You should be able to do this as part of an entry script for your service, but you will have to have access to the root user to obviously carry this out. If you chown nobody:nobody /data using this, then the rest of the app should run as expected.
I’ve pinged others in the team, as this is probably something they’ve come across before.
thank you for the feedback and the heads up regarding the scratch image (to be honest, I don’t remember if I saw the resin/scratch in the docs or if I just used resin out of habit).
I’ve tried with chown -R nobody:nobody /data on the building stage of my Dockerfile (as I can’t do that on the scratch container) and then copy that folder to the scratch image, but without success.
How can I send you a complete example of the application? (I can’t update .zip files here)
I think this also comes down to how the root user is setup. I’ve managed to make this work in a debian container by ensuring root actually has a password associated with it, using the following Dockerfile:
FROM balenalib/raspberrypi3-debian:stretch
RUN echo "root:pword" | chpasswd root
RUN addgroup nobody
WORKDIR /usr/src/app
USER nobody:nobody
COPY entry.sh /usr/src/app/entry.sh
CMD ["/usr/src/app/entry.sh"]
that would definitely work if I was not using scratch.
The problem is that when running your application from scratch, you can’t run any of those bash commands (unless you copy everything from the builder image, but then doesn’t make sense using scratch anymore).
unfortunately, that doesn’t address the issue we are discussing here. Multistaged builds is what I’m currently doing and works fine if my app doesn’t need to create or write files in named volumes.
The problem we are trying to figure out here is how to create/write files in named volumes when running from scratch without being root.
My idea was that you may copy files from one image to the other - the different images that are part of the multistage builds. In this way you may issue a command in the first image and when you start from scratch, you may copy particular files to the scratch image. The example demonstrates how to copy the /etc/passwd file in particular which is created in the first phase.
Please let us know if I misunderstood your last comment.
yes, that’s exactly how scratch images are supposed to be used since they are empty images, you need to copy everything you need from the outside or from other build stages.
I’m not sure you are understanding the issue though. The issue is not how to use scratch images or how to create a user and copy it to scratch and run the container with that user. The issue is creating/writing to files in the persistent storage layer though named volumes.
Please consider the following example:
Dockerfile:
FROM golang:1.12-alpine as builder
RUN apk add --no-cache ca-certificates git
# Create the user and group files that will be used in the running container to
# run the process as an unprivileged user.
RUN mkdir /user && \
echo 'nobody:x:65534:65534:nobody:/:' > /user/passwd && \
echo 'nobody:x:65534:' > /user/group
# Add source code.
WORKDIR /src
COPY main.go .
RUN go mod init example-app && go mod vendor
RUN CGO_ENABLED=0 GOOS=linux GOARCH=arm go build -mod=vendor -a -ldflags="-s -w" -o /app .
# create final image
FROM scratch as final
COPY --from=builder /user/group /user/passwd /etc/
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=builder /app /app
USER nobody:nobody
ENTRYPOINT ["/app"]
Hi,
The problem is that the named volume is owned by the root user.
If you don’t want to use bash scripts, you could start your application as root, change the ownership of /data (chown syscall) to the non root user and then drop root privileges (setuid setgid syscalls) in your application. Depending on what type of filesystem access you need you could also open the fd while still running as root and then dropping privileges. Already open fd will still operate with the uid they where opened with.
Hope this helps.
Best regards,
FROM golang:1.12-alpine as builder
RUN apk add --no-cache ca-certificates git
# Create the user and group files that will be used in the running container to
# run the process as an unprivileged user.
RUN mkdir /user && \
echo 'nobody:x:65534:65534:nobody:/:' > /user/passwd && \
echo 'nobody:x:65534:' > /user/group
# Add source code.
WORKDIR /src
COPY main.go .
RUN go mod init example-app && go mod vendor
RUN CGO_ENABLED=0 GOOS=linux GOARCH=arm go build -mod=vendor -a -ldflags="-s -w" -o /app .
# create final image
FROM scratch as final
COPY --from=builder /user/group /user/passwd /etc/
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=builder /app /app
RUN mkdir /data
RUN chown nobody:nobody /data
USER nobody:nobody
ENTRYPOINT ["/app"]
I have not tested this on a balena device, but maybe this works. I think it would be important, that the named volume is newly created.
RUN chown nobody:nobody /data
USER nobody:nobody
ENTRYPOINT ["/app"]
There’s no chown on scratch, so unless we would copy it to the image (and all its dependencies), that will not work.
What I’ve tried was to run chown on the builder stage and then copy the folder to scratch, but that doesn’t seem to work either. I’ve confirmed that the folder is owned by nobody by attaching to the container, and that’s where I got stuck becauseI can’t figure out why can’t my app, that is running as nobody, create files on a folder also owned by nobody!
FROM golang:1.12-alpine as builder
RUN apk add --no-cache ca-certificates git
# Create the user and group files that will be used in the running container to
# run the process as an unprivileged user.
RUN mkdir /user && \
echo 'nobody:x:65534:65534:nobody:/:' > /user/passwd && \
echo 'nobody:x:65534:' > /user/group
VOLUME ["/data"]
RUN chown -R nobody:nobody /data
# Add source code.
WORKDIR /src
COPY main.go .
RUN go mod init example-app && go mod vendor
RUN CGO_ENABLED=0 GOOS=linux GOARCH=arm go build -mod=vendor -a -ldflags="-s -w" -o /app .
# create final image
FROM scratch as final
COPY --from=builder /data /data
COPY --from=builder /user/group /user/passwd /etc/
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=builder /app /app
USER nobody:nobody
ENTRYPOINT ["/app"]
Note that this works fine when running on my machine. It just not working when running on my balena devices…
When the local /data directory is mounted on the device (to a local volume), the permissions are effectively overwritten (as the Supervisor, running as root, carries out this bind). This obviously means, regardless of any volume that’s been given permissions previously, the local volume storage will default to root.
This is why under a distribution (such as the small example I posted last week with Debian Stretch) you need to change the permissions at runtime as part of your entry script. Similarly, if you want to use a scratch image, you’re going to have to copy appropriate tools, such as chown into that image to allow the alteration of permissions at runtime.
Unfortunately, I don’t believe there’s any other way to currently carry this out. We can’t easily alter the ability to change permissions of ownership on a volume bind, because we don’t know in advance the user that needs to own it (uid/gid). I’m going to raise a ticket internally attached to this conversation, so we can consider this as I do understand why you’d like to execute and own the volume as a non-root user, but unfortunately I can’t think of any other solution at the moment.
I encountered the same issue - didn’t have permissions to access the volume.
It turned out, I previously created the volume and Balena created it with root permissions. Then I changed my Dockerfile to use a non-root user and I gave the non-root user ownership of the folder that is mapped to the volume. But Balena didn’t change the ownership of the folder behind the volume once it was created.
Solution was to “Purge Data” and re-create the volume with non-root user permissions.