Auto-mount external storage device in container (UDEV)

Hello there.

I am trying to mount a USB stick as storage device inside one of my containers. I followed the example at https://github.com/balena-io-playground/balena-storage and my container has all the necessary files and commands in the dockerfile (including the UDEV env variable set to on). I am also running the alpine balena base image (python flavor), the container is running in privileged mode.

If I monitor for udev events with udevadm monitor --property, I see a bunch of KERNEL events when I plug and unplug the device, which seems to indicate it’s properly detected. But I see no UDEV event, although I checked and the rules file is present at /etc/udev/rules.d/usb.rules.

Here’s the rules file:

ACTION=="add", SUBSYSTEM=="block", ENV{DEVTYPE}=="partition", RUN+="/bin/sh -c '/usr/src/scripts/mount.sh'"
ACTION=="remove", SUBSYSTEM=="block", ENV{DEVTYPE}=="partition", RUN+="/bin/sh -c '/usr/src/scripts/unmount.sh'"

The scripts are also present at their respective locations, and have execution permissions. Here’s the relevant event that should trigger the mount script:

KERNEL[321895.727976] add      /devices/platform/soc/3f980000.usb/usb1/1-1/1-1.3/1-1.3:1.0/host0/target0:0:0/0:0:0:0/block/sda/sda1 (block)
ACTION=add
DEVNAME=/dev/sda1
DEVPATH=/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.3/1-1.3:1.0/host0/target0:0:0/0:0:0:0/block/sda/sda1
DEVTYPE=partition
MAJOR=8
MINOR=1
PARTN=1
SEQNUM=7672
SUBSYSTEM=block

But nothing happens, nothing in /mnt and I don’t have any log file inside /usr/src, nor do I see any UDEV events in the monitoring.

Any help would be greatly appreciated! Thanks

Hi,

Can you please let us know if you use the same scripts as our example project and did you try to mount your device manually to see if it works?

Thanks for the reply. I’m trying to manually run the mount script commands (I indeed use the unmodified mount/unmount scripts), but this command fails (trying to get the FS type):

$ udevadm info -n /dev/sda1
device node not found

If I proceed anyway (the stick is formatted as FAT32):

$ mkdir /mnt/storage
$ mount -t vfat -o rw /dev/sda1 /mnt/storage
mount: /mnt/storage: special device /dev/sda1 does not exist.

The UDEV monitor shows that the DEVNAME should be /dev/sda1 so I don’t really understand the error.

Output of fdisk -l:

Device         Boot   Start      End  Sectors  Size Id Type
/dev/mmcblk0p1 *       8192    90111    81920   40M  c W95 FAT32 (LBA)
/dev/mmcblk0p2        90112   729087   638976  312M 83 Linux
/dev/mmcblk0p3       729088  1368063   638976  312M 83 Linux
/dev/mmcblk0p4      1368064 31116287 29748224 14.2G  f W95 Ext'd (LBA)
/dev/mmcblk0p5      1376256  1417215    40960   20M 83 Linux
/dev/mmcblk0p6      1425408 31116287 29690880 14.2G 83 Linux

I tried with 2 differents usb sticks but the result is the same.

Found this related topic https://github.com/moby/moby/issues/16160

They suggest binding /dev from the host (I can indeed see /dev/sda1 on the host) which could be a workaround.

Some people in that thread say that, failing to bind /dev, the mounting only works if the USB is inserted before the container is spun up, which completely defies the purpose of auto-mounting.

How is this problem addressed in the example I linked in my first post?

Hello @vbersier, thank you for the detailed explanation. I deployed the balena-storage code on my RaspberryPi 3 and it works as expected. I also used FAT32 formatted USB stick. I could mount and unmount the stick fine anytime (after the container is spun up). So the auto monuting works.

First of all, have you tried the example repo by itself on your device and USB stick?
It’d be great if you do that. So we rule out any hardware issue. If the example repo works, we could also say that the problem is based your own setup.

Next, this message shows that the device cannot read the partition of the USB stick:

$ udevadm info -n /dev/sda1
device node not found

What do you see when you try to read the device itself?
udevadm info -n /dev/sda

If the device is able to read the device, there might be an issue with the partition. So you could try to remount the partition by eject -t /dev/sda.

Hello, thanks for the reply.

Thanks to the suggestions, I was able to rule out any problem with the hardware or the partitions on the usb stick. I ran the balena-storage repo directly and mounting works.

So the issue is with my particular setup. In order to isolate any effects of other containers, I created a test app with a single container which is supposed to mount the USB. Still, it doesn’t work with this container.

The differences I can spot are the following:

  • I am using the base image balenalib/aarch64-alpine-python:3.7-3.11-run Ruled out by changing the example repo to use that image and it still works
  • I have exposed ports and build args in the docker-compose
  • I use a multi-staged build approach
  • my python script interacts with the filesystem thanks to watchdog (actually, I just load the module but I don’t setup any observer in this testing case)

Here is my docker-compose:

version: '2.1'
networks: {}
volumes:
  exchange:
  settings:
services:
  smsl-daemon:
    build:
      context: smsl_daemon/.
      args:
        TESTING: "0"
    restart: always
    ports:
      - "139:139"
      - "9090:9090"
    privileged: true

Here is the last stage of my Dockerfile:

FROM balenalib/%%BALENA_ARCH%%-alpine-python:3.7-3.11-run AS base
ENV PYSETUP_PATH="/opt/pysetup" \
    VENV_PATH="/opt/pysetup/.venv"
ENV PATH="$VENV_PATH/bin:$PATH"
ENV UDEV=on
COPY udev/usb.rules /etc/udev/rules.d/usb.rules
RUN install_packages findmnt util-linux grep
WORKDIR /usr/src
COPY scripts scripts
RUN chmod +x scripts/*

WORKDIR /app
RUN install_packages tzdata && \
    cp /usr/share/zoneinfo/Europe/Zurich /etc/localtime && \
    echo "Europe/Zurich" > /etc/timezone && \
    apk del tzdata
COPY --from=copy-src $PYSETUP_PATH $PYSETUP_PATH # copy-src is a previous stage image
COPY --from=copy-src /app/tests ./tests

ENTRYPOINT ["smsl_daemon"]

I opened a shell in the container and checked that all the files are present where they should (scripts and udev rules). I checked that the UDEV environment variable is set to “on” too.

Hi,

The Dockerfile.template in the balenaStorage example kicks off the UDEV storage with CMD [ "/bin/bash", "/usr/src/scripts/idle.sh" ]. Is your ENTRYPOINT triggering something similar?

John

Yes, my script also runs a loop to monitor for other events (filesystem, time-based etc) so the container keeps running. As I can see, the idle script from the example doesn’t do anything special besides keeping the container running. My best guess at the moment is a conflict with the python library watchdog but I’ll confirm by replacing my python entrypoint with the idle script.

I think I identified that one of the USB ports on the raspberry pi doesn’t work, I’m currently investigating.

EDIT: actually it works on all ports, but only the second time that I plug it in, it seems. Still investigating

Scratch all that, I have found the culprit. It seems this technique doesn’t work when using ENTRYPOINT. If I use CMD instead, it works. Now I want to know why? I am kinda relying on the ENTRYPOINT mechanism to pass additional arguments to the container in some cases, so I would prefer to keep it like shown above.

Hi, can you show us your smsl_daemon entrypoint code?

Unfortunately that would not be allowed by my company’s policy. I can tell you that I’m using watchdog like stated above (but the problem happens even if I don’t add any Observers) and also running a small websocket server with tornado which is actually what keeps the script running in a loop.

Is there a default ENTRYPOINT in base balena images that would be overriden when I set my own ENTRYPOINT and would prevent UDEV from working properly? I found this which seems to be the default entrypoint, but I don’t know which parts are vital to UDEV working.

If you take a look at this image for example, it says:

Working with dynamically plugged devices: each balenalib base image has a default ENTRYPOINT which is defined as ENTRYPOINT ["/usr/bin/entry.sh"], it checks if the UDEV flag is set to true or not (by adding ENV UDEV=1) and if true, it will start udevd daemon and the relevant device nodes in the container /dev will appear.

This explains why overriding entrypoint caused udev problems.

That explains it indeed. I need to make my own entrypoint script that mimicks the default one and then passes the arguments to my smsl-daemon script I guess. Or does the default entrypoint pass arguments to the CMD command? Thanks for the help

Great, let us know if you have more questions. Have a good day!