I am trying to mount a USB stick as storage device inside one of my containers. I followed the example at https://github.com/balena-io-playground/balena-storage and my container has all the necessary files and commands in the dockerfile (including the UDEV env variable set to on). I am also running the alpine balena base image (python flavor), the container is running in privileged mode.
If I monitor for udev events with udevadm monitor --property, I see a bunch of KERNEL events when I plug and unplug the device, which seems to indicate it’s properly detected. But I see no UDEV event, although I checked and the rules file is present at /etc/udev/rules.d/usb.rules.
The scripts are also present at their respective locations, and have execution permissions. Here’s the relevant event that should trigger the mount script:
Thanks for the reply. I’m trying to manually run the mount script commands (I indeed use the unmodified mount/unmount scripts), but this command fails (trying to get the FS type):
$ udevadm info -n /dev/sda1
device node not found
If I proceed anyway (the stick is formatted as FAT32):
$ mkdir /mnt/storage
$ mount -t vfat -o rw /dev/sda1 /mnt/storage
mount: /mnt/storage: special device /dev/sda1 does not exist.
The UDEV monitor shows that the DEVNAME should be /dev/sda1 so I don’t really understand the error.
Output of fdisk -l:
Device Boot Start End Sectors Size Id Type
/dev/mmcblk0p1 * 8192 90111 81920 40M c W95 FAT32 (LBA)
/dev/mmcblk0p2 90112 729087 638976 312M 83 Linux
/dev/mmcblk0p3 729088 1368063 638976 312M 83 Linux
/dev/mmcblk0p4 1368064 31116287 29748224 14.2G f W95 Ext'd (LBA)
/dev/mmcblk0p5 1376256 1417215 40960 20M 83 Linux
/dev/mmcblk0p6 1425408 31116287 29690880 14.2G 83 Linux
I tried with 2 differents usb sticks but the result is the same.
They suggest binding /dev from the host (I can indeed see /dev/sda1 on the host) which could be a workaround.
Some people in that thread say that, failing to bind /dev, the mounting only works if the USB is inserted before the container is spun up, which completely defies the purpose of auto-mounting.
How is this problem addressed in the example I linked in my first post?
Hello @vbersier, thank you for the detailed explanation. I deployed the balena-storage code on my RaspberryPi 3 and it works as expected. I also used FAT32 formatted USB stick. I could mount and unmount the stick fine anytime (after the container is spun up). So the auto monuting works.
First of all, have you tried the example repo by itself on your device and USB stick?
It’d be great if you do that. So we rule out any hardware issue. If the example repo works, we could also say that the problem is based your own setup.
Next, this message shows that the device cannot read the partition of the USB stick:
$ udevadm info -n /dev/sda1
device node not found
What do you see when you try to read the device itself? udevadm info -n /dev/sda
If the device is able to read the device, there might be an issue with the partition. So you could try to remount the partition by eject -t /dev/sda.
Thanks to the suggestions, I was able to rule out any problem with the hardware or the partitions on the usb stick. I ran the balena-storage repo directly and mounting works.
So the issue is with my particular setup. In order to isolate any effects of other containers, I created a test app with a single container which is supposed to mount the USB. Still, it doesn’t work with this container.
The differences I can spot are the following:
I am using the base image balenalib/aarch64-alpine-python:3.7-3.11-run Ruled out by changing the example repo to use that image and it still works
I have exposed ports and build args in the docker-compose
I use a multi-staged build approach
my python script interacts with the filesystem thanks to watchdog (actually, I just load the module but I don’t setup any observer in this testing case)
FROM balenalib/%%BALENA_ARCH%%-alpine-python:3.7-3.11-run AS base
ENV PYSETUP_PATH="/opt/pysetup" \
VENV_PATH="/opt/pysetup/.venv"
ENV PATH="$VENV_PATH/bin:$PATH"
ENV UDEV=on
COPY udev/usb.rules /etc/udev/rules.d/usb.rules
RUN install_packages findmnt util-linux grep
WORKDIR /usr/src
COPY scripts scripts
RUN chmod +x scripts/*
WORKDIR /app
RUN install_packages tzdata && \
cp /usr/share/zoneinfo/Europe/Zurich /etc/localtime && \
echo "Europe/Zurich" > /etc/timezone && \
apk del tzdata
COPY --from=copy-src $PYSETUP_PATH $PYSETUP_PATH # copy-src is a previous stage image
COPY --from=copy-src /app/tests ./tests
ENTRYPOINT ["smsl_daemon"]
I opened a shell in the container and checked that all the files are present where they should (scripts and udev rules). I checked that the UDEV environment variable is set to “on” too.
The Dockerfile.template in the balenaStorage example kicks off the UDEV storage with CMD [ "/bin/bash", "/usr/src/scripts/idle.sh" ]. Is your ENTRYPOINT triggering something similar?
Yes, my script also runs a loop to monitor for other events (filesystem, time-based etc) so the container keeps running. As I can see, the idle script from the example doesn’t do anything special besides keeping the container running. My best guess at the moment is a conflict with the python library watchdog but I’ll confirm by replacing my python entrypoint with the idle script.
I think I identified that one of the USB ports on the raspberry pi doesn’t work, I’m currently investigating.
EDIT: actually it works on all ports, but only the second time that I plug it in, it seems. Still investigating
Scratch all that, I have found the culprit. It seems this technique doesn’t work when using ENTRYPOINT. If I use CMD instead, it works. Now I want to know why? I am kinda relying on the ENTRYPOINT mechanism to pass additional arguments to the container in some cases, so I would prefer to keep it like shown above.
Unfortunately that would not be allowed by my company’s policy. I can tell you that I’m using watchdog like stated above (but the problem happens even if I don’t add any Observers) and also running a small websocket server with tornado which is actually what keeps the script running in a loop.
Is there a default ENTRYPOINT in base balena images that would be overriden when I set my own ENTRYPOINT and would prevent UDEV from working properly? I found this which seems to be the default entrypoint, but I don’t know which parts are vital to UDEV working.
If you take a look at this image for example, it says:
Working with dynamically plugged devices: each balenalib base image has a default ENTRYPOINT which is defined as ENTRYPOINT [“/usr/bin/entry.sh”], it checks if the UDEV flag is set to true or not (by adding ENV UDEV=1) and if true, it will start udevd daemon and the relevant device nodes in the container /dev will appear.
This explains why overriding entrypoint caused udev problems.
That explains it indeed. I need to make my own entrypoint script that mimicks the default one and then passes the arguments to my smsl-daemon script I guess. Or does the default entrypoint pass arguments to the CMD command? Thanks for the help