Docker container cannot access dynamically plugged USB devices

docker

#1

I am trying to access a USB device (camera) inside my container running on Balena OS 2.24.0 rev2 on a Jetson TX2. It is a multi-container application and I am running the respective container with

    privileged: true
    devices:
      - '/dev:/dev'

If the camera is plugged in during container launch I can access it. If I unplug it and replug it I cannot anymore.
For a normal Docker container the solution is to mount /dev: -v /dev:/dev in combination with running privileged. Then devices plugged in after the container is started are supported.
However, on BalenaOS mount binding is not allowed. How could I go about it?


#2

Would it be a possibility to add another flag that will bind mount /dev to /dev?


#3

I haven’t had to use the devices specification, wasn’t even familiar with it. On Intel NUC (ubuntu/debian) accessing /dev/ttyACM0, etc just works. except when the device locks up, of course.


#4

For a normal Docker container the devices specification allows a bit more fine grained control than the broad privileged spec.

On Balena single containers always run in privileged mode, so if your hardware does not change after start of the container you should be able to access it.
My issue is that hardware that is plugged in after the container is started is not accessible.


#5

Sorry to hear “issue is that hardware that is plugged in after the container is started is not accessible”
That doesn’t appear to be an issue with the Intel NUCs.

There is a discussion about restarting a container from inside of the container or from another system by using the https api: Restart container every 24 hours


#6

Hi Jason,
that sounds like an interesting workaround, thank you.

Still a rather annoying way of doing it though.


#7

Ideally, I would like to have a another label to add to a multicontainer docker-compose.yaml that bind mounds /dev:/dev.

I’ve searched for the implementation of the existing labels on github, but couldn’t find it. Could somebody point me to the right spot?


#8

Hi Rapha,

Yes I agree, especially since we often don’t have reliable network access to our devices. Additionally we have found that when the USB connection to our device fails we have to physically power cycle the host system. Resets, reboots, device disconnections (physical) are not sufficient. Likely a device driver/usb chip issue.

So we’re looking into watchdogs and system power cycling.


#9

Hi @jason10,
for us a simple restart of the container/service is sufficient.


#10

It appears that the fixed bind-mounts are defined in the https://github.com/balena-os/meta-balena/blob/master/meta-resin-common/recipes-support/resin-mounts/resin-mounts.inc file.

I will try to add a custom rule for mounting /dev. A label would still be the nicer option, but I am not sure which dev can advise on that.

@petrosagg are you the right person to ask on how to ensure USB devices are accessible if plugged in after the Balena container is started and running?


#11

Can you use udev rules? In my Dockerfile.template I add a rule that identifies the video side of our usb device to appear as /dev/video_boson:

RUN echo 'SUBSYSTEM=="video4linux", ATTR{name}=="Boson", SYMLINK+="video_boson"' > /etc/udev/rules.d/boson.rules

More advice on using udev at http://www.reactivated.net/writing_udev_rules.html


#12

Hi jason10,

I’m interested in how you got your udev rules to work within a docker container. When I try echoing the rule like you’ve suggested I get an error that the /etc/udev/rules.d directory doesn’t exist.

I’m working on a project involving several containers that need to be able to communicate with hardware connected to my Ubuntu 18.04 host. I have set up udev rules to give my hardware aliases when they are connected to ensure that the path to the hardware will never change.

I can get the containers the see and communicate with the HW via the udev alias using a device tag in my docker-compose file. But if the containers are brought up before the hardware is powered on they will error out saying that a device with my alias doesn’t exist. I’m trying to design the containers to be robust enough to come up, wait for the hw to be connected, open a session with it, and even be able to recover if it is ever disconnected.

If I set the containers to be privileged instead of using a device tag, I can communicate with the hardware using their non-aliased paths (ttyUSB0 etc.), but when I try to open a session with the udev alias I created it can’t find the device.

Is there a way to communicate with a device’s udev alias from a container without using a device tag? Or is there a way to recover from the container crashing when the path in the device tag isn’t found?

Your solution looks like it may be what I’m looking for but I haven’t gotten it to work so far.

Thanks!


#13

Hey @jaredro

This is for an Intel NUC, we’re connecting the devices before we power on, but they also work if we connect them after boot.

I’m not completely sure what else I had to add to my Dockerfile.template to get the udev rules to work. Using git pickaxe (git log -p -S “video4linux” …) to try to find what I added:

apt-get libsm6 libext6

# switch on systemd init system in container
ENV INITSYSTEM on
RUN echo 'SUBSYSTEM=="video4linux", ATTR{name}=="Boson", SYMLINK+="video_boson"' > /etc/udev/rules.d/boson.rules

If that doesn’t help, let me know!


#14

@jason10 I found the solution for our needs:

I had not seen the UDEV=1 env one can set in the balena base images:
https://www.balena.io/docs/reference/base-images/base-images/#working-with-dynamically-plugged-devices

With that one set, as well as privileged: true plugging and replugging works. The service in the docker-compose.yml looks like this:

  cameracontroller:
    image:  balenalib/jetson-tx2-ubuntu:melodic
    environment:
      - UDEV=1
    restart: never
    privileged: true
    devices:
      - '/dev:/dev'
    command: /bin/bash

#15

Excellent!