Docker container cannot access dynamically plugged USB devices

Product support balenaOS
1

I am trying to access a USB device (camera) inside my container running on Balena OS 2.24.0 rev2 on a Jetson TX2. It is a multi-container application and I am running the respective container with

    privileged: true
    devices:
      - '/dev:/dev'

If the camera is plugged in during container launch I can access it. If I unplug it and replug it I cannot anymore.
For a normal Docker container the solution is to mount /dev: -v /dev:/dev in combination with running privileged. Then devices plugged in after the container is started are supported.
However, on BalenaOS mount binding is not allowed. How could I go about it?

2

Would it be a possibility to add another flag that will bind mount /dev to /dev?

3

I haven’t had to use the devices specification, wasn’t even familiar with it. On Intel NUC (ubuntu/debian) accessing /dev/ttyACM0, etc just works. except when the device locks up, of course.

4

For a normal Docker container the devices specification allows a bit more fine grained control than the broad privileged spec.

On Balena single containers always run in privileged mode, so if your hardware does not change after start of the container you should be able to access it.
My issue is that hardware that is plugged in after the container is started is not accessible.

5

Sorry to hear “issue is that hardware that is plugged in after the container is started is not accessible”
That doesn’t appear to be an issue with the Intel NUCs.

There is a discussion about restarting a container from inside of the container or from another system by using the https api: Restart container every 24 hours

6

Hi Jason,
that sounds like an interesting workaround, thank you.

Still a rather annoying way of doing it though.

7

Ideally, I would like to have a another label to add to a multicontainer docker-compose.yaml that bind mounds /dev:/dev.

I’ve searched for the implementation of the existing labels on github, but couldn’t find it. Could somebody point me to the right spot?

8

Hi Rapha,

Yes I agree, especially since we often don’t have reliable network access to our devices. Additionally we have found that when the USB connection to our device fails we have to physically power cycle the host system. Resets, reboots, device disconnections (physical) are not sufficient. Likely a device driver/usb chip issue.

So we’re looking into watchdogs and system power cycling.

9

Hi @jason10,
for us a simple restart of the container/service is sufficient.

10

It appears that the fixed bind-mounts are defined in the https://github.com/balena-os/meta-balena/blob/master/meta-resin-common/recipes-support/resin-mounts/resin-mounts.inc file.

I will try to add a custom rule for mounting /dev. A label would still be the nicer option, but I am not sure which dev can advise on that.

@petrosagg are you the right person to ask on how to ensure USB devices are accessible if plugged in after the Balena container is started and running?

11

Can you use udev rules? In my Dockerfile.template I add a rule that identifies the video side of our usb device to appear as /dev/video_boson:

RUN echo 'SUBSYSTEM=="video4linux", ATTR{name}=="Boson", SYMLINK+="video_boson"' > /etc/udev/rules.d/boson.rules

More advice on using udev at http://www.reactivated.net/writing_udev_rules.html

12

Hi jason10,

I’m interested in how you got your udev rules to work within a docker container. When I try echoing the rule like you’ve suggested I get an error that the /etc/udev/rules.d directory doesn’t exist.

I’m working on a project involving several containers that need to be able to communicate with hardware connected to my Ubuntu 18.04 host. I have set up udev rules to give my hardware aliases when they are connected to ensure that the path to the hardware will never change.

I can get the containers the see and communicate with the HW via the udev alias using a device tag in my docker-compose file. But if the containers are brought up before the hardware is powered on they will error out saying that a device with my alias doesn’t exist. I’m trying to design the containers to be robust enough to come up, wait for the hw to be connected, open a session with it, and even be able to recover if it is ever disconnected.

If I set the containers to be privileged instead of using a device tag, I can communicate with the hardware using their non-aliased paths (ttyUSB0 etc.), but when I try to open a session with the udev alias I created it can’t find the device.

Is there a way to communicate with a device’s udev alias from a container without using a device tag? Or is there a way to recover from the container crashing when the path in the device tag isn’t found?

Your solution looks like it may be what I’m looking for but I haven’t gotten it to work so far.

Thanks!

13

Hey @jaredro

This is for an Intel NUC, we’re connecting the devices before we power on, but they also work if we connect them after boot.

I’m not completely sure what else I had to add to my Dockerfile.template to get the udev rules to work. Using git pickaxe (git log -p -S “video4linux” …) to try to find what I added:

apt-get libsm6 libext6

# switch on systemd init system in container
ENV INITSYSTEM on
RUN echo 'SUBSYSTEM=="video4linux", ATTR{name}=="Boson", SYMLINK+="video_boson"' > /etc/udev/rules.d/boson.rules

If that doesn’t help, let me know!

14

@jason10 I found the solution for our needs:

I had not seen the UDEV=1 env one can set in the balena base images:
https://www.balena.io/docs/reference/base-images/base-images/#working-with-dynamically-plugged-devices

With that one set, as well as privileged: true plugging and replugging works. The service in the docker-compose.yml looks like this:

  cameracontroller:
    image:  balenalib/jetson-tx2-ubuntu:melodic
    environment:
      - UDEV=1
    restart: never
    privileged: true
    devices:
      - '/dev:/dev'
    command: /bin/bash
1 Like
15

Excellent!

16

One small addendum to my earlier solution:

  • If you are building your own image based on one iof the balena images: make sure to not overwrite the ENTRYPOINT in your Dockerfile as this would disable the udev.
17

Hey @rapha, UDEV=1 is not outdated, see in the source code of the entry script here

The ENTRYPOINT indeed shouldn’t be rewritten, as it’s in that entry sript that we are setting up the dymanic plugging (for the dev file system and udev properly)

Based on this, I think https://www.balena.io/docs/reference/base-images/base-images/#working-with-dynamically-plugged-devices should be up to date, and the important part was that your container needed privileged: true, right? Or anything else is out of data in that page?

Thanks a lot for the lot of feeback!

19

Hey @imrehg, thanks for the quick feedback. My main issue was that we overwrote the entrypoint.

The https://www.balena.io/docs/reference/base-images/base-images/#working-with-dynamically-plugged-devices section is a little confusing:

It mentions how to enable dynamic repluggin (showing UDEV=1), but follows that advice with Major Changes where you talk about UDEV=off.

Could you clarify that page to say that both true, 1, on will work and also point out that one should not overwrite the entrypoint :wink:

Thanks for the quick response!

20

Yeah, that clarification is due indeed, will mark it as our docs improvement. The point being is that by default UDEV is no longer turned on (which is the same as setting it to off. On the other hand, either setting it to 1, true or on will turn it on, to keep with backwards compatibility.

Overwriting the ENTRYPOINT is always going to mess with base images (any kind of base images, in my experience) but I guess it’s worth clarifying on our side too not to do that. Thanks @rapha!

1 Like
22

Hi,

I am also facing the same issue as I have defined udev=1 and device volumes. Lets see my docker compose file,

version: '2'
volumes: 
  node-data:
services:
  wpe:
    restart: always
    build: ./
    privileged: true
    ports:
      - 8080:8080
  nodejs:
      build: './nodejs' # specify the directory of the Dockerfile
      environment:
      - UDEV=1
      ports:
        - "3000:3000" #specify ports forewarding
      volumes:
        - 'node-data:/node-data'
      devices:
      - '/dev:/dev'
      privileged: true
      restart: always

Can anyone help me?

Thanks,
Nikunj