Trouble with udev

Product support
1

I have a USB camera that is controlled via a userspace libusb driver. I use a udev rule to set the permissions of the device node to allow access, but I am getting inconsistent behaviour. I am using the systemd init system (ENV INITSYSTEM on). If the camera is plugged in when the application container starts, the rules are correctly applied, but they are not applied on hotplug with the container running. To investigate, I ran the following commands in the container:

# udevadm control --log-priority=debug
# journalctl -f

When I plug in the camera, I see this which looks promising:

Jan 18 08:29:00 d46c446 systemd-udevd[1382]: GROUP 44 /etc/udev/rules.d/99-raytrix.rules:1
Jan 18 08:29:00 d46c446 systemd-udevd[1382]: MODE 0660 /etc/udev/rules.d/99-raytrix.rules:1
Jan 18 08:29:00 d46c446 systemd-udevd[1382]: handling device node '/dev/bus/usb/004/019', devnum=c189:402, mode=0660, uid=0, gid=44
Jan 18 08:29:00 d46c446 systemd-udevd[1382]: set permissions /dev/bus/usb/004/019, 020660, uid=0, gid=44

However, the permissions are not actually set.

# ls -l /dev/bus/usb/004
total 0
crw-rw-r-- 1 root root 189, 384 Jan 18 08:17 001
crw-rw-r-- 1 root root 189, 402 Jan 18 08:29 019

I can manually set the group and permissions, with

# chown :video /dev/bus/usb/004/019
# chmod 660 /dev/bus/usb/004/019
# ls -l /dev/bus/usb/004/
total 0
crw-rw-r-- 1 root root  189, 384 Jan 18 08:52 001
crw-rw---- 1 root video 189, 403 Jan 18 09:19 019

it seems systemd-udevd can’t and I haven’t found an error report indicating why. Is this expected to work? Do I need to do some other configuration of systemd-udevd?

4

Hi @russel, this is indeed strange. I’ve chatted with @abrodersen and we are wondering if this might be a race condition between udev on the host and in the container.

Could you log into the host OS (which you can do either by getting a development version of resinOS or through the web console directly on the most recent releases) and including the output of journalctl as you did above?

If this is a race condition, it might explain why this occurs with a hotplug but not on boot.

Thanks!

5

Indeed, this seems to be a race condition. I ran journalctl in both environemts (host os and app container) and saw this in the Hoost OS

Jan 23 20:05:57 c14dac2 systemd-udevd[7259]: handling device node '/dev/bus/usb/004/004', devnum=c189:387, mode=0664, uid=0, gid=0

And this in the App container

Jan 23 20:05:57 c14dac2 systemd-udevd[1067]: set permissions /dev/bus/usb/004/004, 020660, uid=0, gid=44

So to the available time resolution, who knows what happens? What next?

8

Hi @russel, just out of curiosity, does running udevadm trigger from inside the container cause the device to be properly set up after this race condition occurs? I’m just trying to narrow down where we could potentially address this.

9

Good suggestion! just running udevadm trigger from the container causes udev to run in both the host and the container. It looks like it checks all the devices mentioned in the acpi table and the ttys but it does not appear to check any USB devices. Running

# udevadm trigger --name-match=/dev/bus/usb/004/028

to specifically target the device I am interested in causes a udev change action to be emitted which the host os uses to check that the permissions are still what it set (they are) and no change in the container. This is not unexpected as my rule is explicitly for an add event. Running

# udevadm trigger -c add --name-match=/dev/bus/usb/004/028

To explicitly trigger an add event causes both the host and the container to run all the relevant rules, unfortunately it seems the host still always runs last and its permissions take effect.

16

Thanks for verifying the udevadm behavior. We’ll dig in some more on this and see if we can find a solution for you shortly.

21

Hi @russel,

Can you please do 2 tests so we can narrow down what is causing this issue? In both test, you don’t need to run udevadm trigger yourself so you should remove it from your start script.

  • In the first test, can you try to turn off the init system ENV INITSYSTEM off and see how it goes?
  • For the second one, can you try to push a non-resin base images like debian:jessie-slim. That container shouldn’t have udev setup from resin.

By doing those tests, we will know its some weird interaction within the entry point of the resin.io base images with the hostOS.

22

OK, here are the tests I ran:

Using resin/intel-nuc-debian:stretch, I tried INITSYSTEM on and got the race behavior described above. with INITSYSTEM off, udevadm monitor shows the add event, but none of the rules in the container run and the permissions are as expected by the host. ps shows that systemd-udevd is running in the app container.

Using debian:jessie-slim, no udev packages are installed by default. Installing them allows udevadm to run, and monitor shows the add event, but no rules are executed. udevd is not running and attempting to manually start it with `/etc/init.d/udev/ start’ results in

root@da67e76:~# /etc/init.d/udev start
[warn] udev does not support containers, not started ... (warning).

Project is here russel/udevTest, I have granted support access to damp-frost if you want to take a look.

24

Hi @russel, can you try again with INITSYSTEM on and please remove ACTION=="add" from your udev rules?

I don’t have a camera but managed to reproduce the issue on my end with an usb flash drive. So I think the problem is, the default udev rule on the HostOS set the permissions for your device again on add event when the device is connected to the board and remove that action from the rule will fix it. It works on my end with the usb so please try it and let me know how it goes.

25

OK, this looks promising. I removed the ACTION restriction from my rule. When I hotplug the device, my rules is run, but the it appears the host still forces the permissions. However, now running udevadm trigger results in the correct permissions. Progress!

26

Thanks, please keep us posted!

28

Hmm, so what do you imagine the solution is here? I can’t reasonably run udevamd trigger myself as that just moves the race condition around. I would really like to have udev set the permissions correctly on hotplug. I see several prossibilities:

  • Ensure all application container udev rules run after all host udev rules
  • Allow custom udev rules to be added to the host OS, similar to the way network connections can be added with /boot/system-connections
  • Use some method other than udev (dbus?) to notify the application that the host has added a new device.
  • Detect incorrect permissions and fix them independently of udev

The first one seems like the correct solution, the others might be made to work, but without a notification and coordination between host and application there is always the danger of a race to set permissions. The results of my latest test indicate that there are some scenarios where the application container does correctly set the permissions, so I have hope for the first one. What can I do to help get this working correctly?

29

Hey @russel,

You can setup your udev rule like this

ACTION=="ADD", SUBSYSTEM=="usb", ATTR{idVendor}=="0951", ATTR{idProduct}=="1665" , RUN+="/udev.sh"
SUBSYSTEM=="usb", ATTR{idVendor}=="0951", ATTR{idProduct}=="1665",MODE="0667"

and here is the udev.sh

#!/bin/bash

sleep 1
udevadm trigger

So the first rule will be run on ADD event which put the container to sleep for a second (you can change the time here to make sure the udev rule on HostOS executed first) then automatically run udevadm trigger to load the second rule which correctly set the permissions. I’ve tried this on my end and it works fine.

30

This fixed my problem. I needed to spell it ACTION="add", uppercase did not work. I ended up using a pair of rules like this:

ACTION=="add", SUBSYSTEM=="usb", ATTR{idVendor}=="2951", ATTR{idProduct}=="0804", RUN+="/udev_pause.sh"
ACTION=="change", SUBSYSTEM=="usb", ATTR{idVendor}=="2951", ATTR{idProduct}=="0804", GROUP="video", MODE="0660"

With exactly the script you suggested. Thanks for your help!