libudev cannot monitor udev events inside container

We are running a GStreamer based pipeline inside balena including the hot-plugging of cameras. The container is based off balenalib and the hot-plugging part works fine with devices showing up inside the container.

GStreamer has the gst-device-monitor-1.0 -f command, which can list the current audio video devices connected as well as continuously listen on changes to the devices

The command uses the libgudev library (glib binding to libudev) to listen on udev device changes, which it then filters and acts upon.

Outside of a container this works perfectly fine. Whenever a camera is plugged in while gst-device-monitor-1.0 -f is running, it prints information about the device.

Inside the container running gst-device-monitor -f will correctly identify and analyse all currently connected devices, but it will not monitor devices as they are plugged in. Re-running it after plugging in new hardware will also work as expected.

Directly monitoring the udev events using udevadm monitor inside the container works fine.

This can be reproduced by building the following image:

# docker build -t gst-device-monitor-test . -f device-monitor.dockerfile

FROM balenalib/generic-ubuntu:focal
LABEL org.opencontainers.image.source https://gitlab.com/aivero/prop/workspace

# Prevent apt-get prompting for input
ENV DEBIAN_FRONTEND noninteractive

# Download and install BSP binaries for L4T 32.4.2
RUN \
    apt-get update && apt-get install -y --no-install-recommends \
    gstreamer1.0-plugins-good \
    gstreamer1.0-plugins-base-apps \
    && rm -rf /var/lib/apt/lists/* 

ENV UDEV=1

Then run it as docker run --rm -ti --privileged -v /sys:/sys -e UDEV=1 gst-device-monitor-test:latest /bin/bash

Inside the container run gst-device-monitor -f and plug in some audio/video device such as a webcam.

I would like to know if there are any changes to the containers that might interfere with the libudev library and that might prevent the callbacks from being called when a device is plugged in?

Can reproduce this further. This small c code example using libudev and compiled inside the container also does not react to device replugs:

#include <stdio.h>
#include <unistd.h>
#include <libudev.h>

int main()
{
    struct udev *udev;
    struct udev_device *dev;
    struct udev_monitor *mon;
    int fd;

    udev = udev_new();
    if (!udev) {
        fprintf(stderr, "Can't create udev\n");
        return 1;
    }

    mon = udev_monitor_new_from_netlink(udev, "udev");
    udev_monitor_filter_add_match_subsystem_devtype(mon, "video4linux", NULL);
    udev_monitor_enable_receiving(mon);
    fd = udev_monitor_get_fd(mon);

    while (1) {
        fd_set fds;
        struct timeval tv;
        int ret;

        FD_ZERO(&fds);
        FD_SET(fd, &fds);
        tv.tv_sec = 0;
        tv.tv_usec = 0;

        ret = select(fd+1, &fds, NULL, NULL, &tv);
        if (ret > 0 && FD_ISSET(fd, &fds)) {
            dev = udev_monitor_receive_device(mon);
            if (dev) {
                printf("I: ACTION=%s\n", udev_device_get_action(dev));
                printf("I: DEVNAME=%s\n", udev_device_get_sysname(dev));
                printf("I: DEVPATH=%s\n", udev_device_get_devpath(dev));
                printf("---\n");

                udev_device_unref(dev);
            }
        }
    }
    udev_unref(udev);

    return 0;
}

This required me to install libudev-dev and build-essential on the container.

1 Like

Hi,

It is not clear if you are running your containers on your developer machine OS or on a BalenaOS device (from your command-line with docker, I lean towards the dev machine side).

The UDEV=1 should work in a Balena project (run with the same command on you HostOS on your target device, replacing docker with balena).

If you are running on your computer, we don’t have guidance on how to use UDEV on Docker.

I will try to reproduce your behavior on one of my device to see if I am missing something.

Hello again,

I tried this on my device and succeed to reproduce your problem.
It turns out that you probably should install udev and start the service on your container.

I could do it with your example C code and following Dockerfile

FROM debian:bullseye

RUN apt-get update &&\
    apt-get install -y build-essential libudev-dev udev

WORKDIR /test

COPY code.c .
RUN gcc -o code code.c -ludev

CMD service udev restart && ./code

Let me know if it solves your issue.

Hi @wolvi-lataniere
You are right, the examples we posted were tested on x86, however the problem is on arm64 (Jetson TX2).

I tried the essentials of your solution:

  • Installing udev
  • restarting the udev service

Upon restarting the udev service I get an error/warning:
udev does not support containers, not started

I found docker - Why udev init script default disable container support while in fact it works? - Stack Overflow which claims that one can patch the udev service script and it works anyway.

I’ve patched the udev service script in the container and I get events in the container.
However, events are now continuously spamming. My camera is constantly being reported as attached and detached:

...
03.10.22 14:04:34 (+0200)  dcd  0:57:32.357796800    97   0x557da72f00 INFO         deepcore-daemon deepcore-daemon/src/devices.rs:115:deepcore_daemon::devices: Camera detached RealsenseD415-928222061521
03.10.22 14:04:34 (+0200)  dcd  0:57:32.585963520    97   0x557da72f00 INFO         deepcore-daemon deepcore-daemon/src/devices.rs:100:deepcore_daemon::devices: Camera attached RealsenseD415-928222061521
03.10.22 14:04:35 (+0200)  dcd  0:57:33.711682944    97   0x557da72f00 INFO         deepcore-daemon deepcore-daemon/src/devices.rs:115:deepcore_daemon::devices: Camera detached RealsenseD415-928222061521
03.10.22 14:04:35 (+0200)  dcd  0:57:33.962839616    97   0x557da72f00 INFO         deepcore-daemon deepcore-daemon/src/devices.rs:100:deepcore_daemon::devices: Camera attached RealsenseD415-928222061521
03.10.22 14:04:35 (+0200)  dcd  0:57:34.049533024    97   0x557da72f00 INFO         deepcore-daemon deepcore-daemon/src/devices.rs:115:deepcore_daemon::devices: Camera detached RealsenseD415-928222061521
...

@wolvi-lataniere Can you test the C example code in a balenalib based container that runs with UDEV=1 ? I just wonder if there is a conflict between udev and udevd when applying your suggestions to a balenalib image running udevd.

Hi Rapha,

I’ve tested on my laptop in Docker with the following file:

FROM balenalib/aarch64-ubuntu:focal

RUN apt-get update &&\
    apt-get install -y build-essential libudev-dev udev

WORKDIR /test

COPY code.c .
RUN gcc -o code code.c -ludev

CMD service udev restart && ./code

And here is what I get:

 * Stopping hotplug events dispatcher systemd-udevd                      [ OK ] 
 * Starting hotplug events dispatcher systemd-udevd                      [ OK ] 
I: ACTION=remove
I: DEVNAME=video1
I: DEVPATH=/devices/platform/2300000.pci/pci0000:00/0000:00:03.0/usb3/3-3/3-3:1.0/video4linux/video1
---
I: ACTION=remove
I: DEVNAME=video0
I: DEVPATH=/devices/platform/2300000.pci/pci0000:00/0000:00:03.0/usb3/3-3/3-3:1.0/video4linux/video0
---
I: ACTION=add
I: DEVNAME=video0
I: DEVPATH=/devices/platform/2300000.pci/pci0000:00/0000:00:03.0/usb3/3-3/3-3:1.0/video4linux/video0
---
I: ACTION=add
I: DEVNAME=video1
I: DEVPATH=/devices/platform/2300000.pci/pci0000:00/0000:00:03.0/usb3/3-3/3-3:1.0/video4linux/video1
---

I get the signals as expected when events are occurring.

I don’t have access to a USB camera at the moment to test on a RaspberryPi with Balena OS, but I don’t think it would do any major difference on the behavior.

Thank you @wolvi-lataniere Can you also show how you started the container of that Dockerfile?

I am confused by this line in your logs:

 * Stopping hotplug events dispatcher systemd-udevd                      [ OK ] 

That sounds as if in your container simply the udevd is being restarted, not the actual udev service.
Thanks for your help :slight_smile:

Yes, I had to use service udev restart to start the service which doesn’t want to start with startalone.

And udevd stands for udev daemon which is the service, so everything is good here.

For the starting command, I used
docker run --rm -v /sys:/sys -e UDEV=1 --privileged -it support-udev-test

with support-udev-test being my the tag I used for my docker build.

Hope this will help you.

The problem lies with the difference between -ubuntu:focal and my -ubuntu:bionic, which we run on the Jetson devices (TX2, Nano)

Your container example works fine with focal, and exhibits my problems of udev not supported on bionic.
I will see if a backport version of udev is available and may help here.

@wolvi-lataniere Any insight on the latest info provided by @rapha

We’ve tried installing focal udev in the container with no luck

Hi,
Thanks for the feedback.
About Focal vs Bionic, I would recommend going for the newest version whenever possible as end of life can come pretty quickly considering a standard development/deployment cycle.
As you can see here Bionic will stop receiving software update in the next 12 months. For newer development I would recommend using the latest available LTS (Currently for Ubuntu it is Jammy) to spare you some maintenance work in the near future.
I will have a look at a build with bionic to see if I can figure it out.

Hi @wolvi-lataniere, the reason we cannot switch from Bionic is that this will run on a Jetson TX2.
From what I can see Focal will only be available on the new Orin…
So I’m curious what Balena’s policy is in supporting Jetson devices that won’t receive major L4T updates anymore?
See NVIDIA Jetpack 5.0.2 release supports Ubuntu 20.04, Jetson AGX Orin - CNX Software

One more clarification on our problem: I built a container based on Bionic and then installed udev from Focal. That get’s me around the udev does not support problem.

On my laptop with the above example code and with focal I see events coming in.
I need to check again on our device.

Hi @rapha,

I looked into it and the fix I proposed doesn’t work on bionic.
I managed to get the code working without starting the udev service first, but I had to use network_mode: "host" in order to allow your udev_monitor_new_from_netlink(udev, "udev"); function to grab the host netlink.

Basically, here is my docker-compose.yml file:

version: '2.4'

services:
    test:
        build: .
        privileged: true
        network_mode: 'host' 
        environment:
            - "UDEV=1"

and here is my Dockerfile:

FROM balenalib/aarch64-ubuntu:bionic

RUN apt-get update &&\
    apt-get install -y build-essential libudev-dev udev

WORKDIR /test

COPY code.c .
RUN gcc -o code code.c -ludev

CMD ./code

I tested it on a RaspberryPi3 64 bits system and it worked (I don’t have any Jetson on hand for further validation).

Let me know if you manage to get it working.

PS: I forwarded your question about our support policy to relevant team members.