libudev cannot monitor udev events inside container

rapha · September 28, 2022, 8:12pm

We are running a GStreamer based pipeline inside balena including the hot-plugging of cameras. The container is based off balenalib and the hot-plugging part works fine with devices showing up inside the container.

GStreamer has the gst-device-monitor-1.0 -f command, which can list the current audio video devices connected as well as continuously listen on changes to the devices

The command uses the libgudev library (glib binding to libudev) to listen on udev device changes, which it then filters and acts upon.

Outside of a container this works perfectly fine. Whenever a camera is plugged in while gst-device-monitor-1.0 -f is running, it prints information about the device.

Inside the container running gst-device-monitor -f will correctly identify and analyse all currently connected devices, but it will not monitor devices as they are plugged in. Re-running it after plugging in new hardware will also work as expected.

Directly monitoring the udev events using udevadm monitor inside the container works fine.

This can be reproduced by building the following image:

# docker build -t gst-device-monitor-test . -f device-monitor.dockerfile

FROM balenalib/generic-ubuntu:focal
LABEL org.opencontainers.image.source https://gitlab.com/aivero/prop/workspace

# Prevent apt-get prompting for input
ENV DEBIAN_FRONTEND noninteractive

# Download and install BSP binaries for L4T 32.4.2
RUN \
    apt-get update && apt-get install -y --no-install-recommends \
    gstreamer1.0-plugins-good \
    gstreamer1.0-plugins-base-apps \
    && rm -rf /var/lib/apt/lists/* 

ENV UDEV=1

Then run it as docker run --rm -ti --privileged -v /sys:/sys -e UDEV=1 gst-device-monitor-test:latest /bin/bash

Inside the container run gst-device-monitor -f and plug in some audio/video device such as a webcam.

I would like to know if there are any changes to the containers that might interfere with the libudev library and that might prevent the callbacks from being called when a device is plugged in?

jimmi · September 29, 2022, 7:44am

Can reproduce this further. This small c code example using libudev and compiled inside the container also does not react to device replugs:

#include <stdio.h>
#include <unistd.h>
#include <libudev.h>

int main()
{
    struct udev *udev;
    struct udev_device *dev;
    struct udev_monitor *mon;
    int fd;

    udev = udev_new();
    if (!udev) {
        fprintf(stderr, "Can't create udev\n");
        return 1;
    }

    mon = udev_monitor_new_from_netlink(udev, "udev");
    udev_monitor_filter_add_match_subsystem_devtype(mon, "video4linux", NULL);
    udev_monitor_enable_receiving(mon);
    fd = udev_monitor_get_fd(mon);

    while (1) {
        fd_set fds;
        struct timeval tv;
        int ret;

        FD_ZERO(&fds);
        FD_SET(fd, &fds);
        tv.tv_sec = 0;
        tv.tv_usec = 0;

        ret = select(fd+1, &fds, NULL, NULL, &tv);
        if (ret > 0 && FD_ISSET(fd, &fds)) {
            dev = udev_monitor_receive_device(mon);
            if (dev) {
                printf("I: ACTION=%s\n", udev_device_get_action(dev));
                printf("I: DEVNAME=%s\n", udev_device_get_sysname(dev));
                printf("I: DEVPATH=%s\n", udev_device_get_devpath(dev));
                printf("---\n");

                udev_device_unref(dev);
            }
        }
    }
    udev_unref(udev);

    return 0;
}

This required me to install libudev-dev and build-essential on the container.

wolvi-lataniere · September 29, 2022, 11:38am

Hi,

It is not clear if you are running your containers on your developer machine OS or on a BalenaOS device (from your command-line with docker, I lean towards the dev machine side).

The UDEV=1 should work in a Balena project (run with the same command on you HostOS on your target device, replacing docker with balena).

If you are running on your computer, we don’t have guidance on how to use UDEV on Docker.

I will try to reproduce your behavior on one of my device to see if I am missing something.

wolvi-lataniere · September 29, 2022, 12:54pm

Hello again,

I tried this on my device and succeed to reproduce your problem.
It turns out that you probably should install udev and start the service on your container.

I could do it with your example C code and following Dockerfile

FROM debian:bullseye

RUN apt-get update &&\
    apt-get install -y build-essential libudev-dev udev

WORKDIR /test

COPY code.c .
RUN gcc -o code code.c -ludev

CMD service udev restart && ./code

Let me know if it solves your issue.

rapha · October 3, 2022, 12:07pm

Hi @wolvi-lataniere
You are right, the examples we posted were tested on x86, however the problem is on arm64 (Jetson TX2).

I tried the essentials of your solution:

Installing udev
restarting the udev service

Upon restarting the udev service I get an error/warning:
udev does not support containers, not started

I found docker - Why udev init script default disable container support while in fact it works? - Stack Overflow which claims that one can patch the udev service script and it works anyway.

I’ve patched the udev service script in the container and I get events in the container.
However, events are now continuously spamming. My camera is constantly being reported as attached and detached:

...
03.10.22 14:04:34 (+0200)  dcd  0:57:32.357796800    97   0x557da72f00 INFO         deepcore-daemon deepcore-daemon/src/devices.rs:115:deepcore_daemon::devices: Camera detached RealsenseD415-928222061521
03.10.22 14:04:34 (+0200)  dcd  0:57:32.585963520    97   0x557da72f00 INFO         deepcore-daemon deepcore-daemon/src/devices.rs:100:deepcore_daemon::devices: Camera attached RealsenseD415-928222061521
03.10.22 14:04:35 (+0200)  dcd  0:57:33.711682944    97   0x557da72f00 INFO         deepcore-daemon deepcore-daemon/src/devices.rs:115:deepcore_daemon::devices: Camera detached RealsenseD415-928222061521
03.10.22 14:04:35 (+0200)  dcd  0:57:33.962839616    97   0x557da72f00 INFO         deepcore-daemon deepcore-daemon/src/devices.rs:100:deepcore_daemon::devices: Camera attached RealsenseD415-928222061521
03.10.22 14:04:35 (+0200)  dcd  0:57:34.049533024    97   0x557da72f00 INFO         deepcore-daemon deepcore-daemon/src/devices.rs:115:deepcore_daemon::devices: Camera detached RealsenseD415-928222061521
...

rapha · October 3, 2022, 1:29pm

@wolvi-lataniere Can you test the C example code in a balenalib based container that runs with UDEV=1 ? I just wonder if there is a conflict between udev and udevd when applying your suggestions to a balenalib image running udevd.

wolvi-lataniere · October 3, 2022, 6:19pm

Hi Rapha,

I’ve tested on my laptop in Docker with the following file:

FROM balenalib/aarch64-ubuntu:focal

RUN apt-get update &&\
    apt-get install -y build-essential libudev-dev udev

WORKDIR /test

COPY code.c .
RUN gcc -o code code.c -ludev

CMD service udev restart && ./code

And here is what I get:

 * Stopping hotplug events dispatcher systemd-udevd                      [ OK ] 
 * Starting hotplug events dispatcher systemd-udevd                      [ OK ] 
I: ACTION=remove
I: DEVNAME=video1
I: DEVPATH=/devices/platform/2300000.pci/pci0000:00/0000:00:03.0/usb3/3-3/3-3:1.0/video4linux/video1
---
I: ACTION=remove
I: DEVNAME=video0
I: DEVPATH=/devices/platform/2300000.pci/pci0000:00/0000:00:03.0/usb3/3-3/3-3:1.0/video4linux/video0
---
I: ACTION=add
I: DEVNAME=video0
I: DEVPATH=/devices/platform/2300000.pci/pci0000:00/0000:00:03.0/usb3/3-3/3-3:1.0/video4linux/video0
---
I: ACTION=add
I: DEVNAME=video1
I: DEVPATH=/devices/platform/2300000.pci/pci0000:00/0000:00:03.0/usb3/3-3/3-3:1.0/video4linux/video1
---

I get the signals as expected when events are occurring.

I don’t have access to a USB camera at the moment to test on a RaspberryPi with Balena OS, but I don’t think it would do any major difference on the behavior.

rapha · October 4, 2022, 9:26am

Thank you @wolvi-lataniere Can you also show how you started the container of that Dockerfile?

I am confused by this line in your logs:

 * Stopping hotplug events dispatcher systemd-udevd                      [ OK ]

That sounds as if in your container simply the udevd is being restarted, not the actual udev service.
Thanks for your help

wolvi-lataniere · October 4, 2022, 12:19pm

Yes, I had to use service udev restart to start the service which doesn’t want to start with startalone.

And udevd stands for udev daemon which is the service, so everything is good here.

For the starting command, I used
docker run --rm -v /sys:/sys -e UDEV=1 --privileged -it support-udev-test

with support-udev-test being my the tag I used for my docker build.

Hope this will help you.

rapha · October 25, 2022, 2:26pm

The problem lies with the difference between -ubuntu:focal and my -ubuntu:bionic, which we run on the Jetson devices (TX2, Nano)

Your container example works fine with focal, and exhibits my problems of udev not supported on bionic.
I will see if a backport version of udev is available and may help here.

jimmi · November 9, 2022, 10:48am

@wolvi-lataniere Any insight on the latest info provided by @rapha

We’ve tried installing focal udev in the container with no luck

wolvi-lataniere · November 11, 2022, 8:31am

Hi,
Thanks for the feedback.
About Focal vs Bionic, I would recommend going for the newest version whenever possible as end of life can come pretty quickly considering a standard development/deployment cycle.
As you can see here Bionic will stop receiving software update in the next 12 months. For newer development I would recommend using the latest available LTS (Currently for Ubuntu it is Jammy) to spare you some maintenance work in the near future.
I will have a look at a build with bionic to see if I can figure it out.

rapha · November 11, 2022, 5:04pm

Hi @wolvi-lataniere, the reason we cannot switch from Bionic is that this will run on a Jetson TX2.
From what I can see Focal will only be available on the new Orin…
So I’m curious what Balena’s policy is in supporting Jetson devices that won’t receive major L4T updates anymore?
See NVIDIA Jetpack 5.0.2 release supports Ubuntu 20.04, Jetson AGX Orin - CNX Software

One more clarification on our problem: I built a container based on Bionic and then installed udev from Focal. That get’s me around the udev does not support problem.

On my laptop with the above example code and with focal I see events coming in.
I need to check again on our device.

wolvi-lataniere · November 17, 2022, 10:14am

Hi @rapha,

I looked into it and the fix I proposed doesn’t work on bionic.
I managed to get the code working without starting the udev service first, but I had to use network_mode: "host" in order to allow your udev_monitor_new_from_netlink(udev, "udev"); function to grab the host netlink.

Basically, here is my docker-compose.yml file:

version: '2.4'

services:
    test:
        build: .
        privileged: true
        network_mode: 'host' 
        environment:
            - "UDEV=1"

and here is my Dockerfile:

FROM balenalib/aarch64-ubuntu:bionic

RUN apt-get update &&\
    apt-get install -y build-essential libudev-dev udev

WORKDIR /test

COPY code.c .
RUN gcc -o code code.c -ludev

CMD ./code

I tested it on a RaspberryPi3 64 bits system and it worked (I don’t have any Jetson on hand for further validation).

Let me know if you manage to get it working.

PS: I forwarded your question about our support policy to relevant team members.

rapha · February 1, 2023, 9:20am

Thanks @wolvi-lataniere and apologies for the late reply. It seems @skuenstler came up with a similar solution: USB re-connect fails from container - #3 by skuenstler

Any update from the support team about the topic? Would be great not to have to rely on the network-mode: host

skuenstler · February 1, 2023, 3:35pm

The solution can be found here.

You have to restart the UDEV as bversluijs describes in his solution.

rapha · February 2, 2023, 1:28pm

Thanks @skuenstler

That was more or less what we tried further up: libudev cannot monitor udev events inside container - #4 by wolvi-lataniere

But after restarting the udev service we are stuck in a constant loop of devices being registered and unregistered. That holds for the solution by @bversluijs as well as what @wolvi-lataniere suggested.
Still stuck at this I’m afraid.

skuenstler · February 2, 2023, 2:05pm

It is not enough to make the UDEV restart from within docker-compose.yml cmd’s.

This tricked me also!

You have to add the “restart” cmd into a separate start.sh file that will execute prior to your container-startup.

Inside docker-compose.yml of your service at question, add the following:

COPY start.sh /opt/
# We are running our entrypoint commands through the start.sh script
CMD ["bash", "/opt/start.sh"]

(i.e. in our case the root folder is /opt (in your case it might be a different folder)…

And the start.sh script looks as follows:

#!/bin/bash
if which udevadm >/dev/null; then
    echo "Restarting UDEV Service to enable hotpluging"
    set +e # Disable exit on error
    service udev restart
    set -e # Re-enable exit on error
    echo "UDEV Service restarted"
fi
value="yarn start:docker"
echo "Starting MyContainer Service"
exec $value

Hope this helps.

rapha · February 2, 2023, 3:02pm

Thanks @skuenstler
I’m afraid that is already what I am doing.