USB hotplug for USB serial devices

Product support balenaOS
,
1

What is the current recommended way to handle USB serial devices being plugged in?

I am working on porting an open source IoT gateway to balenaOS for the commercial pilot of an IoT hub product. The gateway has an add-ons system which adds support for a wide range of hardware including Zigbee and Z-Wave USB dongles (e.g. the Aeotec Z-Stick Z-Wave dongle or the Digi XStick or Conbee II Zigbee dongle).

On Raspbian the add-ons can automatically detect USB serial devices that are plugged into the device at runtime and use them.

Running inside a Docker container on balenaOS I can statically map a device in docker-compose.yml if I know what path it will be mounted at, e.g.

    devices:
      - /dev/ttyACM0:/dev/ttyACM0

However, if that device is not plugged in when the Docker image starts up it will simply refuse to start and the (headless) gateway becomes inoperable.

Ideally I would like to be able to dynamically detect USB dongles when they are plugged in using USB hotplug and automatically map them into the Docker container.

If I could create a bind mount to just bind the whole of the host’s /dev directory then I could configure device_cgroup_rules to allow the kernel to access certain classes of device, and perhaps use cap_add to add SYS_RAWIO for low level access (I do not want the container to be running privileged mode since that negates most of the security benefits of using a container), e.g.

    volumes:
      - '/dev:/dev' # Mount the host's dev directory
    device_cgroup_rules:
      - 'c 188:* rmw' # Allow USB serial devices (XBee, Conbee III, Sonoff)
      - 'c 166:* rmw' # Allow ACM devices (Conbee II)
    cap_add:
        - SYS_RAWIO

However, since bind mounts are not allowed this is not possible.

I can imagine some elaborate sidecar container that runs in privileged mode and pipes serial interfaces into unix sockets, but that would require re-architecting my whole gateway application and its 100 or so adapter add-ons in order for it to run on balenaOS.

Is there a recommended way to detect USB devices being plugged in via USB hotplug and automatically map them into a container?

2

Hello @benfrancis,

Well we do have a way to let the container use the whole /dev, however I believe it does need to be privileged:

Ideally I would like to be able to dynamically detect USB dongles when they are plugged in using USB hotplug and automatically map them into the Docker container.

I think that could be doable with a set of carefully crafted udev rules with proper access rights and clever volume management.

If you start this work and report here, we can try to guide you to the best possible outcome.

Regards,

Yann

3

Hi,

I’m still struggling with this.

I thought I might have found a solution by using device_cgroup_rules in docker-compose.yml to allow allow the container cgroup to access USB serial device classes, so that device nodes created at runtime via mknod are functional:

    labels:
      io.balena.features.sysfs: '1'
    cap_add:
      ## Raw I/O access for interacting with hardware devices
      - SYS_RAWIO
    # Allow container cgroup to access USB serial device classes so that
    # device nodes created at runtime via mknod are functional.
    # Major 166 = ttyACM (USB CDC ACM devices, e.g. Zigbee dongles)
    # Major 188 = ttyUSB (USB-to-serial converters, e.g. FTDI/CH340)
    device_cgroup_rules:
      - 'c 166:* rwm'
      - 'c 188:* rwm'

I then created a usb-monitor.sh script which dynamically creates and removes /dev nodes for USB serial devices (ttyACM*, ttyUSB*) inside the container by polling /sys/class/tty/.

Whilst this made the serial ports dynamically discoverable from inside the container, when the gateway application actually tried to use them it caused an EPERM error which suggests the kernel’s device cgroup is rejecting the open() call:

[Logs]    [2026-03-26T17:21:05.903Z] [webthings-gateway] 2026-03-26 17:21:05.903 INFO   : zigbee-adapter: Probing serial ports
[Logs]    [2026-03-26T17:21:05.918Z] [webthings-gateway] 2026-03-26 17:21:05.918 ERROR  : zigbee-adapter: [Error: Error: Operation not permitted, cannot open /dev/ttyACM0]
[Logs]    [2026-03-26T17:21:05.919Z] [webthings-gateway] 2026-03-26 17:21:05.918 ERROR  : zigbee-adapter: [Error: Error: Operation not permitted, cannot open /dev/ttyACM0]
[Logs]    [2026-03-26T17:21:05.919Z] [webthings-gateway] 2026-03-26 17:21:05.919 INFO   : zigbee-adapter: Serial ports that were found:
[Logs]    [2026-03-26T17:21:05.929Z] [webthings-gateway] 2026-03-26 17:21:05.929 INFO   : zigbee-adapter: USB Serial Device 1cf1:0030 Vendor: dresden elektronik ingenieurtechnik GmbH Serial: DE2706277 found @ /dev/ttyACM0
[Logs]    [2026-03-26T17:21:05.929Z] [webthings-gateway] 2026-03-26 17:21:05.929 ERROR  : zigbee-adapter: Failed to start add-on zigbee-adapter: No Zigbee dongle found

Upon further investigation I discovered that despite ostensibly supporting docker-compose version 2.4 files, the balena supervisor does not appear to support device_cgroup_rules and was dropping it and logging a warning:

Mar 26 17:20:38 webthings balena-supervisor[2247]: [warn]    Ignoring unsupported or unknown compose fields: containerName, logging, deviceCgroupRules

Can you confirm that no version of the supervisor currently supports the device_cgroup_rules directive in docker-compose.yml?

If so, can you provide any further guidance on the custom udev rules approach you suggested? I can’t figure out a way to make that work which doesn’t require hard coding vendor/product IDs and/or a static device mapping which causes the container to fail to start if the device is not plugged in on boot.

Thank you

4

Reporting back on this, I have come up with a rather elaborate workaround.

Essentially this PR:

  • Adds a usb-monitor.sh script inside the gateway container which dynamically creates and removes /dev nodes for USB serial devices (ttyACM*, ttyUSB*) by polling /sys/class/tty/
  • Adds a tiny privileged sidecar container that whitelists USB serial device classes (ttyACM major 166, ttyUSB major 188) in the gateway container’s cgroup

This makes it possible to dynamically map serial USB devices into the gateway application’s container as they are plugged and unplugged at runtime, without needing to make the whole gateway application’s container privileged (or needing to give it the SYS_ADMIN capability), which would have largely defeated the object of running the application inside a container in the first place.

This negates the need to hard code device paths in docker-compose.yml and means that:

  1. For limited classes of devices it behaves as if the gateway container supports USB hotplug
  2. Add-on adapters like our Zigbee and Z-Wave adapters can detect USB devices at runtime
  3. The gateway can still start up if one those devices are not connected at boot

This is obviously a bit of an ugly hack, so if there is a cleaner way to achieve this please let me know. And if not, I feel like this is something balenaOS could offer a much cleaner solution to, since USB hotplug seems like a feature that people very commonly need.

6

Hello @benfrancis

That’s quite a neat workaround!

7

Ugh, after all that work this no longer works in balenaOS v7.

As far as I can tell the issue is that v7 switched to cgroup v2 which doesn’t have a devices.allow file, so this whole approach fails.

@rahul-thakoor Can you suggest a way to get this to work in balenaOS v7? Is there a supported way for containers to support USB hotplug without being privileged?

8

hey @benfrancis
let me have a look and get back to you

9

hey @benfrancis which device type and balenaOS version are you using?

from the project page GitHub - WebThingsIO/gateway: WebThings Gateway - a self-hosted web application for monitoring and controlling a building over the web · GitHub is seems that raspberry pi is the main target, but wanted to confirm the exact setup and versions you used

thanks

10

@rahul-thakoor The balenaOS port is in a downstream fork specifically for Krellian Hub.

We recently switched from the Intel NUC Elements line to targeting ASUS NUC 13 Rugged, which required switching from the intel-nuc device type to generic-amd64 to support the Wi-Fi module.

The re-build of our OS image for that device type pulled in the latest balenaOS 7.0.5 which is how I discovered the problem.

It actually failed pretty hard because once the newly flashed hub got a network connection balenaCloud tried to push an update but the usb-cgroup-helper was stuck in a loop and caused some kind of lock which prevented the supervisor from shutting down the container and caused the update process to get stuck in an unrecoverable state where the containers got stuck at “stopping” and SSH to the hostOS no longer worked. Fortunately just in testing.

11

Bug report with STR here.

12

hi again @benfrancis ,
discussing with some colleagues, and this is related to an updated systemd version on v7.x.

I noticed your current workaround is to pin the image to balenaOS 6.x

The best way to achieve your goal would have been via device_cgroup_rules but as you already tested this, it’s currently not supported on balenaOS. We are working to add support for device_cgroup_rules in future releases of the supervisor.

In the meantime, i tried to think of some workarounds:

  • I explored devtmpfs but it seems that would require your container to be privileged
  • I tested another approach:
    Using socatto create an internal pseudo-terminal (PTY) bridge and the socket being shared by the two containers via a volume. this is based on something you suggested:
I can imagine some elaborate sidecar container that runs in privileged mode and pipes serial interfaces into unix sockets, but that would require re-architecting my whole gateway application and its 100 or so adapter add-ons in order for it to run on balenaOS.

I set up a simple demo: GitHub - balena-solutions/usb-serial-pty-sidecar: USB Serial Access via Privileged Sidecar · GitHub

Can you check and see if that helps you?

You can tweak it for your specific use case and extend it accordingly

Looking forward to your feedback

Thanks,
Rahul

13

Hi @rahul-thakoor,

Thank you very much for taking the time to look into this and work on a potential solution.

Yes that seems like it would be the cleanest solution, it would be great to have that supported by the supervisor.

If this is indeed transparent to the container and just appears to be a real device node at its usual location inside the container’s filesystem then this could indeed be a potential workaround. I will experiment with that, thank you very much.

I’m currently preparing for the launch of a commercial pilot which may have to ship with 6.x until I have this figured out. Given the major architectural changes in 7.x, is 6.x still receiving security updates (including on the free plan) while people make the transition?

Thank you

14

hey @benfrancis rolling releases only receive updates with a new release. ESR on the other hand has scheduled release cadence and clear support periods.

15

The devtmpfs approach works but yeah, you’re back to privileged mode. Udev rules for just the serial devices could be worth exploring if Docker on balenaOS even allows it, but I’d guess you hit the same wall either way.