phidget usb device cannot be hot plugged, libusb_open() failed error

cowens · September 14, 2021, 9:42pm

We have a multi-container application where 1 container is accessing USB phidget devices. Upon boot the USB devices are detected and run normally. If the devices are unplugged and plugged back in the phidget application sees the devices plugged back in but cannot open the usb as can be seen in the following logs. Any ideas on how to fix this?

Logs where phidget application cannot open the USB

2021-09-14T21:36:11  <Info> phidget22usb[PhidgetUSBScanDevices()] : New Phidget found in PhidgetUSBBuildList: 1/4
2021-09-14T21:36:11  <Warn> phidget22usb[PhidgetUSBScanDevices()] : libusb_open() failed: LIBUSB_ERROR_NO_DEVICE - No such device (it may have been disconnected).
2021-09-14T21:36:11  <Info> phidget22usb[PhidgetUSBScanDevices()] : This usually means you need to run as root, or install the udev rules.
2021-09-14T21:36:11  <Info> phidget22usb[PhidgetUSBScanDevices()] : New Phidget found in PhidgetUSBBuildList: 1/5
2021-09-14T21:36:11  <Warn> phidget22usb[PhidgetUSBScanDevices()] : libusb_open() failed: LIBUSB_ERROR_NO_DEVICE - No such device (it may have been disconnected).
2021-09-14T21:36:11  <Info> phidget22usb[PhidgetUSBScanDevices()] : This usually means you need to run as root, or install the udev rules.
2021-09-14T21:36:12  <Info> phidget22usb[PhidgetUSBScanDevices()] : New Phidget found in PhidgetUSBBuildList: 1/7
2021-09-14T21:36:12  <Warn> phidget22usb[PhidgetUSBScanDevices()] : libusb_open() failed: LIBUSB_ERROR_NO_DEVICE - No such device (it may have been disconnected).
2021-09-14T21:36:12  <Info> phidget22usb[PhidgetUSBScanDevices()] : This usually means you need to run as root, or install the udev rules.

Things we have tried
Setting the service to privileged mode and the following…

We have tried enabling UDEV=1 in the compose file and dockerfile.template. When we do this we can see the systemd-udevd process running (in top). But we only see kernel events but no UDEV events.

We have tried running UDEV=0, and starting /lib/systemd/systemd-udevd --daemon in our dockerfile CMD. When we do this, we see udevadmn monitor UDEV events. But we still get the errors show above.

Here is a partial log of the UDEV events coming from udevadm monitor:

KERNEL[279.664196] add      /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.4/1-1.4.7 (usb)
KERNEL[279.665966] add      /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.4/1-1.4.7/1-1.4.7:1.0 (usb)
KERNEL[279.670201] add      /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.4/1-1.4.7/1-1.4.7:1.0/0003:06C2:003F.0006 (hid)
KERNEL[279.670723] add      /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.4/1-1.4.7/1-1.4.7:1.0/usbmisc/hiddev2 (usbmisc)
KERNEL[279.670800] add      /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.4/1-1.4.7/1-1.4.7:1.0/0003:06C2:003F.0006/hidraw/hidraw2 (hidraw)
KERNEL[279.684558] bind     /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.4/1-1.4.7/1-1.4.7:1.0/0003:06C2:003F.0006 (hid)
KERNEL[279.684653] bind     /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.4/1-1.4.7/1-1.4.7:1.0 (usb)
KERNEL[279.684727] bind     /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.4/1-1.4.7 (usb)
UDEV  [279.690442] add      /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.4/1-1.4.7 (usb)
UDEV  [279.692619] add      /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.4/1-1.4.7/1-1.4.7:1.0 (usb)
UDEV  [279.696777] add      /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.4/1-1.4.7/1-1.4.7:1.0/0003:06C2:003F.0006 (hid)
UDEV  [279.702063] add      /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.4/1-1.4.7/1-1.4.7:1.0/usbmisc/hiddev2 (usbmisc)
UDEV  [279.707213] add      /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.4/1-1.4.7/1-1.4.7:1.0/0003:06C2:003F.0006/hidraw/hidraw2 (hidraw)
UDEV  [279.713770] bind     /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.4/1-1.4.7/1-1.4.7:1.0/0003:06C2:003F.0006 (hid)
UDEV  [279.715866] bind     /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.4/1-1.4.7/1-1.4.7:1.0 (usb)
UDEV  [279.718781] bind     /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.4/1-1.4.7 (usb)

Here is our udev rules file:

# Very old Phidgets
SUBSYSTEMS=="usb", ACTION=="add", ATTRS{idVendor}=="0925", ATTRS{idProduct}=="8101", MODE="666"
SUBSYSTEMS=="usb", ACTION=="add", ATTRS{idVendor}=="0925", ATTRS{idProduct}=="8104", MODE="666"
SUBSYSTEMS=="usb", ACTION=="add", ATTRS{idVendor}=="0925", ATTRS{idProduct}=="8201", MODE="666"

# All current and future Phidgets - Vendor = 0x06c2, Product = 0x0030 - 0x00af
SUBSYSTEMS=="usb", ACTION=="add", ATTRS{idVendor}=="06c2", ATTRS{idProduct}=="00[3-a][0-f]", MODE="666"

# SPI VINT Hub
SUBSYSTEMS=="spi", DRIVERS=="phidget_hub0004", ACTION=="add", MODE="666

Here is our dockerfile for the service:

FROM balenalib/%%BALENA_MACHINE_NAME%%:latest-run as run

WORKDIR /usr/src/phidget

RUN set -ex; \
    tempPkgs=' \
        ca-certificates \
        wget \
        unzip \
    '; \
    apt-get update; \
    apt-get install -q -y --no-install-recommends \
        $tempPkgs; \
    wget https://www.phidgets.com/downloads/phidget22/libraries/linux/libphidget22/libphidget22-1.7.20210816.tar.gz; \
    wget https://www.phidgets.com/downloads/phidget22/libraries/linux/libphidget22extra/libphidget22extra-1.7.20210816.tar.gz; \
    wget https://www.phidgets.com/downloads/phidget22/servers/linux/phidget22networkserver/phidget22networkserver-1.7.20210816.tar.gz; \
    # Cleanup tempPkgs
    apt-get purge -y --auto-remove $tempPkgs; \
    # Install persistend packages
    apt-get update; \
    apt-get install -q -y --no-install-recommends \
        build-essential \
        usbutils \
        libusb-1.0-0-dev \
        ; \
    apt-get autoremove -q -y; \
    apt-get clean -q -y; \
    rm -rf /var/lib/apt/lists/*; \
    # Phidget 22 Library
    tar -xzf libphidget22-*.tar.gz; \
    cd libphidget22*; \
    ./configure --prefix=/usr; \
    make -j; \
    make -j install; \
    cd ..; \
    # Phidget 22 Library Extra
    tar -xzf libphidget22extra-*.tar.gz; \
    cd libphidget22extra*; \
    ./configure --prefix=/usr; \
    make -j; \
    make -j install; \
    cd ..; \
    # Phidget 22 Network Server
    tar -xzf phidget22networkserver-*.tar.gz; \
    cd phidget22networkserver*; \
    ./configure --prefix=/usr; \
    make -j; \
    make -j install

# Config    
COPY ./phidget22networkserver.pc /etc/phidgets/phidget22networkserver.pc
COPY ./99-libphidget22.rules /etc/udev/rules.d

# Cleanup
RUN rm -rf libphidget22*; \
    rm -rf phidget22networkserver*

CMD /lib/systemd/systemd-udevd --daemon && phidget22networkserver

Docker compose file

version: '2'
volumes:
  solvere-data:
services:

  log:
    build: log
    expose:
      - '80'
    volumes:
      - 'solvere-data:/data'
  
  phidget:
    build: phidget
    expose:
      - '5661'
    volumes:
      - 'solvere-data:/data'
    privileged: true
    labels:
      io.balena.features.kernel-modules: true

  api:
    build: api
    expose:
      - '3000'
    volumes:
      - 'solvere-data:/data'
    privileged: true
    labels:
      io.balena.features.kernel-modules: true
      io.balena.features.supervisor-api: true
    depends_on:
      - log
      - phidget
      # - db


  app:
    build: app
    volumes:
      - 'solvere-data:/data'
    privileged: true
    labels:
      io.balena.features.kernel-modules: true
    depends_on:
      - log

embeddeddan · September 15, 2021, 3:06pm

I am working with @cowens on this. The root of this problem seems to be related to the fact the the /dev/bus/usb/001/ directory is not being updated in the container. The directory is being updated in the in the hostOS but not in the container.

Here is the container:

Here is the hostOS

mpous · September 16, 2021, 11:12am

Thank you for your detailed logs @cowens @embeddeddan

Could you please confirm what balenaOS version are you using and for what type of devices?

Thanks!

cowens · September 16, 2021, 9:12pm

Quick update here.

We ended up putting UDEV=1 back in our compose file, removing the /lib/systemd/systemd-udevd --daemon from the dockerfile CMD. This lead to no longer seei the libusb errors and we were seeing /dev/bus/usb/001 directory update properly within the container.

The phidget22networkserver did not acknowledge the USB devices being reconnected. I am hoping to do some more testing on this this weekend to provide a more detailed update.

The device we are using is a raspberry pi 4 and the balenaos version is balenaOS 2.83.10+rev1.

builder555 · October 21, 2021, 6:05pm

Hello,

We are pleased to hear that the libusb errors disappeared. Regarding the hot-pluggable USB devices, perhaps these examples will help.