X11 hotplug USB

Product support
1

Hi all,

I’ve been working with Balena for quite a while know, and I’ve succeeded to get an UI container working with the GPU correct GPU drivers for an UP Squared. So far so good.

The interface requires user interaction, so the user has a keyboard and a mouse/touchscreen. All works fine. However, if I inplug the keyboard USB and/or the touchscreen (the problem occurs with any USB), and plug it back in, the udev monitor sees changes inside the UI container, but the X11 session doesn’t listen to udev (I think). The X11 session just doesn’t add the USB’s, and I can’t type or click anywhere.

Some information:
Hardware: UP Squared (Atom E5 chipset)
Balena OS: 2.29.0+rev2
Docker Image used: up-board-debian:latest
Docker image runs as privileged (UDEV changes are triggered, but X11 doesn’t do anything with them)

Dockerfile:

FROM node:slim

RUN apt-get update && apt-get -y install --no-install-recommends libgtkextra-dev libgconf2-dev libnss3 libasound2 libxtst-dev libxss1 libgtk-3-0

WORKDIR /usr/src/app

COPY ./app/build/package.json ./

RUN JOBS=MAX npm install --unsafe-perm && npm cache clean --force && \
  rm -rf /tmp/*

COPY ./app/build ./

RUN ["bash", "/usr/src/app/init.sh"]

FROM balenalib/up-board-debian:latest

ENV UDEV=1

RUN apt-get update \
    && apt-get -y install --no-install-recommends \
      fuse \
      xserver-xorg-core xserver-xorg-input-all xserver-xorg-video-intel xorg \
      libnss3 libgtk-3-0 libasound2-dev \
    && rm -rf /var/lib/apt/lists/* \
    && apt-get clean

WORKDIR /usr/src/app

COPY ./app/run/ ./

COPY --from=0 /usr/src/app/dist/kiosk.AppImage /usr/src/app/kiosk.AppImage

CMD ["bash", "/usr/src/app/init.sh"]

X11 session start:

startx /usr/src/app/kiosk.AppImage

I hope some of you guys can help me out, because it’s really frustrating that some USB changes just don’t get picked up by the X11 session.

Thanks in advance!

3

I think this is because the dev directory is linked into the container on startup, but it doesn’t reflect any changes. We have plans to support mounting the dev filesystem into containers (with a specific flag) so that these changes are mirrored, for the exact use case of hotplugging etc.

4

Hi @CameronDiver,

Even if I add to my docker-compose.yml:

devices:
- "/dev:/dev"

I have the same issues. And I’ve checked if both the HostOS and the container have the same /dev/bus/usb/*, and that’s the case. So that can’t be the case?

Also the /dev/input/ is the same, and cat /dev/input/event2 (the keyboard) also gives some input when I press some keys.

5

Do you mean that the events still come through after replugging from within the container? That’s very strange, it might be because the filename of the device doesn’t change. In that case I would have expected the xserver to support this.

6

Hi @CameronDiver,

Yes, the events still come through in the container. The only thing that changes with lsusb is the device number of the keyboard USB when I replug the USB. So it’s very strange…

13

@bversluijs not sure if this will be help, but maybe if you add the mounting of /dev as a devtmpfs into your container, you can see this here: https://github.com/balena-io-library/base-images/blob/master/scripts/assets/entry.sh#L10 which is applied when a balenalib container runs as privileged, but doesn’t apply when unprivileged. I think that will make the /dev refresh when hotplug events happen, but its just a guess :slight_smile:

15

Hi @shaunmulligan,

Thanks for your reply. For testing purposes, this is my docker-compose.yml at the moment:

version: '2'
services:
  kiosk:
    build: ./kiosk
    privileged: true
    restart: always
    environment:
      - URL_LAUNCHER_URL=https://google.com
      - ELECTRON_ENABLE_HW_ACCELERATION=1
    tmpfs:
      - /dev

I’ve added the /dev as a tmpfs. This is what you mean, right?
I’ve also changed my Dockerfile FROM to FROM balenalib/up-board-ubuntu:latest, because the command startx didn’t work reliable on the debian stretch image, which was really weird (I had to start and stop startx and then start it again, before I got something to show up on my screen. The first time it started, the screen stayed black and didn’t get any signal. The Ubuntu image just starts like it’s supposed to).

Back to the devtmpfs, I got this error after all steps have been completed:

ERROR: for f7af2ec82207_kiosk_1  Cannot start service kiosk: b'OCI runtime create failed: container_linux.go:296: starting container process caused "process_linux.go:398: container init caused \\"rootfs_linux.go:112: reopening /dev/null inside container caused \\\\\\"Failed to open /dev/null - open /dev/null: permission denied\\\\\\"\\"": unknown'

For your information, I’m using the docker-compose command to build and start containers on my Balena device, because it’s much much faster than balena push <ipAddress>.

I hope you can help me sort out this issue! :slight_smile:

16

hmm, im not sure tmpfs: - /dev will work for /dev. could you try running the function i linked to in your entrypoint or start up script?

Also interesting data point on docker-compose being faster than balena push <IP> perhaps @camerondiver can point to why that would be, it shouldn’t really make much of a difference. Also just be careful if you are developing that way, because it won’t always be 1-to-1 compatible with balena push

18

Hi @shaunmulligan,

Do you mean just copy the mount_dev function and running that in my init.sh script? I’ve done that
But this is the output:

kiosk_1  | mount: /tmp/tmpmount/console: special device /dev/console does not exist.
kiosk_1  | mount: /tmp/tmpmount/console: wrong fs type, bad option, bad superblock on /dev/console, missing codepage or helper program, or other error.
kiosk_1  | umount: /dev: target is busy.
kiosk_1  | mount: /sys/kernel/debug: nodev already mounted on /sys/kernel/debug.

And the USB hotplugging still doesn’t work. I’m thinking that the problem is with the X11/xorg session, because when I use udev monitor inside the container, every UDEV event is triggered and seems fine. The problem is, I’ve been using debian jessie before and everything worked fine (If I remember correctly), but I want to use the GPU of the UP Squared, and those drivers only work in debian stretch in combination with X11…

And regarding the balena push <IP>, it takes long before the push starts and after building, it takes some time before it really shows some logs. However, when I use docker-compose, it’s all instant. Also, stuff like docker-compose down or docker-compose kill aren’t supported (afaik) with balena push <IP>. And I know it isn’t 1-to-1 compatible, because I’m missing the balena ENV vars, which is too bad. But I don’t have any problems so far with just using docker-compose except for the balena ENV vars and the labels, which is becoming a problem because I need to contact the supervisor in the future. But I’m postponing that because the development time is much longer.

1 Like
20

Ah, thats weird, especially since the container is running privileged, i wouldn’t expect that to fail. If it would help, I could try reproduce here on my end (although I don’t have an UPboard on hand) if you can create a minimal project that has the issue.

On the balena push <IP> side of things, we have been improving it quite a bit and also added a new --live flag on the latest version, that will do live reloads, so that might make things faster, would be interested in hearing your take on it :slight_smile:

22

Hey, I wrote the balena push code so I’ll go through the reasons for this.

And regarding the balena push <IP> , it takes long before the push starts and after building, it takes some time before it really shows some logs. However, when I use docker-compose , it’s all instant.

This is more noticeable with larger code repositories, but essentially it boils down to the way that balena push tars the input to stream to the docker daemon. Because of the handling of things like Dockerfile.templates etc, the pipeline has to read the input at least once, transforming it as it moves through. Add in that the node module we use for this has some inherent problems, the performance is definitely not as good as raw docker-compose. Two things I’ve been planning for this, one is to be able to skip the Dockerfile.template et al. resolution, which will improve things, and another is to start working on a tar module (or changing the current) to be more performance aware (for example we don’t need to add files one by one as we do now, if we have the size beforehand, which we do). Unfortunately these just haven’t taken as large a priority as other more fun features (--live like @shaunmulligan mentioned :slight_smile: )

Another reason for this is that the CLI must talk to the supervisor to negotiate the push, and update the target state. This is necessary, but also can be optimized for fewer calls. Factor in that those calls are often handled by an rpi or similarly low-powered device, the responses can take some time.

Basically, we have this on our radar, it’s just not a huge priority right now.

Also, stuff like docker-compose down or docker-compose kill aren’t supported (afaik) with balena push <IP> .

Again, these are going to be upcoming, we are making a lot of changes to the CLI right now, and this is definitely earmarked as one of them.

Also, this feedback is fantastic :slight_smile:

23

Hi @shaunmulligan,

I’m going to create a simple public project for you to test it. But it’s made for an UP Squared, so it installs the intel drivers for the GPU and starts the display with the intel drivers. This is quite important for me that these work, because that’s the reason why I’ve updated the kiosk (Also other reasons, like I’m a big fan of using the latest stable Debian release, and the kiosk I’ve used before was 1.1GB and this is around 500MB, which is a big improvement). But nonetheless, probably the issue still occurs when you try it on another device without GPU drivers, because USB-devices don’t use the USB :slight_smile:

So a big thank you for helping me with this issue! I’m going to create the project right now!

And hi @CameronDiver,

Thanks for explaining the balena push for me. I’m going to try the --live option. I prefer to use the balena push ofcourse, but it just takes too long to build and start the container when comparing it to the docker-compose as I explained. But using the Balena ENV vars and the ability to use the Balena labels for the supervisor is becoming more and more important. As explained in other topics, I want to use the supervisor for the current state, release ID, reboot’s etc.

I was planning on creating a topic to address this “issue”, but it was relevant for this topic too. I understand this isn’t a high priority issue, but the faster the balena push command is, the more it’s going to be used and I can imagine it’s also a hugh relieve when developing the command when it’s fast :slight_smile:

For the record, I’ve been developing on an UP Squared board and a RPI3 with Balena, and there is a significant improvement in speed on the UP Squared board, like expected. But it’s still quite slow. On the RPI 3 I understand it’s slow, docker-compose isn’t much faster.

If creating a new topic is best for this “issue”, I’ll do that! Because we must not lose focus on the issue of the X11 hotplug USB ofcourse! And I’m really eager to learn from the Balena team and other users on how they develop for Balena. Because, like I said, on the RPI3 it’s really slow to keep pushing all the code (which is the issue of the RPI3, not Balena ofcourse).

24

Great, let us know when you have the example put together, we share your love of staying on latest stable debian and it would be good to get to the bottom of the issue :slight_smile:

For live push and general development, it would be awesome to start a new topic and get feedback and info on how people are developing, so please definitely do that :slight_smile:

26

I’ve created a repository to replicate the issue @ https://github.com/vedicium/balena-X11-hotplug
I hope you can find something!

Update
When using the intel drivers and the Ubuntu image, it isn’t reliable either. Same as the debian image, the X11 session stays blank for the first time…

And I’ll create a topic to share some idea’s about how people are developing on Balena!

Tadaaa Developing on a Balena device

27

Thanks @bversluijs , I’ll see if we can figure it out using your repro!

29

Hi @shaunmulligan,

Did you/someone have had some time to look at this issue?

Thanks in advance!

30

After a lot of tweaking, I’ve figured it out.
I’ve added the following code before starting the X11 process:

if which udevadm > /dev/null; then
  set +e # Disable exit on error
  udevadm control --reload-rules
  service udev restart
  udevadm trigger
  set -e # Re-enable exit on error
fi

I restarted udev and did udevadm trigger, and hotplugging USB’s started working. And I’ve combined it with the intel graphics drivers, so the UP Board uses the GPU instead of CPU.

2 Likes
31

Awesome, @bversluijs thats great new. Sorry I haven’t managed to get you anymore clarity on this. Glad you managed to get there in the end. I’ll ask our base images engineer to have a look at this and see if there is something we can add to our base images to make this work more out of the box.

33

No problem @shaunmulligan. Let me know if you have any updates from the engineers!

34

@bversluijs: In my case, adding your extra code at the bottom of the start.sh script inside xserver and creating my own custom balena-block from it, did not work.

However, adding your extra code into the correct docker Container finally worked as well.

Adding your code to the start.sh and making sure to execute it from the correct Container finally worked

Thank you very much.

COPY start.sh /
CMD ["bash", "/start.sh"]