Container can't see new /dev/tty* device on host

support
raspberrypi3
docker

#1

My application can’t see a fresh new /dev/tty* device on host.

The device in question is /dev/ttyXRUSB0, that is an Exar USB device driver, needed by an USB to RJ45 adapter for RS-485 communication.

I had to build a kernel module for that, inspiring on https://github.com/balena-io-projects/kernel-module-build.
In order to see the /dev/ttyXRUSB0 device, I have to load the xr_usb_serial_common.ko module.
On previous version of resinOS/balenaOS I was able to see the new device within my container, but now, on my Raspberry Pi 3 B running balenaOS 2.29.2+rev1 and Supervisor 9.0.1, I see the device only on the host, not within the container!

To see the device within the container I have to reboot/restart the container.
Note also that rebooting the device would lead to the same problem.

Is there a way I can refresh the list of /dev/tty* devices from the host within my container?

Regards,
Danilo


#4

Hi @daghemo, Can you tell me what baseimage your Dockerfile starts FROM, it sounds like you don’t have udev running in the container, which is responsible for picking up the addition of dynamic devices in /dev.


#5

Can you try set a device environment variable UDEV == 1 on you device?


#6

Hi @shaunmulligan,
I’m using:

FROM balenalib/%%RESIN_MACHINE_NAME%%-python:3.7.2-stretch-20190123

I’ve also tried with “UDEV=1”.


#7

@daghemo can you post your dockerfile and/or docker-compose.yml here. Would it also be possible for you to post the contents of /usr/bin/entry.sh here


#8

Ok, I’m goint to strip down my Dockerfile, since it’s over 38k right now. :slight_smile:
I’ll be back as soon as I reflash my RPi.


#9

I’m back and I was able to reproduce the problem:

[daghemo@asgard ~]$ ./sshResin.sh young-fog
connecting to container on young-fog (04d0b30) running application TLtestingEXAR...
Connecting to: 04d0b30
root@04d0b30:/usr/src/app# ls -l /dev/ttyXRUSB0
ls: cannot access '/dev/ttyXRUSB0': No such file or directory
root@04d0b30:/usr/src/app# exit
exit
Connection to ssh.resindevice.io closed.
[daghemo@asgard ~]$ ./sshResin.sh young-fog ashost
connecting to host young-fog (04d0b30) running application TLtestingEXAR...
Connecting to: 04d0b30
=============================================================
    Welcome to balenaOS
=============================================================
root@04d0b30:~# ls -l /dev/ttyXRUSB0
crw-rw---- 1 root dialout 266, 0 Feb  6 11:36 /dev/ttyXRUSB0
root@04d0b30:~# exit
logout
Connection to ssh.resindevice.io closed.

Here is my Dockerfile:

FROM balenalib/%%RESIN_MACHINE_NAME%%-python:3.7.2-stretch-20190123

ENV INSTALL_EXAR_USB_DRIVER=Yes
ENV RESIN_OS_VERSION="2.29.2+rev1.prod"
ENV INITSYSTEM on
ENV container docker
RUN apt-get update && apt-get install -y --no-install-recommends systemd systemd-sysv \
    && rm -rf /var/lib/apt/lists/*
RUN systemctl mask \
    dev-hugepages.mount \
    sys-fs-fuse-connections.mount \
    sys-kernel-config.mount \
    display-manager.service \
    getty@.service \
    systemd-logind.service \
    systemd-remount-fs.service \
    getty.target \
    graphical.target \
    kmod-static-nodes.service
COPY tlshoot.sh /usr/bin/tlshoot.sh
COPY tlshoot.service /etc/systemd/system/tlshoot.service
RUN chmod a+x /usr/bin/tlshoot.sh
RUN systemctl enable /etc/systemd/system/tlshoot.service
STOPSIGNAL 37
ENTRYPOINT ["/usr/bin/tlshoot.sh"]
ENV UDEV=1
WORKDIR /usr/src/app

RUN echo 'APT::Install-Recommends "false";'                         >>/etc/apt/apt.conf.d/docker-minimal         && \
    echo 'APT::Install-Suggests   "false";'                         >>/etc/apt/apt.conf.d/docker-minimal         && \
    echo 'APT::Get::Show-Upgraded "true";'                          >>/etc/apt/apt.conf.d/docker-minimal         && \
    echo 'APT::Get::Assume-Yes    "true";'                          >>/etc/apt/apt.conf.d/docker-non-interactive && \
    echo 'APT::Quiet "true";'                                       >>/etc/apt/apt.conf.d/docker-non-verbose     && \
    echo 'Dpkg::Use-Pty 0;'                                         >>/etc/apt/apt.conf.d/docker-non-verbose     && \
    echo 'DPkg::Options { "--force-confmiss"; "--force-confnew" };' >>/etc/apt/apt.conf.d/docker-force-config
RUN apt-get update && \
    apt-get install apt-utils \
                    libcurl4-openssl-dev curl \
                    wget git \
                    libmagic1 \
                    libssl-dev \
                    zlib1g-dev \
                    libdbus-1-dev libglib2.0-dev  && \
    apt-get clean && \
    rm --recursive --force /var/lib/apt/*
RUN eval [ "x$INSTALL_EXAR_USB_DRIVER" = "xYes" ] && ( \
        apt-get update && \
        apt-get install build-essential && \
        apt-get clean && \
        rm --recursive --force /var/lib/apt/* && \
        pip install awscli && \
        rm -rf /root/.cache/pip && \
        git clone https://github.com/kasbert/epsolar-tracer.git && \
        curl --silent --location --remote-name https://raw.githubusercontent.com/balena-io-projects/kernel-module-build/master/build.sh && \
        chmod a+x ./build.sh && \
        ./build.sh %%RESIN_MACHINE_NAME%% $RESIN_OS_VERSION epsolar-tracer/xr_usb_serial_common-1a && \
        rm --recursive --force epsolar-tracer/xr_usb_serial_common-1a \
    ) || true

COPY . /usr/src/app
CMD ["bash","/usr/src/app/start.sh"]

Here is my /usr/bin/entry.sh:

#!/bin/bash

hostname "$HOSTNAME" &> /dev/null
if [[ $? == 0 ]]; then
        PRIVILEGED=true
else
        PRIVILEGED=false
fi

function mount_dev()
{
        mkdir -p /tmp
        mount -t devtmpfs none /tmp
        mkdir -p /tmp/shm
        mount --move /dev/shm /tmp/shm
        mkdir -p /tmp/mqueue
        mount --move /dev/mqueue /tmp/mqueue
        mkdir -p /tmp/pts
        mount --move /dev/pts /tmp/pts
        touch /tmp/console
        mount --move /dev/console /tmp/console
        umount /dev || true
        mount --move /tmp /dev

        # Since the devpts is mounted with -o newinstance by Docker, we need to make
        # /dev/ptmx point to its ptmx.
        # ref: https://www.kernel.org/doc/Documentation/filesystems/devpts.txt
        ln -sf /dev/pts/ptmx /dev/ptmx
        mount -t debugfs nodev /sys/kernel/debug
}

function start_udev()
{
        if [ "$UDEV" == "on" ]; then
                if $PRIVILEGED; then
                        mount_dev
                        if command -v udevd &>/dev/null; then
                                unshare --net udevd --daemon &> /dev/null
                        else
                                unshare --net /lib/systemd/systemd-udevd --daemon &> /dev/null
                        fi
                        udevadm trigger &> /dev/null
                else
                        echo "Unable to start udev, container must be run in privileged mode to start udev!"
                fi
        fi
}

function init()
{
        # echo error message, when executable file doesn't exist.
        if CMD=$(command -v "$1" 2>/dev/null); then
                shift
                exec "$CMD" "$@"
        else
                echo "Command not found: $1"
                exit 1
        fi
}

UDEV=$(echo "$UDEV" | awk '{print tolower($0)}')

case "$UDEV" in
        '1' | 'true')
                UDEV='on'
        ;;
esac

start_udev
init "$@"

The tlshoot.service is:

[Unit]
Description=Timelapse.in TLshoot Balena.io Application

[Service]
EnvironmentFile=/etc/docker.env
ExecStart=/etc/TLshootService.sh
StandardOutput=tty
StandardError=tty
TTYPath=/dev/console
Restart=on-failure

[Install]
WantedBy=basic.target

The tlshoot.sh is:

#!/bin/bash

set -m

# Send SIGTERM to child processes of PID 1.

function signal_handler()

{

kill "$pid"

}

function init_systemd()

{

GREEN='\033[0;32m'

echo -e "${GREEN}Systemd init system enabled."

for var in $(compgen -e); do

printf '%q=%q\n' "$var" "${!var}"

done > /etc/docker.env

echo 'source /etc/docker.env' >> ~/.bashrc

printf '#!/bin/bash\n exec ' > /etc/TLshootService.sh

printf '%q ' "$@" >> /etc/TLshootService.sh

chmod +x /etc/TLshootService.sh

mkdir -p /etc/systemd/system/balena.service.d

cat <<-EOF > /etc/systemd/system/balena.service.d/override.conf

[Service]

WorkingDirectory=$(pwd)

EOF

sleep infinity &

exec env DBUS_SYSTEM_BUS_ADDRESS=unix:path=/run/dbus/system_bus_socket /sbin/init quiet systemd.show_status=0

}

function init_non_systemd()

{

# trap the stop signal then send SIGTERM to user processes

trap signal_handler SIGRTMIN+3 SIGTERM

# echo error message, when executable file doesn't exist.

if CMD=$(command -v "$1" 2>/dev/null); then

shift

"$CMD" "$@" &

pid=$!

wait "$pid"

exit_code=$?

fg &> /dev/null || exit "$exit_code"

else

echo "Command not found: $1"

exit 1

fi

}

INITSYSTEM=$(echo "$INITSYSTEM" | awk '{print tolower($0)}')

case "$INITSYSTEM" in

'1' | 'true')

INITSYSTEM='on'

;;

esac

if [ "$INITSYSTEM" = "on" ]; then

init_systemd "$@"

else

init_non_systemd "$@"

fi

#10

Hello !

It seems that you’re using the example code from here to build the module - that’s great !
I’ve noticed that there is an accompanying script there to nudge the module to be loaded - see run.sh in the said repo. Could you try explicitly invoking insmod from within the container to see if this helps - this is what the said run.sh does.

Thanks !


#11

No need for run.sh, since it has been replaced by my own start.sh. It contains:

#!/bin/bash

while [[ `lsmod | grep cdc_acm | tr --squeeze-repeats " " | cut --delimiter " " --fields 3` -ne '0' ]]
do
    echo "cdc_acm in use, waiting..." 2>&1
    sleep 5
done
rmmod cdc_acm && insmod /usr/src/app/epsolar-tracer/xr_usb_serial_common-1a_raspberrypi3_*.prod/xr_usb_serial_common.ko

The module has been loaded, that is, the /dev/ttyXRUSB0 can be seen on the host. But I do expect to see the same device within the container, too.


#12

Are you running the container in privileged mode?

i.e. in a docker-compose.yml there should be
privileged=true


#13

Yes. No docker-compose.yml here, but the container is running in privileged more.


#14

@shaunmulligan, I can also grant access to the device, if you need.


#15

@shaunmulligan, support access granted for one week. See PM for UUID.


#16

I solved it on my own.

As shown on https://www.balena.io/docs/reference/base-images/base-images/#installing-your-own-initsystem, latest balenaOS does not come with an INITSYSTEM anymore, so you have to install your own, if needed. Examples are provided, but these do not take into account mounting /dev as devtmpfs!