Balena Dash Request

cpollock · August 21, 2019, 4:33pm

Just a quick update; I have been looking into this code to listen for input. If this yields any results, I will post back here.

pdcastro · August 23, 2019, 6:49pm

@cpollock, I am trying to understand the thread: the original issue was “I am trying to modify the Balena Dash project so the web page reverts to a home page after a period of inactivity,” then Daniel suggested you tried a detect-touch branch of the balena-dash project, then you reported issues with a HDMI+USB touch screen (external monitor).

Was the touch screen (external monitor) working before you tried using the detect-touch branch?
When people are signing the waiver (EZWaiver), are they using a virtual on-screen keyboard or a physical keyboard? The reason I ask is to determine whether “a period of inactivity” can be reset by keys typed on a physical keyboard.
Would you say that the problem that now needs to be solved is the touch screen (external monitor)?
Does the HDMI+USB touch screen (external monitor) work with Linux? E.g. if you flash the SD card with plain Raspbian (without the balena Dash project), does the touch screen work?

cpollock · August 23, 2019, 8:09pm

Hi @pdcastro

I’m sorry for any confusion, but I’ll try to be clearer in this post.

I have my Raspberry Pi hooked up to an external touch screen monitor via USB and HDMI. If you touch the monitor, it simulates a mouse click on-screen. You can see a very similar setup in this video. The issue is that the detect-touch branch of the balena-dash project seems to read the 7" Pimoroni touch screen, which I am not using.

To answer your questions in succession:

The touch screen monitor was working, and continues to work. The only thing that isn’t working is the script trying to detect input.
Users, at this point, are using a physical keyboard connected to the Pi, so the “period of inactivity” could be reset by the keyboard.
The problem is the same as before-- I’m unable to detect mouse and/or keyboard input for our reset script.
The touch screen works with Debian, Linux, Mac and PC. It is currently working in BalenaOS, and doesn’t appear to need any drivers installed.

Thanks so much for your time!

richbayliss · August 23, 2019, 8:30pm

So the issue isnt with the touch not allowing you to navigate your UI, it is just that you want a way to detect of the keyboard/screen hasn’t been touched…?

cpollock · August 23, 2019, 8:32pm

@richbayliss that is correct-- basically an idle timeout feature.

pdcastro · August 23, 2019, 8:49pm

Thank you for the clarification @cpollock. Two thoughts / approaches to consider:

Revisiting your earlier in-browser event interception:

I tried making my own webpage with an iFrame pointing to our EZWaiver, but the refresh script didn’t work inside the iFrame.

Were you intercepting DOM events like clicks and key presses? Might it be worth debugging the “refresh script” that didn’t work? (Sharing the code, mentioning error messages, what the issues were…)

Modifying Daniel’s ‘touch’ container in his ‘detect-touch’ branch of the ‘balena-dash’ project, so that instead of detecting touches, it detects key presses of the physical keyboard. Specifically the start.py script:

from ft5406 import Touchscreen, TS_PRESS, TS_RELEASE, TS_MOVE

ts = Touchscreen()

def touch_handler(event, touch):
    if event == TS_PRESS:
        ...

So that instead of using the ft5406 package to detect touches, it used “something else” that intercepted the physical keyboard presses. It does not have to be Python, by the way… “Anything that runs on Linux” might do.

cpollock · August 23, 2019, 9:51pm

Thanks, I will revisit the in-browser solution, but I think that would break some of the modularity I was hoping to get by using Balena. I will update this thread if I get anywhere with it.

As for solution #2, that was what I was hoping to achieve. I have tried using Pynput and Pyxhook but I am getting the following error (which I posted about previously):

Error Logs

 pyxhook  Traceback (most recent call last):
 pyxhook    File "/usr/lib/python3/dist-packages/Xlib/support/unix_connect.py", line 96, in get_socket
 pyxhook      s = _get_tcp_socket(host, dno)
 pyxhook    File "/usr/lib/python3/dist-packages/Xlib/support/unix_connect.py", line 79, in _get_tcp_socket
 pyxhook      s.connect((host, 6000 + dno))
 pyxhook  ConnectionRefusedError: [Errno 111] Connection refused
 pyxhook  
 pyxhook  During handling of the above exception, another exception occurred:
 pyxhook  
 pyxhook  Traceback (most recent call last):
 pyxhook    File "example_with_parameters.py", line 24, in <module>
 pyxhook      hookman = pyxhook.HookManager(parameters=True)
 pyxhook    File "/usr/src/app/pyxhook.py", line 115, in __init__
 pyxhook      self.local_dpy = display.Display()
 pyxhook    File "/usr/lib/python3/dist-packages/Xlib/display.py", line 89, in __init__
 pyxhook      self.display = _BaseDisplay(display)
 pyxhook    File "/usr/lib/python3/dist-packages/Xlib/display.py", line 71, in __init__
 pyxhook      protocol_display.Display.__init__(self, *args, **keys)
 pyxhook    File "/usr/lib/python3/dist-packages/Xlib/protocol/display.py", line 90, in __init__
 pyxhook      self.socket = connect.get_socket(name, protocol, host, displayno)
 pyxhook    File "/usr/lib/python3/dist-packages/Xlib/support/connect.py", line 87, in get_socket
 pyxhook      return mod.get_socket(dname, protocol, host, dno)
 pyxhook    File "/usr/lib/python3/dist-packages/Xlib/support/unix_connect.py", line 113, in get_socket
 pyxhook      raise error.DisplayConnectionError(dname, str(val))
 pyxhook  Xlib.error.DisplayConnectionError: Can't connect to display "localhost:0": [Errno 111] Connection refused

I think both of your solutions have merit, but the 2nd approach would be ideal. Let me know if if you find anything-- I will continue my efforts on my end.

pdcastro · August 23, 2019, 11:27pm

What you are saying makes sense – thanks for explaining it further. However… Something I noticed with the Pynput and Pyxhook potential solutions is that they intercept input events (touch, mouse, keyboard) using an X11 server, however the “wpe” container (WPE WebKit web browser) uses Wayland instead of X11. If you are running an X11 server independently of Wayland… I wonder how they would play together: they can’t both control the same screen, for one thing. The following page describes a scenario where X clients connect to Wayland for compatibility:

https://wayland.freedesktop.org/xserver.html

Talking of X “clients”:

pyxhook  Xlib.error.DisplayConnectionError: Can't connect to display "localhost:0": [Errno 111] Connection refused

I understand that X11 applications are client-server applications, communicating via network sockets. I think the error message above means that an X client (like a Python process trying to intercept input events) cannot connect to the port number of the X11 server. If I recall correctly, when it comes to X11 connections, the “:0” notation is an offset to TCP port number 6000. So “localhost:0” actually means TCP port 6000 on localhost, “localhost:1” means TCP port 6001 on localhost, and so on. Are you running the X11 sever in the same container as the client? Is an X11 server even actually running? The “netstat -ant” command shows listening sockets: check for a process listening on port 6000.

pdcastro · August 23, 2019, 11:40pm

I wonder if the following Python projects would allow detecting input events without the need to use either X11 or Wayland, building directly on Linux kernel support:

These would be replacements for the ft5406 touchscreen package in the start.py script.

cpollock · August 29, 2019, 9:24pm

Thanks for that info @pdcastro

While I work out the keyboard issue, I edited the Scheduler container to restart the browser every 20 minutes.

I’m having a very difficult time getting python-evdev and keyboard installed, since there are quite a few dependancies, and I run into new errors every time I try a new method of installing. Since the keyboard package needs to be installed via PIP, I’ve tried installing Python by using Balena’s base image, and also installing Python via apt-get in the Debian base. Both work, but neither will allow me to install python-dev or linux-headers. I also get errors with apt-utils depending on what I comment out.

Have you had any luck? Here is my current dockerfile: https://pastebin.com/n8dKAMj4

chrisys · August 29, 2019, 10:43pm

@cpollock have you tried using the build variant of our base images? You could use balenalib/%%BALENA_MACHINE_NAME%%-debian-python:build

cpollock · August 30, 2019, 8:35am

@chrisys yes— I’ve tried Debian, Debian-python, Debian-python:3, Debian-python:3.7, and Debian-python:2.7.

imrehg · August 30, 2019, 3:18pm

Hey @cpollock none of those you mentioned are build variants, unless you shortened what tags you tried. Was it actually ...-debian:build, ...debian-python:build, ..debian-python:3-buildetc…?

In general, the Python packaging is tricky. I would suggest:

If you use balealib/....-python images, install your Python dependencies always from Pip. This way you won’t badly interact with the system python
If you want to use Debian Python packages for anything (any python package installed with apt-get), then use the balenalib/.....-debian images, and install the system Python (can use both apt-get and pip in that case, but you have to use the system Python, there’s no second balena-installed Python there.

You can see more about the base images in our base images documentation

If you do this, I would suggest using balenalib/%BALENA_MACHINE_NAME%%-debian:buster-build, for example.

In the Dockerfile you shared, you also install linux-headers-$(uname -r), that might not work since here the OS has different headers, not the debian kernel. Is that really needed?

For the packages, you also likely need python3-dev python3-pip (not python-, which is python v2 packages, depending on what Python you install, as you seem to mix it in the Dockerfile.

FROM balenalib/%%BALENA_MACHINE_NAME%%-debian

RUN apt-get update
RUN apt-get update && apt-get install -y --no-install-recommends apt-utils
RUN sudo apt-get install python3-pip
RUN install_packages wget unzip curl
RUN apt-get install python-dev python-pip gcc

in the beginning for example can be replaced by

FROM balenalib/%%BALENA_MACHINE_NAME%%-debian:buster

RUN install_packages wget unzip curl python3-dev python3-pip gcc

and use pip3 as a command later, as that’s the correct one in Debian
(or RUN install_packages wget unzip curl python-dev python-pip gcc and use pip, with Python 2).

Does this help to progress somewhat? We are looking deeper too.

cpollock · August 30, 2019, 10:47pm

Thanks for that-- I apologize for my sloppy writing before-- I was typing on my iPhone. I meant to say that I had tried multiple base images including:

balenalib/%%BALENA_MACHINE_NAME%%-debian
balenalib/%%BALENA_MACHINE_NAME%%-debian-python
balenalib/%%BALENA_MACHINE_NAME%%-debian-python3
balenalib/%%BALENA_MACHINE_NAME%%-debian-python2.7
balenalib/%%BALENA_MACHINE_NAME%%-python
balenalib/%%BALENA_MACHINE_NAME%%-python3
balenalib/%%BALENA_MACHINE_NAME%%-python2.7

All of these yielded similar results, including your example code. I am trying to execute the code to build the dependencies for the Keyboard package. From what I understand, you need to install python-evdev, which requires this code to install..

From there, I should be able to execute pip install keyboard and run some test code to check for input. Since I’m running into so many errors, I feel like I’m doing something wrong though. Thanks for your time!

Terminal Log

    [keyboard]        Step 5/13 : RUN apt-get install linux-headers-$(uname -r)
    [keyboard]         ---> Running in dbc9592aa18b
    [keyboard]        Reading package lists...
    [keyboard]        Building dependency tree...
    [keyboard]        Reading state information...
    [keyboard]        E: Unable to locate package linux-headers-4.15.0-45-generic
    [keyboard]        E: Couldn't find any package by glob 'linux-headers-4.15.0-45-generic'
    [keyboard]        E: Couldn't find any package by regex 'linux-headers-4.15.0-45-generic'
    [keyboard]
    [keyboard]        Removing intermediate container dbc9592aa18b
    [Info]            Uploading images
    [keyboard]        The command '/bin/sh -c apt-get install linux-headers-$(uname -r)' returned a non-zero code: 100`

Edit: I am also getting the following error a lot-- any insight on this?
[keyboard] debconf: delaying package configuration, since apt-utils is not installed

Edit 2: I’m sorry but it looks like I skipped over a paragraph of your last post. I am looking into the necessity of linux-headers-$(uname -r). At this point, it looks like a necessity for python-evdev.

cpollock · August 31, 2019, 12:21am

I apologize for the double post, but I was finally able to get the Dockerfile to compile. I will see if I can modify the code and get the idle script to work after the holiday weekend.

Here is the Dockerfile that eventually compiled: https://pastebin.com/a3MtUX4p

pdcastro · August 31, 2019, 2:37am

@cpollock, I don’t think the Keyboard package requires python-evdev – I think they’re independent, alternative packages to choose from. I was able to install the Keyboard package with “pip install keyboard”, but then it failed to run on an RPi 3 with some errors related to input device detection. I debugged it a bit and came up with a one-line patch that worked for my USB keyboard (I hardcoded '/dev/input/event0' instead of using the automatic detection). Because of the patch, in the end I used “git clone” instead of “pip install” to install the Keyboard package.

The following Dockerfile.template file can be pushed to balena and results in an application that prints keyboard key presses to the web dashboard Logs window:

FROM balenalib/%%BALENA_MACHINE_NAME%%-python:3
RUN apt-get update && apt-get install kbd git patch
RUN mkdir -p /usr/src/mydash
WORKDIR /usr/src/mydash
RUN git clone https://github.com/boppreh/keyboard
COPY keyboard.patch /usr/src/mydash/keyboard/
RUN cd keyboard && patch -p1 < keyboard.patch
ENV PYTHONPATH="/usr/src/mydash/keyboard:${PYTHONPATH}"
CMD python keyboard/examples/stdin_stdout_events.py

There’s a COPY instruction above that refers to a keyboard.patch file that can be found in this gist. Place the file in the same folder as the Dockerfile.template file, then run “balena push appName” or similar deployment command.

cpollock · September 6, 2019, 11:27pm

Thanks everyone-- I finally got it to work! I am publishing the code that worked to my new GitHub. Feel free to use it for whatever you want, and let me know if I can improve it.

pdcastro · September 7, 2019, 2:39am

Nice one, and thanks for sharing!