Having difficulty using OpenCV to display a video through HDMI

raspberrypi3
docker

#1

Hello everybody,

I just wanted to find out if anybody knows of a solution to output via HDMI with Balena OS. I’m currently using OpenCV to display a video, which works perfectly fine on my Mac, but when put onto Balena, it throws this error when it runs:

(window:212): Gtk-WARNING **: cannot open display:

I’m currently using a Raspberry Pi model 3 for testing. I’ve been searching the web for similar issues, and they do exist, but none provide a specific way to fix this with Balena/ Resin. The following code is my Dockerfile:

FROM resin/%%BALENA_MACHINE_NAME%%-python:2.7
# Install dependencies needed for building and running OpenCV
RUN apt-get update && apt-get install -y --no-install-recommends \
# to build and install
unzip \
build-essential cmake pkg-config \
# to work with images
libjpeg-dev libtiff-dev libjasper-dev libpng-dev \
# to work with videos
libavcodec-dev libavformat-dev libswscale-dev libv4l-dev \
# needed by highgui tool
libgtk2.0-dev \
# for opencv math operations
libatlas-base-dev gfortran \
# others
libtbb2 libtbb-dev \
# cleanup
&& rm -rf /var/lib/apt/lists/* \
&& apt-get -y autoremove

# Install python packages
RUN pip install --no-cache-dir \
# OpenCV dependency
numpy \
# other usefull stuff
ipython \
# cleanup
&& find /usr/local \
   \( -type d -a -name test -o -name tests \) \
   -o \( -type f -a -name '*.pyc' -o -name '*.pyo' \) \
   -exec rm -rf '{}' + \
&& cd / \
&& rm -rf /usr/src/python ~/.cache

# Install OpenCV
COPY download_build_install_opencv.sh download_build_install_opencv.sh
RUN chmod +x ./download_build_install_opencv.sh && ./download_build_install_opencv.sh

# Setting working directory
WORKDIR /usr/src/app

COPY ./requirements.txt /requirements.txt
# Gotta get those requirements
RUN pip install -r /requirements.txt

COPY . ./

# Initializing system
ENV INITSYSTEM on

# Running initial script
CMD ["python","tv.py"] 

This is my Python script:

import cv2
import os

def playVideo():
    cap = cv2.VideoCapture('test.mp4')

    cv2.namedWindow("window", cv2.WND_PROP_FULLSCREEN)
    cv2.setWindowProperty("window", cv2.WND_PROP_FULLSCREEN, cv2.WINDOW_FULLSCREEN)

    while cap.isOpened():
        ret, frame = cap.read()
        if ret:
            cv2.imshow('window', frame)
            if cv2.waitKey(10000) & 0xFF == ord('q'):
                break
        else:
            cap.set(cv2.CAP_PROP_POS_FRAMES, 0)

    cap.release()
    cv2.destroyAllWindows()

playVideo()

Balena is some amazing software, but I can’t seem to figure out how to solve this issue! If anybody knows the answer to my question, please send it here!

Thanks,
Tate


#4

@tate, my very first guess would be that you need to add a windows manager into your container, by default the debian in base images doesn’t have any GUI or windows manager. Have a look at https://github.com/balena-io/resin-electronjs as an example of setting up a basic windows manager, you don’t need all the electron stuff though.


#5

Thank you for the amazingly quick response. I’m actually still much of a newbie at Docker, so I’m wondering what packages I’ll need to add to my project and how to use them in Python?
Thanks!


#6

@shaunmulligan So I’ve done a bit more research, and I came across this post:


I tried installing pygame, but I could not figure out how to use it with my OpenCV script despite an example being given.
I also tried installing dependencies listed at the bottom of the thread, but this only opened a login screen when I rebooted the device. I couldn’t figure out how to login with credentials, and the python script threw the same error.


#7

When I tried running startx I received this error:

root@45b7d30:/usr/src/app# startx hostname: No address associated with hostname xauth: (stdin):1: bad display name "45b7d30:0" in "add" command /etc/X11/xinit/xserverrc: 3: exec: /usr/bin/X: not found

#8

@tate did you install all of the packages listed in the link you posted? If you look at the last line of your error message you can see that startx wasn’t able to do so because the X binary itself wasn’t found. I believe it’s part of xserver-xorg-core.


#10

hey @tate, it looks like we need to investigate a bit more on this issue and probably try create a demo project on how to use X11 on balenaOS, so one of our team is gonna look into that over the coming days.


#12

xserver-xorg-core Couldn’t be found when I did apt-get install xserver-xorg-core


#13

@shaunmulligan This would be incredibly beneficial for me! Sorry for my late response as well.

Should I allow support to see my project code if this would help you to find my error?


#14

Hi,

First of all, we took a note to ping you once we create a demo project. We cannot promise a timeline but it will happen eventually : )

About the error you are getting: Could you share the error message and logs you see?
A clean re-install might remedy the issue. But it’s tough to advise before understanding the error.


#15

@tate it is possible, that you just need the package cache to be updated?

Normally the base images are shipped with package cache taken out, to reduce the size. If you see something like this:

root@a861bbb6ef31:/usr/src/app# apt-get install xserver-xorg-core
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package xserver-xorg-core

then you have to run apt-get update beforehand.
After that on my device at least it finds the xserver-xorg-core package and can install it.

Or if you use our new balenalib base images, you can use the install_packages utility that we add:

https://www.balena.io/docs/reference/base-images/base-images/#installing-packages

Let us know if it helps you progress !


#17

Hi @tate, not sure if you are still trying to get x11 working on your device, but we put together a simple example app over here: https://github.com/balena-io-playground/x11-window-manager which sets up a pretty standard desktop environment, so that would be a good place to start from.


#20

This fixed most of my problems! Thanks!

I’m now dealing with removing the cursor during video playback, and also the resolution of the display has been lowered dramatically. I’m not quite sure how to fix either of these issues


#21

Hiding the cursor

In order to hide cursor from X for the duration of its runtime, you can use the nocursor option when starting X. Below are several variants depending on how you start X:

Xorg -nocursor
X -nocursor
xinit -- -nocursor
startx -- -nocursor

The repo linked earlier in the thread starts X in start.sh (linked) using startx, so you can just add the flag to that file.

If you want to toggle the cursor, for example during playback of videos, you’d be looking at using one of the various more-or-less hack-y solutions out there, since X doesn’t have a uniform way of hiding the cursor across runtimes / toolkits. Two such examples are:

Resolution

The resolution of the display can be controlled by xrandr, for example, this is a command I use to control the resolution of my HDMI1 output:

xrandr --output HDMI1 --mode 1360x768

You can run xrandr with no arguments to see the available outputs, their states, and the resolutions they support.