creating a lowish-latency camera for a video stream: close yet still high latency

Product support
1

Hello balena gurus,

I’m working out a Twitch stream that could use a couple of cameras, one for me and one for a robot co-host (which is also being built on balena). It’s a bit silly for sure, yet is helping in getting more comfortable on camera, good for my work (and it’s fun!).

Challenge:
Webcams are really expensive right now.

Opportunity:
I have a couple of Pis with camera modules that are collecting dust. Note: I’m recording audio separately, just focusing on video.

I’ve made some progress so far. balenaCam was a great start, streaming video, accessible via a web page that I could view in the browser. Browser sources can be included as a layer in an OBS (Open Broadcaster Software) video streams and I could see decent quality images coming in from balenaCam. Cool!

balenaCam hasn’t been the best for a live stream use case though as there’s about a one second delay to see it in a browser on the same wired network (viewing over ethernet on same switch). Much was learned from it though and I see it for other uses.

rpisurveillance in the balena playground was great to look at. It streams to its own RTSP server that can be accessed via VLC Player on the PC. I liked that the frame rate was higher and less compressed, using the Pi’s video encoding capabilities as part of that. As OBS has VLC Player built-in for layer use, I was also able to include it efficiently there.

Challenge with this is that it still had roughly 700ms-ish to one second of latency. I could use this in a pinch for sure as it’s possible to delay audio to match video. It just hits a social awkward point in a stream with that extra second, kind of like you would see with live news coverage on the other side of the world. Not the best for a Twitch stream where there’s timely action happening. Maybe there are settings that can be tweaked to reduce latency (tried everything in VLC Player)? Frame rate and dimensions don’t seem to make a noticeable difference on the Pi side so far.

Then I cam across a service called alohacam. They claim “Zero latency (well, almost)”. I set up an account and took a look at their install code. I haven’t moved too far down this road as I didn’t really want another 3rd party in the mix and the browser viewer didn’t quite work for stream needs. It seems like an earlier stage experiment for a company (nice site design in general though). It also would need some work to get it to work in balena where I want to deploy and monitor.

One last idea: I read that you could pipe raspivid output to an internal RTSP stream with one line of code (hah will expand for sure):

raspivid -o - -t 0 -n | cvlc -vvv stream:///dev/stdin --sout '#rtp{sdp=rtsp://:8554/}' :demux=h264

That’s cool and could be something to try. However, when I do, I get vlc errors on the Pi as VLC really doesn’t want to run as root. I’ll need to figure out how to build from source with the --enable-run-as-root flag in the Docker build to try it out. Otherwise/and, I get errors when running with bless:

raspivid -o - -t 0 -n | bless cvlc -vvv stream:///dev/stdin --sout '#rtp{sdp=rtsp://:8554/}' :demux=h264

Error recalling from bad memory: Not able to to connect to a likely undefined display (more setup likely with x server on headless Pi?

Here’s a starting point to play with:

FROM balenalib/%%BALENA_MACHINE_NAME%%-node

# Install dependencies
RUN install_packages build-essential libraspberrypi-bin vlc bless

WORKDIR /usr/src/app

# This will copy all files in our root to the working  directory in the container
COPY . ./

# Enable udevd so that plugged dynamic hardware devices show up in our container.
ENV UDEV=1

# server.js will run when container starts up on the device
CMD ["/bin/bash", "/usr/src/app/start.sh"]

Also, maybe I could make a Pi Zero a webcam?? This makes me think it could* be possible. However I have no idea on where to start there:

At the moment I need to give it a short break and document somewhere. How about here? :smiley:

Any thoughts on this? Have I reached max potential or do you have ideas on how to make a less-latency video stream from a Pi? Thanks for taking a look in advance!

4

I have used RTSP recently to livestream some sports events and it seems that delay is just par for the course. Even with my desktop live streaming there is always some delay. The RT part of RTSP should be (D)RT, for delayed :laughing:

I wouldn’t go down the Zero route as the CPU performance is quite bad vs a Pi 3a which isn’t much bigger :+1:

1 Like
6

Hi, you may give gstreamer a go, since as far as I know it gives lowest latency after good optimizing. An example: https://stackoverflow.com/questions/16750395/raspberry-pi-no-delay-10ms-video-stream
Thanks,
Zahari

1 Like
8

@majorz thanks for the feedback! That helps clear the path to start.

I’m excited to check this one out and will post what I find. :slight_smile:

10

What a wild weekend! I’ll take another please.

So far I’ve learned that gstreamer is very robust and a rabbit hole (will get into that shortly). I haven’t given up yet.

While taking a break from that last evening, I was able to get raspivid (Pi) -> cvlc (Pi) -> VLC Player (PC/Mac) working. It needed a user created and and assigned to the video group as part of the build processes. Downside with this one though is the latency was 1+ seconds and somewhat inconsistent. It’s not ideal, so moving on. Here’s that solution for future reference once I block out the pain, forget and ask this question again a year later…

docker-compose.yml

version: '2'
services:
  server:
    privileged: true
    build: .
    ports:
      - "8090:8090"

Dockerfile.template

FROM balenalib/%%BALENA_MACHINE_NAME%%-node

# Install dependencies
RUN install_packages build-essential vlc

RUN useradd -m -g video vlcuser

WORKDIR /usr/src/app

# This will copy all files in our root to the working  directory in the container
COPY . ./

# Enable udevd so that plugged dynamic hardware devices show up in our container.
ENV UDEV=1

# server.js will run when container starts up on the device
CMD ["/bin/bash", "/usr/src/app/start.sh"]

start.sh

su vlcuser -c "raspivid -o - -w 640 -h 480 -t 0 -n -hf -vf -g 25 -pf high -md 4 -fps 25 -b 10000000 | cvlc -vvv stream:///dev/stdin --no-audio --low-delay --sout '#standard{access=http,mux=ts,dst=:8090}' :live-caching=150 :network-caching=150 :clock-jitter=0 :clock-syncro=0 :demux=h264"

After deployment, in VLC on the PC/Mac simply load a network/stream e.g. http://192.168.0.19:8090

Back to gstreamer…

I feel like I’m very close. There are a lot of examples out there from different years in getting either /dev/video0 or raspivid to send data to gstreamer and broadcasting. Some of the examples were old and just didn’t run on the Pi. Some were generating errors about not being able to access a display. Others I could get “running”, yet I couldn’t get VLC connected with a proper .sdp file due to garbled data/garbage (data mismatch or bad conversion I’m guessing). It’s a bit tough as the documentation itself is HUGE and I can see many others out there having unanswered questions when getting started.

I may try installing a full Raspbian (can I do that with balena?) next and see if I can get one of those examples running. Maybe I’m missing a permission or other libraries. Here’s what I have so far that’s buffering on the Pi and not being read properly in VLC:

Dockerfile.template

FROM balenalib/%%BALENA_MACHINE_NAME%%-node

# Install dependencies
# NOTE: This could be overkill, and missing something
# TODO: Remove overkill, and find something
RUN install_packages build-essential libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev gstreamer1.0-plugins-base-apps gstreamer1.0-plugins-bad gstreamer1.0-plugins-good gstreamer1.0-plugins-ugly gstreamer1.0-tools gstreamer1.0-omx

RUN useradd -m -g video vlcuser

WORKDIR /usr/src/app

# This will copy all files in our root to the working  directory in the container
COPY . ./

# Enable udevd so that plugged dynamic hardware devices show up in our container.
ENV UDEV=1

# server.js will run when container starts up on the device
CMD ["/bin/bash", "/usr/src/app/start.sh"]

start.sh (many other failed trials if interested in those)

vlcuser -c "cvlc -vvv v4l2:// :v4l2-dev=/dev/video0 :v4l2-width=640 :v4l2-height=480 :v4l2-fps 25 --sout '#standard{access=http,mux=raw,dst=:8090}'"

.sdp file that gets loaded like a video file on PC/Mac VLC:

c=IN IP4 192.168.0.21
m=video 8090 RTP/AVP 96
a=rtpmap:96 H264/90000

Any thoughts on how to proceed? I’ll keep digging.

UPDATE: The answer is in gstreamer -> gstreamer. I’ll post a full solution very soon.

12

@majorz @richbayliss:

Here’s where I landed (works great):

I need to take another look soon after a break as I wrote a lot of README. Feedback welcome. LOVE the balena button functionality for sharing. :smiley:

Thanks for your great support and ideas!

1 Like