Using Gstreamer with video encoder/decoder accelerators

Hello,

I am trying to get Video encoder/decoder to work with Gstreamer in container on Jetson TX2. I have installed Gstreamer following their documents,

add-apt-repository universe
add-apt-repository multiverse
apt-get update
apt-get install gstreamer1.0-tools gstreamer1.0-alsa gstreamer1.0-
plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad
gstreamer1.0-plugins-ugly gstreamer1.0-libav
apt-get install libgstreamer1.0-dev libgstreamer-plugins-base1.0-
dev libgstreamer-plugins-good1.0-dev libgstreamer-plugins-bad1.0-dev

Gstreamer with h.264 encoder works fine on CPU,

gst-launch-1.0 videotestsrc ! 'video/x-raw, format=(string)I420,width=(int)640, height=(int)480' ! x264enc ! 'video/x-h264, stream-format=(string)byte-stream' ! h264parse ! qtmux ! filesink location=test.mp4 -e

however, on HW accelerator it doesn’t work,

gst-launch-1.0 videotestsrc ! 'video/x-raw, format=(string)I420,width=(int)640, height=(int)480' ! omxh264enc ! 'video/x-h264, stream-format=(string)byte-stream' ! h264parse ! qtmux ! filesink location=test.mp4 -e
WARNING: erroneous pipeline: no element "omxh264enc"

The libgstomx.so is installed on /usr/lib/aarch64-linux-gnu/gstreamer-1.0 but gstreamer cannot use gst-omx and the following command returns nothing,

gst-inspect-1.0 | grep omx

Does anyone know how to enable video accelerators on Jetson tx2?

Hi there @Farzad . I haven’t tried this myself, could you link to the document/tutorial you have been following. I would also maybe try install the packages for gstreamer we have in this tx2 sample project: https://github.com/balena-io-playground/tx2-sample-app/blob/master/Dockerfile#L101

Hi @shaunmulligan , I tried the sample project. It had some error and I added the libraries and packages into my dockefile. Gstreamer still does not work with HW accelerators and gst-inspect returns the following error,

    root@test:/#gst-inspect-1.0 --version

(gst-plugin-scanner:190): GStreamer-WARNING **: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstvideocuda.so': /usr/lib/aarch64-linux-gnu/libgbm.so.1: undefined symbol: drmGetDevice2

(gst-plugin-scanner:190): GStreamer-WARNING **: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvivafilter.so': /usr/lib/aarch64-linux-gnu/libgbm.so.1: undefined symbol: drmGetDevice2

(gst-plugin-scanner:190): GStreamer-WARNING **: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstomx.so': /usr/lib/aarch64-linux-gnu/libgbm.so.1: undefined symbol: drmGetDevice2

(gst-plugin-scanner:190): GStreamer-WARNING **: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libnvgstjpeg.so': /usr/lib/aarch64-linux-gnu/libgbm.so.1: undefined symbol: drmGetDevice2

(gst-plugin-scanner:190): GStreamer-WARNING **: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvegltransform.so': /usr/lib/aarch64-linux-gnu/libgbm.so.1: undefined symbol: drmGetDevice2

(gst-plugin-scanner:190): GStreamer-WARNING **: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnveglglessink.so': /usr/lib/aarch64-linux-gnu/libgbm.so.1: undefined symbol: drmGetDevice2

(gst-plugin-scanner:190): GStreamer-WARNING **: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvvideosink.so': /usr/lib/aarch64-linux-gnu/libgbm.so.1: undefined symbol: drmGetDevice2

(gst-plugin-scanner:190): GStreamer-WARNING **: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstgtksink.so': /usr/lib/aarch64-linux-gnu/libgbm.so.1: undefined symbol: drmGetDevice2

(gst-plugin-scanner:190): GStreamer-WARNING **: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstopengl.so': /usr/lib/aarch64-linux-gnu/libgbm.so.1: undefined symbol: drmGetDevice2
gst-inspect-1.0 version 1.8.3
GStreamer 1.8.3

Could you try installing libdrm2 explicitly? I would have thought this would be brought in with the other packages, but it seems like there can be versioning issues too: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=879508

Hi Cameron,
Thanks for suggestion but it didn’t fix the issue. I am not sure if the following dmesg logs are helpful,

[ 2358.533609] gst-plugin-scan[5588]: unhandled level 2 translation fault (11) at 0x00000000, esr 0x83000006
[ 2358.543211] pgd = ffffffc1e067d000
[ 2358.546626] [00000000] *pgd=0000000259a1a003, *pud=0000000259a1a003, *pmd=0000000000000000

[ 2358.556442] CPU: 3 PID: 5588 Comm: gst-plugin-scan Not tainted 4.4.38-l4t-r28.2+g174510d #2
[ 2358.564792] Hardware name: quill (DT)
[ 2358.568461] task: ffffffc1e4daf080 ti: ffffffc101d14000 task.ti: ffffffc101d14000
[ 2358.575948] PC is at 0x0
[ 2358.578487] LR is at 0x7fa6079b60
[ 2358.581810] pc : [<0000000000000000>] lr : [<0000007fa6079b60>] pstate: 00000000
[ 2358.589204] sp : 0000007fdb439640
[ 2358.592524] x29: 0000007fdb439640 x28: 0000000000000001 
[ 2358.597860] x27: 0000000000000093 x26: 0000000000000093 
[ 2358.603193] x25: 0000000000000000 x24: 0000000000000001 
[ 2358.608543] x23: 0000007fa320c878 x22: 0000007fdb43a4a0 
[ 2358.613878] x21: 0000007fdb43a488 x20: 0000000000000002 
[ 2358.619212] x19: 0000000000000001 x18: 000000000044fcc0 
[ 2358.624554] x17: 0000007fa5d2fa40 x16: 0000007fa10f6528 
[ 2358.629885] x15: 0000000000000061 x14: 0000000000000075 
[ 2358.635219] x13: 000000000000006c x12: 0000000000000077 
[ 2358.640559] x11: 0000000000000000 x10: 0101010101010101 
[ 2358.645892] x9 : 0000000000000042 x8 : 0000000000000060 
[ 2358.651225] x7 : 0000007fa10d9ee4 x6 : 000000000000002f 
[ 2358.656573] x5 : 0000000000500f00 x4 : 0000000000000000 
[ 2358.661911] x3 : 0000007fa30e1dc0 x2 : 0000007fdb43a4a0 
[ 2358.667244] x1 : 0000007fdb43a488 x0 : 0000000000000002 

[ 2358.674083] Library at 0x7fa6079b60: 0x7fa606c000 /lib/aarch64-linux-gnu/ld-2.23.so
[ 2358.681739] vdso base = 0x7fa6097000

The problem here appears to be that your libdrm library has a versioning mismatch than is expected, and doesn’t provide drmGetDevice2 - can you try running ldd /usr/lib/aarch64-linux-gnu/libgbm.so.1 | grep -i drm from inside the container, and it might reveal something helpful.

Hi @CameronDiver

root@3d6c4a4:/# ldd /usr/lib/aarch64-linux-gnu/libgbm.so.1 | grep -i drm 
        libdrm.so.2 => /usr/lib/aarch64-linux-gnu/tegra/libdrm.so.2 (0x0000007f87e77000)

Please let me know if you need any more information. I can also grant support access if you want.

Thanks for getting back to us, it’s always good to enable support access if you could, then when @camerondiver or @shaunmulligan come online again they’ll hopefully be able to take a look.

Hi @chrisys ,
Thanks for your support. The support access is enabled on here is the uuid,
3d6c4a4b6789d98de25d25c788654990
Please let me know if you need any more information.

Thanks, looking at the device now.

oh cool! thank!

Still no luck, checked your device and it seems that there’s really some versioning mismatch. Some libs are used from the /usr/lib/aarch64-linux-gnu/ folder, some libs are used from the /usr/lib/aarch64-linux-gnu/tegra folder, etc. Now your device is offline. I started to reproducing it on my device, but didn’t finish yet. I’ll let you know.

oh sorry about that! I forgot to tell the team to not touch the device :))
It is online now!
Thank you

Hi @zrzka ,
were you able to reproduce the issue on your device?

Hi @Farzad,

not yet, I’m sorry. I have a week off (sick wife, taking care of a child) and will have a chance to look at it by the end of the next week.

Hi @zrzka,

Sorry to hear that. I hope everything goes well on your side.

1 Like

Hi @Farzad,

I was able to reproduce your issue. I followed all steps of the tx2-sample-app project and got the same issue. Slightly different, but basically the same one.

root@cbc6fcb9e49a:/# LD_LIBRARY_PATH=/usr/lib/aarch64-linux-gnu/tegra gst-inspect-1.0 /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstomx.so

(gst-inspect-1.0:7481): GStreamer-WARNING **: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstomx.so': /usr/lib/aarch64-linux-gnu/libgbm.so.1: undefined symbol: drmGetDevice
Could not load plugin file: Opening module failed: /usr/lib/aarch64-linux-gnu/libgbm.so.1: undefined symbol: drmGetDevice

There’s something fishy with these libraries. I’ll continue with my investigation on Monday morning.

1 Like

Hi @Farzad,

spent some time on this issue today, but still no luck. Except the fact that I run out of space on the device and had to delete lot of stuff.

1 Like

Hi @Farzad,
This problem needs deeper investigation from our site. I forwarded the problem to our devices team, which will look into this. They will update this thread once they find more information.

1 Like

Hi @Farzad,

I did some searching on the nvidia forums and put together a Dockerfile which builds gstomx from sources for L4T 28.2: Dockerfile.gstomx

If gstomx is built from sources gstreamer is able to load those elements and use them:

omx:  nvoverlaysink: OpenMax Video Sink
omx:  omxvp9enc: OpenMAX VP9 Video Encoder
omx:  omxvp8enc: OpenMAX VP8 Video Encoder
omx:  omxh265enc: OpenMAX H.265 Video Encoder
omx:  omxh264enc: OpenMAX H.264 Video Encoder
....
omx:  omxh265dec: OpenMAX H.265 Video Decoder
omx:  omxh264dec: OpenMAX H.264 Video Decoder
omx:  omxmpeg4videodec: OpenMAX MPEG4 Video Decoder

Using nvoverlaysink I can also see video on HDMI:

root@90ea535:/# gst-launch-1.0 videotestsrc ! 'video/x-raw, format=(string)I420,width=(int)640, height=(int)480' ! omxh264enc ! 'video/x-h264, stream-format=(string)byte-stream' ! h264parse ! omxh264dec ! nvoverlaysink -e
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Framerate set to : 30 at NvxVideoEncoderSetParameterNvMMLiteOpen : Block : BlockType = 4 
===== MSENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
===== MSENC blits (mode: 1) into tiled surfaces =====
NvMMLiteOpen : Block : BlockType = 261 
TVMR: NvMMLiteTVMRDecBlockOpen: 7647: NvMMLiteBlockOpen 
NvMMLiteBlockCreate : Block : BlockType = 261 
TVMR: cbBeginSequence: 1179: BeginSequence  640x480, bVPR = 0
TVMR: LowCorner Frequency = 100000 
TVMR: cbBeginSequence: 1529: DecodeBuffers = 17, pnvsi->eCodec = 4, codec = 0 
TVMR: cbBeginSequence: 1600: Display Resolution : (640x480) 
TVMR: cbBeginSequence: 1601: Display Aspect Ratio : (640x480) 
TVMR: cbBeginSequence: 1669: ColorFormat : 5 
TVMR: cbBeginSequence:1683 ColorSpace = NvColorSpace_YCbCr601
TVMR: cbBeginSequence: 1809: SurfaceLayout = 3
TVMR: cbBeginSequence: 1902: NumOfSurfaces = 24, InteraceStream = 0, InterlaceEnabled = 0, bSecure = 0, MVC = 0 Semiplanar = 1, bReinit = 1, BitDepthForSurface = 8 LumaBitDepth = 8, ChromaBitDepth = 8, ChromaFormat = 5
TVMR: cbBeginSequence: 1904: BeginSequence  ColorPrimaries = 2, TransferCharacteristics = 2, MatrixCoefficients = 2
Allocating new output: 640x480 (x 24), ThumbnailMode = 0
OPENMAX: HandleNewStreamFormat: 3464: Send OMX_EventPortSettingsChanged : nFrameWidth = 640, nFrameHeight = 480 
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
TVMR: FrameRate = 30 
TVMR: NVDEC LowCorner Freq = (100000 * 1024) 

With this I can also see MSENC and NVDEC hw encoding/decoding engines kicking in.