Jetson: Support Nvidia Docker Images

Product support balenaOS
44

Hi @scarlyon ,

i would recommend you to wait for the cleaned up repo, that i am planing to publish. Anyhow my notes might still be interesting to some. Specially the google bot :slight_smile:

Please take these notes with caution, since they are incomplete and are just a documentation of my debugging process.

Here are my notes

  • get it building

    #poky/meta/classes/go.bbclass

remove trimpath

  • Just adding nvidia-container-toolkit layers/meta-balena-jetson/recipes-core/images/balena-image.inc

    IMAGE_INSTALL:append:jetson-xavier-nx-devkit = " \
    tegra194-nxde-sdcard-flash \
    fan-startup \
    parted \
    gptfdisk \
    tegra-nvpmodel \
    tegra-configs-nvstartup \
    tegra-configs-udev \
    mtd-utils \
    tegra-bluetooth \
    tegra-wifi \
    tegra-firmware-rtl8822 \
    tegra-udrm-probeconf \
    linux-firmware-bcm4354 \
    tegra-firmware-xusb \
    cuda-driver \
    tegra-libraries \
    libnvidia-container-tools \
    go-runtime \
    nvidia-container-toolkit \
    tegra-argus-daemon \
    libvisionworks-sfm \
    libvisionworks-tracking \
"

    balena run -it --gpus all nvcr.io/nvidia/l4t-ml:r32.5.0-py3 bash

    python3
>>> import tensorflow as tf
2021-12-02 14:46:19.410548: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.10.2'; dlerror: libcudart.so.10.2: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64:/usr/local/cuda-10.2/targets/aarch64-linux/lib:
2021-12-02 14:46:19.410673: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2021-12-02 14:46:19.410987: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.10.2'; dlerror: libcudart.so.10.2: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64:/usr/local/cuda-10.2/targets/aarch64-linux/lib:
2021-12-02 14:46:19.411033: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Segmentation fault (core dumped)

    ⇒ gpu flag works, but no cuda available

  • exteneded with cudnn, … layers/meta-balena-jetson/recipes-core/images/balena-image.inc

    # Heavily increased to ~8GB
IMAGE_ROOTFS_SIZE = "3899392"

# Bootloader blob is 32MB on the NX
IMAGE_ROOTFS_SIZE:jetson-xavier-nx-devkit-emmc = "2932736"
IMAGE_ROOTFS_SIZE:jetson-xavier-nx-devkit = "2932736"

IMAGE_INSTALL:append:jetson-xavier-nx-devkit = " \
    tegra194-nxde-sdcard-flash \
    fan-startup \
    parted \
    gptfdisk \
    tegra-nvpmodel \
    tegra-configs-nvstartup \
    tegra-configs-udev \
    mtd-utils \
    tegra-bluetooth \
    tegra-wifi \
    tegra-firmware-rtl8822 \
    tegra-udrm-probeconf \
    linux-firmware-bcm4354 \
    tegra-firmware-xusb \
    cuda-driver \
    tegra-libraries \
    libnvidia-container-tools \
    go-runtime \
    nvidia-container-toolkit \
    tegra-argus-daemon \
    cuda-toolkit \
    cudnn \
    libvisionworks \
    libvisionworks-sfm \
    libvisionworks-tracking \
    cuda-libraries \
"

    → Works but library mapping missmatch in l4t.csv, …

  • try to upgrade to newer meta-tegra and corrected csv

    balena run -it --gpus all nvcr.io/nvidia/l4t-ml:r32.6.1-py3 bash

    Copied: recipes-containers from meta tegra honister and removed docker and virtualization. Furthermore removed -stripath arg of go build.

    IMAGE_INSTALL:append:jetson-xavier-nx-devkit = " \
		tegra194-nxde-sdcard-flash \
    fan-startup \
    parted \
    gptfdisk \
    tegra-nvpmodel \
    tegra-configs-nvstartup \
    tegra-configs-udev \
    mtd-utils \
    tegra-bluetooth \
    tegra-wifi \
    tegra-firmware-rtl8822 \
    tegra-udrm-probeconf \
    linux-firmware-bcm4354 \
    tegra-firmware-xusb \
    cuda-driver \
    tegra-libraries-core \
    tegra-libraries-camera \
    tegra-libraries-cuda \
    tegra-libraries-eglcore \
    tegra-libraries-gbm \
    tegra-libraries-glescore \
    tegra-libraries-multimedia \
    tegra-libraries-multimedia-utils \
    tegra-libraries-multimedia-v4l \
    libnvidia-container-tools \
    go-runtime \
    nvidia-container-toolkit \
    nvidia-container-runtime \
    tegra-argus-daemon \
    tegra-tools-tegrastats \
    cuda-toolkit \
    cuda-libraries \
    cudnn \
    libvisionworks \
    libvisionworks-container-csv \ 
    libvisionworks-sfm \
    libvisionworks-sfm-container-csv \
    libvisionworks-tracking \
    libvisionworks-tracking-container-csv
    vim \

    fixing cuda.csv, cudnn.csv references to /usr/lib/ path

  • check l4t.csv links

    cat /etc/nvidia-container-runtime/host-files-for-container.d/l4t.csv | while read line; do file=$(echo $line | sed "s/.*, //g"); if [ -f $file ]; then echo $line >> working.txt; fi; done

cat /etc/nvidia-container-runtime/host-files-for-container.d/l4t.csv | while read line; do file=$(echo $line | sed "s/.*, //g"); if [ ! -f $file ]; then echo $line >> not_working.txt; fi; done

  • add nvargus support, …

    balena run -it --ipc=host -v /tmp/argus_socket:/tmp/argus_socket --cap-add SYS_PTRACE --device /dev/video0:/dev/video0 --gpus all nvcr.io/nvidia/l4t-ml:r32.6.1-py3 bash

    # remove nvargus support
# layers/meta-balena-jetson/conf/layer.conf

-- BBMASK += "/meta-tegra/recipes-multimedia/gstreamer/"

    IMAGE_INSTALL:append:jetson-xavier-nx-devkit = " \
    tegra194-nxde-sdcard-flash \
    fan-startup \
    parted \
    gptfdisk \
    tegra-nvpmodel \
    tegra-configs-nvstartup \
    tegra-configs-udev \
    mtd-utils \
    tegra-bluetooth \
    tegra-wifi \
    tegra-firmware-rtl8822 \
    tegra-udrm-probeconf \
    linux-firmware-bcm4354 \
    tegra-firmware-xusb \
    cuda-driver \
    tegra-libraries-core \
    tegra-libraries-camera \
    tegra-libraries-cuda \
    tegra-libraries-eglcore \
    tegra-libraries-gbm \
    tegra-libraries-glescore \
    tegra-libraries-multimedia \
    tegra-libraries-multimedia-utils \
    tegra-libraries-multimedia-v4l \
    libnvidia-container-tools \
    go-runtime \
    nvidia-container-toolkit \
    nvidia-container-runtime \
    tegra-argus-daemon \
    tegra-tools-tegrastats \
    cuda-toolkit \
    cuda-libraries \
    cudnn \
    libvisionworks \
    libvisionworks-container-csv \ 
    libvisionworks-sfm \
    libvisionworks-sfm-container-csv \
    libvisionworks-tracking \
    libvisionworks-tracking-container-csv \
    tegra-mmapi \
    tegra-mmapi-dev \
    gstreamer1.0 \
    gstreamer1.0-plugins-base \
		gstreamer1.0-plugins-tegra \
    isp-config \
    vim \
"

    # try with
apt update
apt-get install software-properties-common build-essential -y
add-apt-repository ppa:ubuntu-toolchain-r/test

# press enter

apt update
apt-get install software-properties-common -y 
apt install gcc-9 gcc-10 -y 
apt dist-upgrade -y

# Validate existance of GLIBCXX_3.4.29
strings /usr/lib/aarch64-linux-gnu/libstdc++.so.6 | grep GLIBCXX

GST_DEBUG=4 gst-inspect-1.0  /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvarguscamerasrc.so

apt install gtk-doc-tools
git clone https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad.git
cd gst-plugins-bad/
git checkout 1.14.5
./autogen.sh
make

gst-launch-1.0 -e nvcamerasrc sensor-id=0 ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=40/1" ! nvv4l2h264enc ! h264parse ! mp4mux ! filesink location=rpi_v3_imx477_cam0.mp4

    layers/poky/meta/conf/distro/include/tcmode-default.inc
-> set GLIBCVERSION ?= "2.27"

/home/bombus/langhalsdino/yocto/balena-jetson/balena-jetson-nx/layers/poky/meta/conf/distro/include/yocto-uninative.inc
-> set UNINATIVE_MAXGLIBCVERSION = "2.27"

    Issue logs

    GST_DEBUG=4 gst-inspect-1.0  /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvarguscamerasrc.so

(gst-plugin-scanner:13): GStreamer-WARNING **: 05:47:19.297: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvv4l2camerasrc.so': /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /usr/lib/aarch64-linux-gnu/libv4l2.so.0)
0:00:00.828793344    12   0x55aa605a00 INFO            GST_REGISTRY gstregistry.c:1694:scan_and_update_registry: Registry cache changed. Writing new registry cache
0:00:00.828921568    12   0x55aa605a00 INFO            GST_REGISTRY gstregistrybinary.c:369:priv_gst_registry_binary_write_cache: Building binary registry cache image
0:00:00.863567584    12   0x55aa605a00 INFO            GST_REGISTRY gstregistrybinary.c:401:priv_gst_registry_binary_write_cache: Writing binary registry cache
0:00:00.926758272    12   0x55aa605a00 INFO            GST_REGISTRY gstregistrybinary.c:262:gst_registry_binary_cache_finish: Wrote binary registry cache
0:00:00.926833536    12   0x55aa605a00 INFO            GST_REGISTRY gstregistry.c:1703:scan_and_update_registry: Registry cache written successfully
0:00:00.926885760    12   0x55aa605a00 INFO            GST_REGISTRY gstregistry.c:1762:ensure_current_registry: registry reading and updating done, result = 1
0:00:00.927033888    12   0x55aa605a00 INFO                GST_INIT gst.c:807:init_post: GLib runtime version: 2.56.4
0:00:00.927076512    12   0x55aa605a00 INFO                GST_INIT gst.c:809:init_post: GLib headers version: 2.56.4
0:00:00.927105248    12   0x55aa605a00 INFO                GST_INIT gst.c:810:init_post: initialized GStreamer successfully
0:00:00.935838144    12   0x55aa605a00 WARN      GST_PLUGIN_LOADING gstplugin.c:792:_priv_gst_plugin_load_file_for_registry: module_open failed: /usr/lib/aarch64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvarguscamerasrc.so)

(gst-inspect-1.0:12): GStreamer-WARNING **: 05:47:19.427: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvarguscamerasrc.so': /usr/lib/aarch64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvarguscamerasrc.so)
Could not load plugin file: Opening module failed: /usr/lib/aarch64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvarguscamerasrc.so)

    after installing cpp, …

    GST_DEBUG=4 gst-inspect-1.0  /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvarguscamerasrc.so

0:00:00.240528064  6364   0x558fd6ba00 WARN      GST_PLUGIN_LOADING gstplugin.c:792:_priv_gst_plugin_load_file_for_registry: module_open failed: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvarguscamerasrc.so)

(gst-inspect-1.0:6364): GStreamer-WARNING **: 05:53:23.725: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvarguscamerasrc.so': /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvarguscamerasrc.so)
Could not load plugin file: Opening module failed: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvarguscamerasrc.so)

    Quick fix (copy from nvidia base image):

    mv /usr/lib/gstreamer-1.0/libgstnvarguscamerasrc.so /usr/lib/gstreamer-1.0/libgstnvarguscamerasrc.so.bckp
cp /home/root/libgstnvarguscamerasrc.so /usr/lib/gstreamer-1.0/libgstnvarguscamerasrc.so

0:00:05.615482720    12   0x55aa716a00 INFO      GST_PLUGIN_LOADING gstplugin.c:901:_priv_gst_plugin_load_file_for_registry: plugin "/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvarguscamerasrc.so" loaded
Plugin Details:
  Name                     nvarguscamerasrc
  Description              nVidia ARGUS Source Component
  Filename                 /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvarguscamerasrc.so
  Version                  1.0.0
  License                  Proprietary
  Source module            nvarguscamerasrc
  Binary package           NvARGUSCameraSrc
  Origin URL               http://nvidia.com/

  nvarguscamerasrc: NvArgusCameraSrc

  1 features:
  +-- 1 elements

    root@1ec26beb1dde:/# gst-launch-1.0 -e nvcamerasrc sensor-id=0 ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=40/1" ! nvv4l2h264enc ! h264parse ! mp4mux ! filesink location=rpi_v3_imx477_cam0.mp4
WARNING: erroneous pipeline: no element "nvcamerasrc"
root@1ec26beb1dde:/# gst-launch-1.0 -e nvarguscamerasrc sensor-id=0 ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=40/1" ! nvv4l2h264enc ! h264parse ! mp4mux ! filesink location=rpi_v3_imx477_cam0.mp4
nvbuf_utils: Could not get EGL display connection
nvbuf_utils: ERROR getting proc addr of eglCreateImageKHR
nvbuf_utils: ERROR getting proc addr of eglDestroyImageKHR
WARNING: erroneous pipeline: no element "nvv4l2h264enc"
root@1ec26beb1dde:/# ls
bin  boot  dev  dst  etc  home  lib  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
root@1ec26beb1dde:/# gst-launch-1.0 -e nvarguscamerasrc sensor-id=0 ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=40/1" ! omxh264enc ! h264parse ! mp4mux ! filesink location=rpi_v3_imx477_cam0.mp4
nvbuf_utils: Could not get EGL display connection
nvbuf_utils: ERROR getting proc addr of eglCreateImageKHR
nvbuf_utils: ERROR getting proc addr of eglDestroyImageKHR
WARNING: erroneous pipeline: no element "omxh264enc"

    further quick fix idea:

    copy entire gstreamer plugin folder

# funktioniert im host os
GST_DEBUG=4 gst-launch-1.0 -e nvarguscamerasrc sensor-id=0 ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=40/1" ! fakesink

# funktioniert nicht im container
GST_DEBUG=4 gst-launch-1.0 -e nvarguscamerasrc sensor-id=0 ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=40/1" ! fakesink

(Argus) Error NotSupported: EXT_platform_device extension missing (in src/eglutils/EGLUtils.cpp, function getDefaultDisplay(), line 44)
(Argus) Error InvalidState: Failed to load EGL library (in src/eglutils/EGLUtils.cpp, function exports(), line 213)
(Argus) Error InvalidState: Failed to get EGL API access function (in src/eglutils/EGLUtils.cpp, function exports(), line 224)
Caught SIGSEGV

# host os
root@1ec26beb1dde:/# gst-launch-1.0 -e nvarguscamerasrc sensor-id=0 ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=40/1" ! nvv4l2h264enc ! h264parse ! mp4mux ! filesink location=rpi_v3_imx477_cam0.mp4
-> WARNING: erroneous pipeline: no element "h264parse"
"The h264parse element is part of the gstreamer1.0-plugins-bad package, which seems to not be installed by default."

# fixed by adding missing binding
lib, /usr/lib/libgstnvegl-1.0.so.0
lib, /usr/lib/aarch64-linux-gnu/tegra-egl/libEGL_nvidia.so.0
lib, /usr/lib/libnvdecode2eglimage.so
lib, /usr/lib/libnveglstream_camconsumer.so
lib, /usr/lib/libnveglstreamproducer.so
lib, /usr/lib/libnvidia-eglcore.so.32.6.1
lib, /usr/lib/libGLESv1_CM_nvidia.so.1
lib, /usr/lib/libGLESv2.so.2.1.0

    # try jpg
gst-launch-1.0 -e nvarguscamerasrc num-buffers=20 sensor-id=0 ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=40/1" ! nvjpegenc ! multifilesink location=%03d_rpi_v3_imx477_cam0.jpeg

    Next step issue logs:

    # fake sink works but no other sinks 

gst-launch-1.0 -e nvarguscamerasrc sensor-id=0 ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=40/1" ! nvv4l2h264enc ! h264parse ! mp4mux ! filesink location=rpi_v3_imx477_cam0.mp4

# fixed by copying libs from official image
mv /usr/lib/gstreamer-1.0/libgstnvarguscamerasrc.so /usr/lib/gstreamer-1.0/libgstnvarguscamerasrc.so.bckp
cp libgstnvarguscamerasrc.so /usr/lib/gstreamer-1.0/libgstnvarguscamerasrc.so

mv /usr/lib/gstreamer-1.0/libgstnvv4l2camerasrc.so /usr/lib/gstreamer-1.0/libgstnvv4l2camerasrc.so.bckp
cp libgstnvv4l2camerasrc.so /usr/lib/gstreamer-1.0/libgstnvv4l2camerasrc.so

mv /usr/lib/libv4l2.so.0 /usr/lib/libv4l2.so.0.bckp
mv /usr/lib/libv4l2.so.0.0.0 /usr/lib/libv4l2.so.0.0.0.bckp
cp libv4l2.so.0.0.0 /usr/lib/libv4l2.so.0.0.0

mv /usr/lib/libv4l2rds.so.0 /usr/lib/libv4l2rds.so.0.bckp
mv /usr/lib/libv4l2rds.so.0.0.0 /usr/lib/libv4l2rds.so.0.0.0.bckp
cp libv4l2rds.so.0.0.0 /usr/lib/libv4l2rds.so.0.0.0

ln -s /usr/lib/libv4l2.so.0.0.0 /usr/lib/libv4l2.so.0
ln -s /usr/lib/libv4l2rds.so.0.0.0 /usr/lib/libv4l2rds.so.0

mv /usr/lib/gstreamer-1.0/libgstnvvideo4linux2.so /usr/lib/gstreamer-1.0/libgstnvvideo4linux2.so.bckp
cp libgstnvvideo4linux2.so /usr/lib/gstreamer-1.0/libgstnvvideo4linux2.so
3 Likes