Installing CUDA on Jetson base images doesn't work as instructions say it should

Product support
,
1

I’m trying to get CUDA to work on various Jetson devices with Balena, following these instructions. The line This means that installing CUDA is as simple as issuing the following command or including it in your Dockerfile:
apt-get install -y nvidia-l4t-cuda sounds like a dream to anybody who’s worked with Jetsons before, implying a trivial Dockerfile like this should work:

FROM balenalib/jetson-xavier-nx-devkit-debian:bullseye-build
RUN apt-get update && apt-get install -y nvidia-l4t-cuda

but this fails with:

The following packages have unmet dependencies:
 nvidia-l4t-wayland : Depends: libffi6 but it is not installable

I think there’s a newer version of libffi installed, which might or might not be compatible if we care to try to fool it, but that seems sketchy. That same error comes up with a lot of other base images, including debian bookworm.

FROM balenalib/jetson-xavier-ubuntu:focal-build
RUN apt-get update && apt-get install -y nvidia-l4t-cuda

fails with:

The following packages have unmet dependencies:
 nvidia-l4t-cuda : Depends: nvidia-l4t-3d-core (= 32.7.4-20230608211515) but it is not going to be installed

Trying again…

FROM balenalib/jetson-xavier-ubuntu:kinetic-build
RUN apt-get update && apt-get install -y nvidia-l4t-cuda

fails to even get past apt-get update with:

E: The repository 'http://ports.ubuntu.com/ubuntu-ports kinetic Release' does not have a Release file.
E: The repository 'http://ports.ubuntu.com/ubuntu-ports kinetic-updates Release' does not have a Release file.
E: The repository 'http://ports.ubuntu.com/ubuntu-ports kinetic-backports Release' does not have a Release file.
E: The repository 'http://ports.ubuntu.com/ubuntu-ports kinetic-security Release' does not have a Release file.

Going back further:

FROM balenalib/jetson-xavier-debian:buster-build
RUN apt-get update && apt-get install -y nvidia-l4t-cuda

Fails because buster doesn’t understand .zst files:

Preparing to unpack .../16-nvidia-l4t-core_32.7.4-20230608211515_arm64.deb ...
Pre-installing... skip compatibility checking.
Unpacking nvidia-l4t-core (32.7.4-20230608211515) ...
dpkg-deb: error: archive '/tmp/apt-dpkg-install-8rp6Vm/16-nvidia-l4t-core_32.7.4-20230608211515_arm64.deb' uses unknown compression for member 'data.tar.zst', giving up
dpkg: error processing archive /tmp/apt-dpkg-install-8rp6Vm/16-nvidia-l4t-core_32.7.4-20230608211515_arm64.deb (--unpack):
 dpkg-deb --fsys-tarfile subprocess returned error exit status 2
Errors were encountered while processing:
 /tmp/apt-dpkg-install-8rp6Vm/16-nvidia-l4t-core_32.7.4-20230608211515_arm64.deb

This only one base image I’ve found that works is Ubuntu bionic (18.04), which is unfortunate because Ubuntu-bionic (18.04) is EOL, and even NVIDIA has finally added support for focal/20.04.

Any pointers on getting something vaguely-modern to build the way it’s supposed to?

1 Like
2

Hello,

Thanks for reporting this. I’ve dug into the issue and it’s quite the complicated one, but I’ll summarize my findings.

These packages are directly maintained and hosted by Nvidia in their own apt repositories. The “distribution” corresponds with the L4T version, as seen below.

root@dbf9f9de2622:/# cat /etc/apt/sources.list.d/nvidia.list 
deb https://repo.download.nvidia.com/jetson/common r32.7 main
deb https://repo.download.nvidia.com/jetson/t194 r32.7 main

When using L4T r32.7, the nvidia-l4t-wayland package depends on libffi6 and is not installable without it. Debian or Ubuntu images newer than Buster or Bionic respectively don’t package that dependency, and instead ship libffi7 or libffi8.

Updating the distribution in the apt sources to r35.4 fixes the dependency issue with libffi6, and properly supports Debian Bookworm and Ubuntu Focal. The problem is, L4T support for the Jetson Xavier boards in balenaOS is currently at r32.7. The L4T software packages includes the bootloader, kernel, and drivers, and is tightly integrated with the userspace libraries. Consequently, it’s not expected that a newer version of the userspace libraries will work with an older version of the kernel and drivers, which are built into balenaOS.

I’m awaiting confirmation on some of these details, but I’m told that the Jetson Orin series of boards are already on L4T 35.4, and should already work with newer base images.

Let us know if this helps, and we’ll update with more information as we investigate.

2 Likes
3

Thanks for digging into this - it certainly is complicated! Which is a key reason we’re hoping Balena can help simplify things for us. Any updates on your end?

I wouldn’t say this is “solved” though. I understand from your research that the Orin base images should work better, which would be a fine solution. But I tried this simple Dockerfile:

FROM balenalib/jetson-agx-orin-devkit-debian:bullseye-build
RUN apt-get update && apt-get install -y nvidia-l4t-cuda

and it failed with this error:

Unpacking nvidia-l4t-core (35.1.0-20220825113828) ...
dpkg-deb: error: archive '/tmp/apt-dpkg-install-rDH3G0/16-nvidia-l4t-core_35.1.0-20220825113828_arm64.deb' uses unknown compression for member 'data.tar.zst', giving up
dpkg: error processing archive /tmp/apt-dpkg-install-rDH3G0/16-nvidia-l4t-core_35.1.0-20220825113828_arm64.deb (--unpack):
 dpkg-deb --fsys-tarfile subprocess returned error exit status 2
Errors were encountered while processing:
 /tmp/apt-dpkg-install-rDH3G0/16-nvidia-l4t-core_35.1.0-20220825113828_arm64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
The command '/bin/sh -c apt-get update && apt-get install -y nvidia-l4t-cuda' returned a non-zero code: 100

Which is surprising because that looks like the error message from ancient OS’s that can’t read .zst files.

And upgrading to bookworm doesn’t help:

FROM balenalib/jetson-agx-orin-devkit-debian:bookworm-build
RUN apt-get update && apt-get install -y nvidia-l4t-cuda

fails with a new message, this time about libffi7:

The following packages have unmet dependencies:
 nvidia-l4t-wayland : Depends: libffi7 but it is not installable

Still hoping for some supported combination that “just works” the way the docs say.

5

Hello @cvdactyl

did you take a look at our example here? jetson-examples/jetson-orin-nx-xavier-nx-devkit/Dockerfile at master · balena-io-examples/jetson-examples · GitHub

It installs packages like: lbzip2 binutils xz-utils zstd and we would recommend starting from it if you are using an Orin NX or from this one if you are using the Orin Nano.

Let us know your next steps!

6

Thanks for the link - haven’t tried that yet but will!

1 Like
7

Hey @cvdactyl did you fix the issue?

Let’s stay connected