Build kernel module out-of-tree for Jetson

Product support balenaOS
,
1

Hi,

I’m trying to build a camera driver kernel module out-of-tree for the Jetson, I’m using the Nano for now to be exactly but other variants with the same kernel will be necessary. I used this example.

The Makefile:

SRC_TREE=/usr/src/linux-headers-4.9.140-tegra-ubuntu18.04_aarch64
KERNEL_PATH=/lib/modules/4.9.140-l4t-r32.4.2

ccflags-y += -I$(SRC_TREE)/drivers/media/platform/tegra/
obj-m	+= ar0521.o

ar0521:
	make -C $(KERNEL_PATH) M=$(PWD) modules
clean:
	make -C $(KERNEL_PATH) M=$(PWD) clean

To begin is the KERNEL_PATH right? Because I don’t think there is a kernel binary at that location, but most guides say there should be one with the name ‘build’. Where is the kernel binary located on Balena firmware?

Then the SRC_TREE is not the real one received from Balena, I additionally installed these headers from NVIDIA because the one from Balena were missing some headers. Is this a bug or am I looking at the wrong location?

Kind regards,
Clint

3

Hey @clint, that example Makefile looks fine to me but when the actual build step is performed, it’s using the Makefile included with the kernel headers (not the one packaged with the example module). The kernel path is expected to be empty in these cases as we don’t ship the kernel source with the image.

The important logic is actually in build.sh where it downloads the kernel headers from our S3 storage and runs make -C "$tmp_path" modules_prepare first.

Another non-official example you could look at is where I build wireguard out-of-tree. It doesn’t have as many features but it may be more clear where the important pieces are.

Also importantly, if our kernel headers package is missing something we would love to know so we can fix them!

5

Hi @clint!

Just checking in on whether you’ve been able to try out Kyle’s suggestions, and successfully build the module?

Kind regards
Alida

7

Hi @AlidaOdendaal and @klutchell ,

I managed to build the kernel module, the next step is to test the module. For this to work, I have to replace the camera definition in the device tree, compile it to a device tree binary and get the device tree binary on the Balena device.

I lost track in all the docs to manage this with the right kernel sources, any suggestions?

I read some docs about replacing device tree overlay on the fly for Raspberry Pi but is this also available with Jetson Nano?

Any guidance would be appreciated.

Kind regards,
Clint

8

HI @Clint, on the Jetson Nano you can test this by placing your custom compiled device tree in /mnt/sysroot/active/current/boot/ and then, in the dashboard specify the device tree name (i.e mycustom.dtb). After you validate your device-tree changes you can open a PR to include it with the Jetson Nano image in the device repo. Here’s an example PR that does this: jetson-dtbs: Add imx477 dtb for CTI Photon Nano · balena-os/balena-jetson@c0cd185 · GitHub

10

Hey there, did you have any success loading your custom device tree following my colleague’s instructions above?

12

Thanks, I’m on it. Maybe a little off topic, is this feature also available for the Xavier NX?

13

Hi @acostach,
I didn’t manage to update the device tree. I think there’s something wrong.
Is use:

  • balenaOS 2.67.3+rev2 (supervisor 12.3.0)
  • Auvidea JN30B board

The board supports the jetson-nano-emmc version of BalenaOS, just some devices are not working. I tried the feature using jn30b version of BalenaOS and using the jetson-nano-emmc version but the kernel doesn’t boot.

What is the return value in U-Boot of the last line in following fragment? (50)

info: found default FDT entry, look for custom_fdt_file environment variable
info: fdt file now is /boot/tegra210-p3448-0002-p3449-0000-b00-jn30b.dtb - 50

The last thing I tried was flashing the device with jetson-nano-emmc BalenaOS version and just switch using the dashboard to tegra210-p3448-0002-p3449-0000-b00-jn30b.dtb which is already at the device, without success.

Maybe good to mention, we are modifying the config.json in our build process. Could this have an impact on this feature?

14

Hi @Clint, this feature is not available for the Xavier or the NX because they don’t use u-boot.
The value printed is not a return value, it’s a log of the full length of the device tree path, will get it removed if that adds any confusion.

Does the kernel fail to boot with the original JN30B device tree, i.e if you flash an image from the dashboard? Or does it fail to boot when switching to your custom modified device tree?

16

Hi @acostach ,

It fails when switching to another device tree. To exclude the dtb modification step, I tried switching between already installed dtb’s on the boot partition. If I can help with some logs, please let me know.

The length of the device tree path is not confusing, I just wanted to know what it means.

The config.json I was talking about in the previous post has nothing to do with it. I tested with and without a modified config.json file.

In the mean time I already managed to test a modified dtb on a native L4T install, so the dts should be ok. But this got me thinking, am I right that the dtb needs to be compiled using the L4T public kernel sources and not the Balena ones?

18

@Clint the JN30B Nano is a community supported device type, we don’t have this hardware so we don’t run testing on it. The v2.67.3+rev2 runs the 32.4.4 kernel from yocto meta-tegra, while the pre-compiled device tree in the image is for 32.4.2 as updated by the community maintainer for the board in this commit, and is the last 32.4.x version released by Auvidea.

Therefore the dts you need to modify is the one Auvidea provides for the JN30B, not the generic L4T kernel dts nor the one used by Balena.

19

Hi,

Thanks for the clarification, but this is clear to me.

The question is, can I just compile this using NVIDIA L4T kernel sources (Auvidea doesn’t change anything in the sources except dts files). Or does meta-tegra does changes in the NVIDIA kernel sources and do I have to use the meta-tegra kernel sources?

I think the last option. But then the next question, can you maybe provide me some steps to get the meta-tegra kernel sources from the build system?

20

meta-tegra does patch the upstream kernel sources, as you can see for instance on the 32.4.4 branch, but dts changes are probably minimal, if any. You’ll need to check.

Kernel branch and revision are visible in the meta-tegra kernel recipe, on top of which are applied the Balena changes.

If you want to do a local yocto Balena build you’ll need clone the balena-jetson repository with git clone --recursive and start the build with balena-yocto-scripts/build/barys -b build_nano -m jetson-nano

After the build is completed you can see the kernel sources in: <yocto_build_dir>/tmp/work-shared/<machine>/kernel-source.

22

@klutchell @acostach Re: this comment earlier:

Also importantly, if our kernel headers package is missing something we would love to know so we can fix them!

I am trying to adapt the out-of-tree Wireguard example to Jetson Nano myself. I get as far as this, and I believe the reason is some missing ARM (32 bit?) Xen hypervisor-related headers.

root@604da545312b:/build# make -C kernel_modules_headers M=$(pwd)/wireguard-linux-compat/src -j$(nproc)
make: Entering directory '/build/kernel_modules_headers'
  CC [M]  /build/wireguard-linux-compat/src/main.o
  CC [M]  /build/wireguard-linux-compat/src/noise.o
  CC [M]  /build/wireguard-linux-compat/src/device.o
  CC [M]  /build/wireguard-linux-compat/src/peer.o
In file included from ./arch/arm64/include/asm/dma-mapping.h:30,
                 from ./include/linux/dma-mapping.h:210,
                 from ./include/linux/skbuff.h:34,
                 from /build/wireguard-linux-compat/src/compat/compat.h:120,
                 from <command-line>:
./arch/arm64/include/asm/xen/hypervisor.h:1:10: fatal error: ../../arm/include/asm/xen/hypervisor.h: No such file or directory
 #include <../../arm/include/asm/xen/hypervisor.h>
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from ./arch/arm64/include/asm/dma-mapping.h:30,
                 from ./include/linux/dma-mapping.h:210,
                 from ./include/linux/skbuff.h:34,
                 from /build/wireguard-linux-compat/src/compat/compat.h:120,
                 from <command-line>:
./arch/arm64/include/asm/xen/hypervisor.h:1:10: fatal error: ../../arm/include/asm/xen/hypervisor.h: No such file or directory
 #include <../../arm/include/asm/xen/hypervisor.h>
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from ./arch/arm64/include/asm/dma-mapping.h:30,
                 from ./include/linux/dma-mapping.h:210,
                 from ./include/linux/skbuff.h:34,
                 from /build/wireguard-linux-compat/src/compat/compat.h:120,
                 from <command-line>:
./arch/arm64/include/asm/xen/hypervisor.h:1:10: fatal error: ../../arm/include/asm/xen/hypervisor.h: No such file or directory
 #include <../../arm/include/asm/xen/hypervisor.h>
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
In file included from ./arch/arm64/include/asm/dma-mapping.h:30,
                 from ./include/linux/dma-mapping.h:210,
                 from ./include/linux/skbuff.h:34,
                 from /build/wireguard-linux-compat/src/compat/compat.h:120,
                 from <command-line>:
./arch/arm64/include/asm/xen/hypervisor.h:1:10: fatal error: ../../arm/include/asm/xen/hypervisor.h: No such file or directory
 #include <../../arm/include/asm/xen/hypervisor.h>
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
compilation terminated.
compilation terminated.
make[1]: *** [scripts/Makefile.build:335: /build/wireguard-linux-compat/src/noise.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: *** [scripts/Makefile.build:335: /build/wireguard-linux-compat/src/main.o] Error 1
make[1]: *** [scripts/Makefile.build:335: /build/wireguard-linux-compat/src/device.o] Error 1
make[1]: *** [scripts/Makefile.build:335: /build/wireguard-linux-compat/src/peer.o] Error 1
make: *** [Makefile:1656: _module_/build/wireguard-linux-compat/src] Error 2
make: Leaving directory '/build/kernel_modules_headers'

When I look at that folder, all I see is the opcodes header:

root@604da545312b:/build# ls kernel_modules_headers/arch/arm/include/asm
opcodes.h

I am on v2.85.2+rev4

Is this something that can/should be included in the published headers batch, or am I trying to do something that is not supported?

Update: for the record, this incredibly hacky solution worked :slight_smile: – fortunately only that hypervisor header is needed for Wireguard’s module to compile.

# Download missing header(s)
RUN mkdir -p kernel_modules_headers/arch/arm/include/asm/xen && \
  curl -SsL -o kernel_modules_headers/arch/arm/include/asm/xen/hypervisor.h \
    https://raw.githubusercontent.com/OE4T/linux-tegra-4.9/oe4t-patches-l4t-r32.6/arch/arm/include/asm/xen/hypervisor.h
23

Hey @jasonanderson, what version of balenaOS headers are you downloading? Can you try with kernel_source.tar.gz instead of kernel_modules_headers.tar.gz?

It should look kind of like this:

25

I had the same problem with the missing hypervisor.h header. If I download kernel_source.tar.gz instead of kernel_modules_headers.tar.gz than it works. Thanks @klutchell

1 Like
26

Thanks for confirming @petekalvin!

28

Hi,

Good the issue with the hypervisor header got resolved. But there are sources missing from the kernel_source archive. Simple example is the imx219.c file (camera driver).

If we look at the structure of NVIDIA’s sources, we have 3 folders and I think the first folder (kernel-4.9) are the provided sources. So the nvgpu and nvidia sources are missing. However the folders to include the sources are at place, their are just no source files inside.
Screenshot from 2021-12-27 15-52-02

Kind regards,
Clint

29

Hey @klutchell , I’m working with @jasonanderson on this.

Using the build.sh method from GitHub - balena-os/kernel-module-build: Example project for building an OOT kernel module in balena
fails when using the jetson-nano and BALENA_OS_VERSION=‘2.88.4+rev1.dev’, currently the latest available.

I can confirm that downloading kernel_source.tar.gz works for building wireguard out of tree, but it does not appear to be a full source?

I also need to build the ip tunnel modules, but can’t seem to build them even with the kernel_source tarball. Enabling it via make menuconfig, then running make M=net/ipv4 returns the following error:

root@0e89750ad4fe:/usr/src/app/build# make M=net/ipv4
make[1]: *** No rule to make target 'net/ipv4/route.o', needed by 'net/ipv4/built-in.o'.  Stop.
Makefile:1655: recipe for target '_module_net/ipv4' failed
make: *** [_module_net/ipv4] Error 2

Is it possible that the tarball is not a full kernel source? Doing a plain make fails as well, even with no .config changes.

30

Hey @msherman, welcome to the forums!

Using the build.sh method from GitHub - balena-os/kernel-module-build: Example project for building an OOT kernel module in balena
fails when using the jetson-nano and BALENA_OS_VERSION=‘2.88.4+rev1.dev’, currently the latest available.

Can you try with the gcc-10 branch of the kernel-module-build project? If it works for you it might be worth merging the change.
https://github.com/balena-os/kernel-module-build/tree/gcc-10

but it does not appear to be a full source?

No, it’s not the full source. We don’t currently ship the entire kernel source but we can point you to the upstream sources if it helps.

However, in this case I would expect the ip tunnel modules to already be included in balenaOS. I’ve tried a number of Raspberry Pi models, and Generic x86-64 and the modules were already present and just needed to be loaded.
Here’s an example from my wireguard repo: https://github.com/klutchell/balena-wireguard/blob/main/run.sh#L24-L49

Can you confirm your device type that you’ve tried, and the results of lsmod on the host balenaOS? If we are missing some modules for your device we can look into adding them.