# Build kernel module out-of-tree for Jetson

**URL:** <https://forums.balena.io/t/build-kernel-module-out-of-tree-for-jetson/295852>\
**Category:** balenaOS\
**Tags:** kernel-module, jetson\
**Created:** [April 17, 2021, 6:40pm UTC](https://forums.balena.io/t/build-kernel-module-out-of-tree-for-jetson/295852 "2021-04-17T18:40:28Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Clint](https://avatars.discourse-cdn.com/v4/letter/c/da6949/32.png) [@Clint](https://forums.balena.io/u/Clint)\
**Post date:** [April 17, 2021, 6:40pm UTC](https://forums.balena.io/t/build-kernel-module-out-of-tree-for-jetson/295852/1 "2021-04-17T18:40:28Z")

</div>

Hi,

I’m trying to build a camera driver kernel module out-of-tree for the Jetson, I’m using the Nano for now to be exactly but other variants with the same kernel will be necessary. I used [this example](https://github.com/balena-os/kernel-module-build).

The Makefile:

```auto
SRC_TREE=/usr/src/linux-headers-4.9.140-tegra-ubuntu18.04_aarch64
KERNEL_PATH=/lib/modules/4.9.140-l4t-r32.4.2

ccflags-y += -I$(SRC_TREE)/drivers/media/platform/tegra/
obj-m	+= ar0521.o

ar0521:
	make -C $(KERNEL_PATH) M=$(PWD) modules
clean:
	make -C $(KERNEL_PATH) M=$(PWD) clean

```

To begin is the KERNEL\_PATH right? Because I don’t think there is a kernel binary at that location, but most guides say there should be one with the name ‘build’. Where is the kernel binary located on Balena firmware?

Then the SRC\_TREE is not the real one received from Balena, I additionally installed these headers from NVIDIA because the one from Balena were missing some headers. Is this a bug or am I looking at the wrong location?

Kind regards,  
Clint

---

<div class="post-metadata">

**Author:** ![klutchell](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/klutchell/32/861_2.png) [@klutchell](https://forums.balena.io/u/klutchell)\
**Post date:** [April 20, 2021, 6:55pm UTC](https://forums.balena.io/t/build-kernel-module-out-of-tree-for-jetson/295852/3 "2021-04-20T18:55:27Z")

</div>

Hey @clint, that example Makefile looks fine to me but when the actual build step is performed, it’s using the Makefile included with the kernel headers (not the one packaged with the example module). The kernel path is expected to be empty in these cases as we don’t ship the kernel source with the image.

The important logic is actually in `build.sh` where it downloads the kernel headers from our S3 storage and runs `make -C "$tmp_path" modules_prepare` first.

Another non-official example you could look at is where I build wireguard out-of-tree. It doesn’t have as many features but it may be more clear where the important pieces are.

> <https://github.com/klutchell/balena-wireguard/blob/main/Dockerfile.raspberrypi3>

Also importantly, if our kernel headers package is missing something we would love to know so we can fix them!

---

<div class="post-metadata">

**Author:** ![AlidaOdendaal](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/alidaodendaal/32/1641_2.png) [@AlidaOdendaal](https://forums.balena.io/u/AlidaOdendaal)\
**Post date:** [April 30, 2021, 8:13am UTC](https://forums.balena.io/t/build-kernel-module-out-of-tree-for-jetson/295852/5 "2021-04-30T08:13:22Z")

</div>

Hi @clint!

Just checking in on whether you’ve been able to try out Kyle’s suggestions, and successfully build the module?

Kind regards  
Alida

---

<div class="post-metadata">

**Author:** ![Clint](https://avatars.discourse-cdn.com/v4/letter/c/da6949/32.png) [@Clint](https://forums.balena.io/u/Clint)\
**Post date:** [May 4, 2021, 10:28am UTC](https://forums.balena.io/t/build-kernel-module-out-of-tree-for-jetson/295852/7 "2021-05-04T10:28:06Z")

</div>

Hi @AlidaOdendaal and @klutchell ,

I managed to build the kernel module, the next step is to test the module. For this to work, I have to replace the camera definition in the device tree, compile it to a device tree binary and get the device tree binary on the Balena device.

I lost track in all the docs to manage this with the right kernel sources, any suggestions?

I read some docs about replacing device tree overlay on the fly for Raspberry Pi but is this also available with Jetson Nano?

Any guidance would be appreciated.

Kind regards,  
Clint

---

<div class="post-metadata">

**Author:** ![acostach](https://avatars.discourse-cdn.com/v4/letter/a/bcef8e/32.png) [@acostach](https://forums.balena.io/u/acostach)\
**Post date:** [May 4, 2021, 12:29pm UTC](https://forums.balena.io/t/build-kernel-module-out-of-tree-for-jetson/295852/8 "2021-05-04T12:29:00Z")

</div>

HI @Clint, on the Jetson Nano you can test this by placing your custom compiled device tree in /mnt/sysroot/active/current/boot/ and then, in the dashboard specify the device tree name (i.e mycustom.dtb). After you validate your device-tree changes you can open a PR to include it with the Jetson Nano image in the device repo. Here’s an example PR that does this: [jetson-dtbs: Add imx477 dtb for CTI Photon Nano · balena-os/balena-jetson@c0cd185 · GitHub](https://github.com/balena-os/balena-jetson/commit/c0cd185a89e99ddd6e9bd54a76ce5c7789cb8c5d)

---

<div class="post-metadata">

**Author:** ![klutchell](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/klutchell/32/861_2.png) [@klutchell](https://forums.balena.io/u/klutchell)\
**Post date:** [May 11, 2021, 4:50pm UTC](https://forums.balena.io/t/build-kernel-module-out-of-tree-for-jetson/295852/10 "2021-05-11T16:50:15Z")

</div>

Hey there, did you have any success loading your custom device tree following my colleague’s instructions above?

---

<div class="post-metadata">

**Author:** ![Clint](https://avatars.discourse-cdn.com/v4/letter/c/da6949/32.png) [@Clint](https://forums.balena.io/u/Clint)\
**Post date:** [May 12, 2021, 4:18pm UTC](https://forums.balena.io/t/build-kernel-module-out-of-tree-for-jetson/295852/12 "2021-05-12T16:18:53Z")

</div>

Thanks, I’m on it. Maybe a little off topic, is this feature also available for the Xavier NX?

---

<div class="post-metadata">

**Author:** ![Clint](https://avatars.discourse-cdn.com/v4/letter/c/da6949/32.png) [@Clint](https://forums.balena.io/u/Clint)\
**Post date:** [May 13, 2021, 1:45pm UTC](https://forums.balena.io/t/build-kernel-module-out-of-tree-for-jetson/295852/13 "2021-05-13T13:45:33Z")

</div>

Hi @acostach,  
I didn’t manage to update the device tree. I think there’s something wrong.  
Is use:

- balenaOS 2.67.3+rev2 (supervisor 12.3.0)
- Auvidea JN30B board

The board supports the `jetson-nano-emmc` version of BalenaOS, just some devices are not working. I tried the feature using `jn30b` version of BalenaOS and using the `jetson-nano-emmc` version but the kernel doesn’t boot.

What is the return value in U-Boot of the last line in following fragment? `(50)`

```auto
info: found default FDT entry, look for custom_fdt_file environment variable
info: fdt file now is /boot/tegra210-p3448-0002-p3449-0000-b00-jn30b.dtb - 50

```

The last thing I tried was flashing the device with `jetson-nano-emmc` BalenaOS version and just switch using the dashboard to `tegra210-p3448-0002-p3449-0000-b00-jn30b.dtb` which is already at the device, without success.

Maybe good to mention, we are modifying the `config.json` in our build process. Could this have an impact on this feature?

---

<div class="post-metadata">

**Author:** ![acostach](https://avatars.discourse-cdn.com/v4/letter/a/bcef8e/32.png) [@acostach](https://forums.balena.io/u/acostach)\
**Post date:** [May 17, 2021, 6:45am UTC](https://forums.balena.io/t/build-kernel-module-out-of-tree-for-jetson/295852/14 "2021-05-17T06:45:51Z")

</div>

Hi @Clint, this feature is not available for the Xavier or the NX because they don’t use u-boot.  
The value printed is not a return value, it’s a log of the full length of the device tree path, will get it removed if that adds any confusion.

Does the kernel fail to boot with the original JN30B device tree, i.e if you flash an image from the dashboard? Or does it fail to boot when switching to your custom modified device tree?

---

<div class="post-metadata">

**Author:** ![Clint](https://avatars.discourse-cdn.com/v4/letter/c/da6949/32.png) [@Clint](https://forums.balena.io/u/Clint)\
**Post date:** [May 17, 2021, 2:23pm UTC](https://forums.balena.io/t/build-kernel-module-out-of-tree-for-jetson/295852/16 "2021-05-17T14:23:23Z")

</div>

Hi @acostach ,

It fails when switching to another device tree. To exclude the dtb modification step, I tried switching between already installed dtb’s on the boot partition. If I can help with some logs, please let me know.

The length of the device tree path is not confusing, I just wanted to know what it means.

The config.json I was talking about in the previous post has nothing to do with it. I tested with and without a modified config.json file.

In the mean time I already managed to test a modified dtb on a native L4T install, so the dts should be ok. But this got me thinking, am I right that the dtb needs to be compiled using the L4T public kernel sources and not the Balena ones?

---

<div class="post-metadata">

**Author:** ![acostach](https://avatars.discourse-cdn.com/v4/letter/a/bcef8e/32.png) [@acostach](https://forums.balena.io/u/acostach)\
**Post date:** [May 24, 2021, 10:45am UTC](https://forums.balena.io/t/build-kernel-module-out-of-tree-for-jetson/295852/18 "2021-05-24T10:45:07Z")

</div>

@Clint the JN30B Nano is a community supported device type, we don’t have this hardware so we don’t run testing on it. The v2.67.3+rev2 runs the 32.4.4 kernel from yocto meta-tegra, while the pre-compiled device tree in the image is for 32.4.2 as updated by the community maintainer for the board in this [commit](https://github.com/balena-os/balena-jetson/commit/d2f32072dc546b12e564463345e713dac28a024d), and is the last 32.4.x version released by Auvidea.

Therefore the dts you need to modify is the one Auvidea provides for the JN30B, not the generic L4T kernel dts nor the one used by Balena.

---

<div class="post-metadata">

**Author:** ![Clint](https://avatars.discourse-cdn.com/v4/letter/c/da6949/32.png) [@Clint](https://forums.balena.io/u/Clint)\
**Post date:** [May 24, 2021, 11:50am UTC](https://forums.balena.io/t/build-kernel-module-out-of-tree-for-jetson/295852/19 "2021-05-24T11:50:13Z")

</div>

Hi,

Thanks for the clarification, but this is clear to me.

The question is, can I just compile this using NVIDIA L4T kernel sources (Auvidea doesn’t change anything in the sources except dts files). Or does meta-tegra does changes in the NVIDIA kernel sources and do I have to use the meta-tegra kernel sources?

I think the last option. But then the next question, can you maybe provide me some steps to get the meta-tegra kernel sources from the build system?

---

<div class="post-metadata">

**Author:** ![acostach](https://avatars.discourse-cdn.com/v4/letter/a/bcef8e/32.png) [@acostach](https://forums.balena.io/u/acostach)\
**Post date:** [May 24, 2021, 3:02pm UTC](https://forums.balena.io/t/build-kernel-module-out-of-tree-for-jetson/295852/20 "2021-05-24T15:02:19Z")

</div>

meta-tegra does patch the upstream kernel sources, as you can see for instance on the [32.4.4](https://github.com/OE4T/linux-tegra-4.9/commits/oe4t-patches-l4t-r32.4) branch, but dts changes are probably minimal, if any. You’ll need to check.

Kernel branch and revision are visible in the meta-tegra [kernel recipe](https://github.com/OE4T/meta-tegra/blob/07f15cec44977dfba279062b36241e31b130ebfa/recipes-kernel/linux/linux-tegra_4.9.bb), on top of which are applied the Balena [changes](https://github.com/balena-os/balena-jetson/blob/master/layers/meta-balena-jetson/recipes-kernel/linux/linux-tegra_%25.bbappend).

If you want to do a local yocto Balena build you’ll need clone the balena-jetson repository with `git clone --recursive` and start the build with `balena-yocto-scripts/build/barys -b build_nano -m jetson-nano`

After the build is completed you can see the kernel sources in: ` <yocto_build_dir>/tmp/work-shared/<machine>/kernel-source`.

---

<div class="post-metadata">

**Author:** ![jasonanderson](https://avatars.discourse-cdn.com/v4/letter/j/c77e96/32.png) [@jasonanderson](https://forums.balena.io/u/jasonanderson)\
**Post date:** [November 30, 2021, 11:20pm UTC](https://forums.balena.io/t/build-kernel-module-out-of-tree-for-jetson/295852/22 "2021-11-30T23:20:54Z")

</div>

@klutchell @acostach Re: this comment earlier:

> Also importantly, if our kernel headers package is missing something we would love to know so we can fix them!

I am trying to adapt the out-of-tree Wireguard example to Jetson Nano myself. I get as far as this, and I believe the reason is some missing ARM (32 bit?) Xen hypervisor-related headers.

```shell
root@604da545312b:/build# make -C kernel_modules_headers M=$(pwd)/wireguard-linux-compat/src -j$(nproc)
make: Entering directory '/build/kernel_modules_headers'
  CC [M] /build/wireguard-linux-compat/src/main.o
  CC [M] /build/wireguard-linux-compat/src/noise.o
  CC [M] /build/wireguard-linux-compat/src/device.o
  CC [M] /build/wireguard-linux-compat/src/peer.o
In file included from ./arch/arm64/include/asm/dma-mapping.h:30,
                 from ./include/linux/dma-mapping.h:210,
                 from ./include/linux/skbuff.h:34,
                 from /build/wireguard-linux-compat/src/compat/compat.h:120,
                 from <command-line>:
./arch/arm64/include/asm/xen/hypervisor.h:1:10: fatal error: ../../arm/include/asm/xen/hypervisor.h: No such file or directory
 #include <../../arm/include/asm/xen/hypervisor.h>
          ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from ./arch/arm64/include/asm/dma-mapping.h:30,
                 from ./include/linux/dma-mapping.h:210,
                 from ./include/linux/skbuff.h:34,
                 from /build/wireguard-linux-compat/src/compat/compat.h:120,
                 from <command-line>:
./arch/arm64/include/asm/xen/hypervisor.h:1:10: fatal error: ../../arm/include/asm/xen/hypervisor.h: No such file or directory
 #include <../../arm/include/asm/xen/hypervisor.h>
          ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from ./arch/arm64/include/asm/dma-mapping.h:30,
                 from ./include/linux/dma-mapping.h:210,
                 from ./include/linux/skbuff.h:34,
                 from /build/wireguard-linux-compat/src/compat/compat.h:120,
                 from <command-line>:
./arch/arm64/include/asm/xen/hypervisor.h:1:10: fatal error: ../../arm/include/asm/xen/hypervisor.h: No such file or directory
 #include <../../arm/include/asm/xen/hypervisor.h>
          ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
In file included from ./arch/arm64/include/asm/dma-mapping.h:30,
                 from ./include/linux/dma-mapping.h:210,
                 from ./include/linux/skbuff.h:34,
                 from /build/wireguard-linux-compat/src/compat/compat.h:120,
                 from <command-line>:
./arch/arm64/include/asm/xen/hypervisor.h:1:10: fatal error: ../../arm/include/asm/xen/hypervisor.h: No such file or directory
 #include <../../arm/include/asm/xen/hypervisor.h>
          ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
compilation terminated.
compilation terminated.
make[1]: *** [scripts/Makefile.build:335: /build/wireguard-linux-compat/src/noise.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: *** [scripts/Makefile.build:335: /build/wireguard-linux-compat/src/main.o] Error 1
make[1]: *** [scripts/Makefile.build:335: /build/wireguard-linux-compat/src/device.o] Error 1
make[1]: *** [scripts/Makefile.build:335: /build/wireguard-linux-compat/src/peer.o] Error 1
make: *** [Makefile:1656: _module_/build/wireguard-linux-compat/src] Error 2
make: Leaving directory '/build/kernel_modules_headers'

```

When I look at that folder, all I see is the opcodes header:

```auto
root@604da545312b:/build# ls kernel_modules_headers/arch/arm/include/asm
opcodes.h

```

I am on v2.85.2+rev4

Is this something that can/should be included in the published headers batch, or am I trying to do something that is not supported?

* * *

**Update** : for the record, this incredibly hacky solution worked 🙂 – fortunately only that hypervisor header is needed for Wireguard’s module to compile.

```dockerfile
# Download missing header(s)
RUN mkdir -p kernel_modules_headers/arch/arm/include/asm/xen && \
  curl -SsL -o kernel_modules_headers/arch/arm/include/asm/xen/hypervisor.h \
    https://raw.githubusercontent.com/OE4T/linux-tegra-4.9/oe4t-patches-l4t-r32.6/arch/arm/include/asm/xen/hypervisor.h

```

---

<div class="post-metadata">

**Author:** ![klutchell](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/klutchell/32/861_2.png) [@klutchell](https://forums.balena.io/u/klutchell)\
**Post date:** [December 1, 2021, 8:43pm UTC](https://forums.balena.io/t/build-kernel-module-out-of-tree-for-jetson/295852/23 "2021-12-01T20:43:07Z")

</div>

Hey @jasonanderson, what version of balenaOS headers are you downloading? Can you try with `kernel_source.tar.gz` instead of `kernel_modules_headers.tar.gz`?

It should look kind of like this:

> <https://github.com/klutchell/balena-wireguard/blob/main/buildmod.sh#L32-L38>

---

<div class="post-metadata">

**Author:** ![PeterKAlvin](https://avatars.discourse-cdn.com/v4/letter/p/91b2a8/32.png) [@PeterKAlvin](https://forums.balena.io/u/PeterKAlvin)\
**Post date:** [December 19, 2021, 10:19pm UTC](https://forums.balena.io/t/build-kernel-module-out-of-tree-for-jetson/295852/25 "2021-12-19T22:19:22Z")

</div>

I had the same problem with the missing hypervisor.h header. If I download kernel\_source.tar.gz instead of kernel\_modules\_headers.tar.gz than it works. Thanks @klutchell

---

<div class="post-metadata">

**Author:** ![klutchell](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/klutchell/32/861_2.png) [@klutchell](https://forums.balena.io/u/klutchell)\
**Post date:** [December 20, 2021, 2:04pm UTC](https://forums.balena.io/t/build-kernel-module-out-of-tree-for-jetson/295852/26 "2021-12-20T14:04:17Z")

</div>

Thanks for confirming @petekalvin!

---

<div class="post-metadata">

**Author:** ![Clint](https://avatars.discourse-cdn.com/v4/letter/c/da6949/32.png) [@Clint](https://forums.balena.io/u/Clint)\
**Post date:** [December 27, 2021, 2:59pm UTC](https://forums.balena.io/t/build-kernel-module-out-of-tree-for-jetson/295852/28 "2021-12-27T14:59:35Z")

</div>

Hi,

Good the issue with the hypervisor header got resolved. But there are sources missing from the kernel\_source archive. Simple example is the `imx219.c` file (camera driver).

If we look at the structure of NVIDIA’s sources, we have 3 folders and I think the first folder (kernel-4.9) are the provided sources. So the nvgpu and nvidia sources are missing. However the folders to include the sources are at place, their are just no source files inside.  
 ![Screenshot from 2021-12-27 15-52-02](https://global.discourse-cdn.com/balena/original/2X/0/031ec0f26078cc73fbee8fbbc2ad802cfa029728.png)

Kind regards,  
Clint

---

<div class="post-metadata">

**Author:** ![msherman](https://avatars.discourse-cdn.com/v4/letter/m/bc79bd/32.png) [@msherman](https://forums.balena.io/u/msherman)\
**Post date:** [March 18, 2022, 1:43pm UTC](https://forums.balena.io/t/build-kernel-module-out-of-tree-for-jetson/295852/29 "2022-03-18T13:43:01Z")

</div>

Hey @klutchell , I’m working with @jasonanderson on this.

Using the build.sh method from [GitHub - balena-os/kernel-module-build: Example project for building an OOT kernel module in balena](https://github.com/balena-os/kernel-module-build)  
fails when using the jetson-nano and BALENA\_OS\_VERSION=‘2.88.4+rev1.dev’, currently the latest available.

I can confirm that downloading kernel\_source.tar.gz works for building wireguard out of tree, but it does not appear to be a full source?

I also need to build the ip tunnel modules, but can’t seem to build them even with the kernel\_source tarball. Enabling it via `make menuconfig`, then running `make M=net/ipv4` returns the following error:

```auto
root@0e89750ad4fe:/usr/src/app/build# make M=net/ipv4
make[1]: *** No rule to make target 'net/ipv4/route.o', needed by 'net/ipv4/built-in.o'. Stop.
Makefile:1655: recipe for target '_module_net/ipv4' failed
make: *** [_module_net/ipv4] Error 2

```

Is it possible that the tarball is not a full kernel source? Doing a plain `make` fails as well, even with no `.config` changes.

---

<div class="post-metadata">

**Author:** ![klutchell](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/klutchell/32/861_2.png) [@klutchell](https://forums.balena.io/u/klutchell)\
**Post date:** [March 18, 2022, 5:26pm UTC](https://forums.balena.io/t/build-kernel-module-out-of-tree-for-jetson/295852/30 "2022-03-18T17:26:36Z")

</div>

Hey @msherman, welcome to the forums!

> Using the build.sh method from [GitHub - balena-os/kernel-module-build: Example project for building an OOT kernel module in balena](https://github.com/balena-os/kernel-module-build)  
> fails when using the jetson-nano and BALENA\_OS\_VERSION=‘2.88.4+rev1.dev’, currently the latest available.

Can you try with the `gcc-10` branch of the kernel-module-build project? If it works for you it might be worth merging the change.  
[https://github.com/balena-os/kernel-module-build/tree/gcc-10](https://github.com/balena-os/kernel-module-build/tree/gcc-10)

> but it does not appear to be a full source?

No, it’s not the full source. We don’t currently ship the entire kernel source but we can point you to the upstream sources if it helps.

However, in this case I would expect the ip tunnel modules to already be included in balenaOS. I’ve tried a number of Raspberry Pi models, and Generic x86-64 and the modules were already present and just needed to be loaded.  
Here’s an example from my wireguard repo: [https://github.com/klutchell/balena-wireguard/blob/main/run.sh#L24-L49](https://github.com/klutchell/balena-wireguard/blob/main/run.sh#L24-L49)

Can you confirm your device type that you’ve tried, and the results of `lsmod` on the host balenaOS? If we are missing some modules for your device we can look into adding them.

[Next page](https://forums.balena.io/t/build-kernel-module-out-of-tree-for-jetson/295852.md?page=2)
