Coral PCIe / m.2 on jetson nano

Is there a way to run the coral PCIe on a jetson nano board (n30b-nano).
The deb package does not install correctly due to kernel header dependencies/versions.

Can you please share some logs?

I have tired building gasket-dkms against the jn30b-nano 2.56.0+rev1.prod kernel sources in my container. I do not know if this is the best way to do it.

I get the following errors:

Make: Entering directory ‘/usr/src/app/kernel_modules_headers’
LD /var/lib/dkms/gasket/1.0/build/built-in.o
CC [M] /var/lib/dkms/gasket/1.0/build/gasket_core.o
CC [M] /var/lib/dkms/gasket/1.0/build/gasket_ioctl.o
CC [M] /var/lib/dkms/gasket/1.0/build/gasket_interrupt.o
CC [M] /var/lib/dkms/gasket/1.0/build/gasket_page_table.o
CC [M] /var/lib/dkms/gasket/1.0/build/gasket_sysfs.o
CC [M] /var/lib/dkms/gasket/1.0/build/apex_driver.o
In file included from ./arch/arm64/include/asm/dma-mapping.h:30:0,
from ./include/linux/dma-mapping.h:210,
from ./arch/arm64/include/asm/pci.h:7,
from ./include/linux/pci.h:1670,
from /var/lib/dkms/gasket/1.0/build/gasket_interrupt.h:12,
from /var/lib/dkms/gasket/1.0/build/gasket_interrupt.c:4:
./arch/arm64/include/asm/xen/hypervisor.h:1:10: fatal error: …/…/arm/include/asm/xen/hypervisor.h: No such file or directory
#include <…/…/arm/include/asm/xen/hypervisor.h>
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from ./arch/arm64/include/asm/dma-mapping.h:30:0,
from ./include/linux/dma-mapping.h:210,
from ./arch/arm64/include/asm/pci.h:7,
from ./include/linux/pci.h:1670,
from /var/lib/dkms/gasket/1.0/build/gasket_core.h:16,
from /var/lib/dkms/gasket/1.0/build/gasket_ioctl.h:6,
from /var/lib/dkms/gasket/1.0/build/gasket_ioctl.c:4:
./arch/arm64/include/asm/xen/hypervisor.h:1:10: fatal error: …/…/arm/include/asm/xen/hypervisor.h: No such file or directory
#include <…/…/arm/include/asm/xen/hypervisor.h>
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
compilation terminated.
In file included from ./arch/arm64/include/asm/dma-mapping.h:30:0,
from ./include/linux/dma-mapping.h:210,
from ./arch/arm64/include/asm/pci.h:7,
from ./include/linux/pci.h:1670,
from /var/lib/dkms/gasket/1.0/build/gasket_page_table.h:15,
from /var/lib/dkms/gasket/1.0/build/gasket_page_table.c:42:
./arch/arm64/include/asm/xen/hypervisor.h:1:10: fatal error: …/…/arm/include/asm/xen/hypervisor.h: No such file or directory
#include <…/…/arm/include/asm/xen/hypervisor.h>
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
In file included from ./arch/arm64/include/asm/dma-mapping.h:30:0,
from ./include/linux/dma-mapping.h:210,
from ./arch/arm64/include/asm/pci.h:7,
from ./include/linux/pci.h:1670,
from /var/lib/dkms/gasket/1.0/build/gasket_core.h:16,
from /var/lib/dkms/gasket/1.0/build/gasket_core.c:12:
./arch/arm64/include/asm/xen/hypervisor.h:1:10: fatal error: …/…/arm/include/asm/xen/hypervisor.h: No such file or directory
#include <…/…/arm/include/asm/xen/hypervisor.h>
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
scripts/Makefile.build:335: recipe for target ‘/var/lib/dkms/gasket/1.0/build/gasket_ioctl.o’ failed
make[1]: *** [/var/lib/dkms/gasket/1.0/build/gasket_ioctl.o] Error 1
make[1]: *** Waiting for unfinished jobs…
In file included from ./arch/arm64/include/asm/dma-mapping.h:30:0,
from ./include/linux/dma-mapping.h:210,
from ./arch/arm64/include/asm/pci.h:7,
from ./include/linux/pci.h:1670,
from /var/lib/dkms/gasket/1.0/build/apex_driver.c:17:
./arch/arm64/include/asm/xen/hypervisor.h:1:10: fatal error: …/…/arm/include/asm/xen/hypervisor.h: No such file or directory
#include <…/…/arm/include/asm/xen/hypervisor.h>
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
In file included from ./arch/arm64/include/asm/dma-mapping.h:30:0,
from ./include/linux/dma-mapping.h:210,
from ./arch/arm64/include/asm/pci.h:7,
from ./include/linux/pci.h:1670,
from /var/lib/dkms/gasket/1.0/build/gasket_core.h:16,
from /var/lib/dkms/gasket/1.0/build/gasket_sysfs.h:21,
from /var/lib/dkms/gasket/1.0/build/gasket_sysfs.c:3:
./arch/arm64/include/asm/xen/hypervisor.h:1:10: fatal error: …/…/arm/include/asm/xen/hypervisor.h: No such file or directory
#include <…/…/arm/include/asm/xen/hypervisor.h>
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
scripts/Makefile.build:335: recipe for target ‘/var/lib/dkms/gasket/1.0/build/gasket_interrupt.o’ failed
make[1]: *** [/var/lib/dkms/gasket/1.0/build/gasket_interrupt.o] Error 1
scripts/Makefile.build:335: recipe for target ‘/var/lib/dkms/gasket/1.0/build/gasket_page_table.o’ failed
make[1]: *** [/var/lib/dkms/gasket/1.0/build/gasket_page_table.o] Error 1
scripts/Makefile.build:335: recipe for target ‘/var/lib/dkms/gasket/1.0/build/gasket_core.o’ failed
make[1]: *** [/var/lib/dkms/gasket/1.0/build/gasket_core.o] Error 1
scripts/Makefile.build:335: recipe for target ‘/var/lib/dkms/gasket/1.0/build/apex_driver.o’ failed
make[1]: *** [/var/lib/dkms/gasket/1.0/build/apex_driver.o] Error 1
scripts/Makefile.build:335: recipe for target ‘/var/lib/dkms/gasket/1.0/build/gasket_sysfs.o’ failed
make[1]: *** [/var/lib/dkms/gasket/1.0/build/gasket_sysfs.o] Error 1
Makefile:1644: recipe for target ‘module/var/lib/dkms/gasket/1.0/build’ failed
make: *** [module/var/lib/dkms/gasket/1.0/build] Error 2
make: Leaving directory ‘/usr/src/app/kernel_modules_headers’

Hi,

Please try with v2.58.3+rev6 from https://dashboard.balena-staging.com, it has the gasket and apex modules included in the hostOS. You may need to use dma_bit_mask=32 parameter when loading the module if you encounter HIB error when running inferences.

Thank you, sadly the release is not available jet for the n30b-nano boards

I see, so you need those drivers for the jn30b-nano and not for the jetson-nano.

I added them now for jn30b , please check again for 2.58.3+rev6 in the staging dashboard.

I also have a Coral PCIe / m.2 but on a generic x86_64 device. It is running balenaOS 2.58.6+rev1 and the coral accelerator device is listed when calling lspci and shows up as /dev/apex_0. However when trying to use the device I get the following error:

ValueError: Failed to load delegate from libedgetpu.so.1

I experienced a similar issue before when I was trying to use the coral usb stick and was able to solve this by enabling UDEV devices by running the entry.sh script. However switching to the PCIe / m.2 device didn’t work.

@acostach do you have any idea how this can be solved?

Hi Toby,

The x86_64 image does not include the apex driver for the coral, did you build it for this machine and load it from your app container?

Hello and thanks for the quick reply. I did not build it for this device. How would I go about doing that?

The driver sources are available in the Coral kernel sources: https://coral.googlesource.com/linux-imx/+/refs/heads/release-chef/drivers/staging/gasket/

An example for building a module out of tree in container is available here: https://github.com/balena-os/kernel-module-build

An example Dockerfile for installing debs is available here - please note that it’s meant to be used with the coral-dev board, so you’ll need to adapt it to use a genericx86-64 base image, set arch=amd64 for debs, etc.

1 Like

After following your instructions I tried to install the kernel module. However when I ran insmod gasket.ko for the first time it turned out it was already installed and running. The module also has the same version as the one I just build (1.1.2). So the suggested solution probably doesn’t work. Do you have another idea how to solve this?

Hi Toby,

Not sure I understand the new problem, was the module inserted by the script in the container sample app already - https://github.com/balena-os/kernel-module-build/blob/93da0fefec03692c0b5a819355d599b449b9c5c7/run.sh#L10 - or does the one that was already present come from somewhere else? You can try to rmmod and insmod both modules, gasket and apex, as they depend on one another.

Dear @acostach,
I did not insert the module by script as it was already present in the container. I recall reloading both modules and that this did not fix the issue. I’ll see if I can try again this week.

Just checked, both modules are running. I am not able to use ‘rmmod’ on either module as both are in use.

Could it possibly have something to do with conflicting kernel modules? See https://coral.ai/docs/m2/get-started/#workaround-to-disable-apex-and-gasket

Hi Toby,

Yes, the genericx86-64 has those 2 modules in the hostOS already and they are provided by the upstream kernel. And yes, it’s possible that they are the ones that cause the error. You can try blacklist them in the hostOS and try load the ones you built in container, from the coral repositories.