Is there a way to run the coral PCIe on a jetson nano board (n30b-nano).
The deb package does not install correctly due to kernel header dependencies/versions.
Can you please share some logs?
I have tired building gasket-dkms against the jn30b-nano 2.56.0+rev1.prod kernel sources in my container. I do not know if this is the best way to do it.
I get the following errors:
Make: Entering directory ‘/usr/src/app/kernel_modules_headers’
LD /var/lib/dkms/gasket/1.0/build/built-in.o
CC [M] /var/lib/dkms/gasket/1.0/build/gasket_core.o
CC [M] /var/lib/dkms/gasket/1.0/build/gasket_ioctl.o
CC [M] /var/lib/dkms/gasket/1.0/build/gasket_interrupt.o
CC [M] /var/lib/dkms/gasket/1.0/build/gasket_page_table.o
CC [M] /var/lib/dkms/gasket/1.0/build/gasket_sysfs.o
CC [M] /var/lib/dkms/gasket/1.0/build/apex_driver.o
In file included from ./arch/arm64/include/asm/dma-mapping.h:30:0,
from ./include/linux/dma-mapping.h:210,
from ./arch/arm64/include/asm/pci.h:7,
from ./include/linux/pci.h:1670,
from /var/lib/dkms/gasket/1.0/build/gasket_interrupt.h:12,
from /var/lib/dkms/gasket/1.0/build/gasket_interrupt.c:4:
./arch/arm64/include/asm/xen/hypervisor.h:1:10: fatal error: …/…/arm/include/asm/xen/hypervisor.h: No such file or directory
#include <…/…/arm/include/asm/xen/hypervisor.h>
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from ./arch/arm64/include/asm/dma-mapping.h:30:0,
from ./include/linux/dma-mapping.h:210,
from ./arch/arm64/include/asm/pci.h:7,
from ./include/linux/pci.h:1670,
from /var/lib/dkms/gasket/1.0/build/gasket_core.h:16,
from /var/lib/dkms/gasket/1.0/build/gasket_ioctl.h:6,
from /var/lib/dkms/gasket/1.0/build/gasket_ioctl.c:4:
./arch/arm64/include/asm/xen/hypervisor.h:1:10: fatal error: …/…/arm/include/asm/xen/hypervisor.h: No such file or directory
#include <…/…/arm/include/asm/xen/hypervisor.h>
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
compilation terminated.
In file included from ./arch/arm64/include/asm/dma-mapping.h:30:0,
from ./include/linux/dma-mapping.h:210,
from ./arch/arm64/include/asm/pci.h:7,
from ./include/linux/pci.h:1670,
from /var/lib/dkms/gasket/1.0/build/gasket_page_table.h:15,
from /var/lib/dkms/gasket/1.0/build/gasket_page_table.c:42:
./arch/arm64/include/asm/xen/hypervisor.h:1:10: fatal error: …/…/arm/include/asm/xen/hypervisor.h: No such file or directory
#include <…/…/arm/include/asm/xen/hypervisor.h>
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
In file included from ./arch/arm64/include/asm/dma-mapping.h:30:0,
from ./include/linux/dma-mapping.h:210,
from ./arch/arm64/include/asm/pci.h:7,
from ./include/linux/pci.h:1670,
from /var/lib/dkms/gasket/1.0/build/gasket_core.h:16,
from /var/lib/dkms/gasket/1.0/build/gasket_core.c:12:
./arch/arm64/include/asm/xen/hypervisor.h:1:10: fatal error: …/…/arm/include/asm/xen/hypervisor.h: No such file or directory
#include <…/…/arm/include/asm/xen/hypervisor.h>
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
scripts/Makefile.build:335: recipe for target ‘/var/lib/dkms/gasket/1.0/build/gasket_ioctl.o’ failed
make[1]: *** [/var/lib/dkms/gasket/1.0/build/gasket_ioctl.o] Error 1
make[1]: *** Waiting for unfinished jobs…
In file included from ./arch/arm64/include/asm/dma-mapping.h:30:0,
from ./include/linux/dma-mapping.h:210,
from ./arch/arm64/include/asm/pci.h:7,
from ./include/linux/pci.h:1670,
from /var/lib/dkms/gasket/1.0/build/apex_driver.c:17:
./arch/arm64/include/asm/xen/hypervisor.h:1:10: fatal error: …/…/arm/include/asm/xen/hypervisor.h: No such file or directory
#include <…/…/arm/include/asm/xen/hypervisor.h>
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
In file included from ./arch/arm64/include/asm/dma-mapping.h:30:0,
from ./include/linux/dma-mapping.h:210,
from ./arch/arm64/include/asm/pci.h:7,
from ./include/linux/pci.h:1670,
from /var/lib/dkms/gasket/1.0/build/gasket_core.h:16,
from /var/lib/dkms/gasket/1.0/build/gasket_sysfs.h:21,
from /var/lib/dkms/gasket/1.0/build/gasket_sysfs.c:3:
./arch/arm64/include/asm/xen/hypervisor.h:1:10: fatal error: …/…/arm/include/asm/xen/hypervisor.h: No such file or directory
#include <…/…/arm/include/asm/xen/hypervisor.h>
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
scripts/Makefile.build:335: recipe for target ‘/var/lib/dkms/gasket/1.0/build/gasket_interrupt.o’ failed
make[1]: *** [/var/lib/dkms/gasket/1.0/build/gasket_interrupt.o] Error 1
scripts/Makefile.build:335: recipe for target ‘/var/lib/dkms/gasket/1.0/build/gasket_page_table.o’ failed
make[1]: *** [/var/lib/dkms/gasket/1.0/build/gasket_page_table.o] Error 1
scripts/Makefile.build:335: recipe for target ‘/var/lib/dkms/gasket/1.0/build/gasket_core.o’ failed
make[1]: *** [/var/lib/dkms/gasket/1.0/build/gasket_core.o] Error 1
scripts/Makefile.build:335: recipe for target ‘/var/lib/dkms/gasket/1.0/build/apex_driver.o’ failed
make[1]: *** [/var/lib/dkms/gasket/1.0/build/apex_driver.o] Error 1
scripts/Makefile.build:335: recipe for target ‘/var/lib/dkms/gasket/1.0/build/gasket_sysfs.o’ failed
make[1]: *** [/var/lib/dkms/gasket/1.0/build/gasket_sysfs.o] Error 1
Makefile:1644: recipe for target ‘module/var/lib/dkms/gasket/1.0/build’ failed
make: *** [module/var/lib/dkms/gasket/1.0/build] Error 2
make: Leaving directory ‘/usr/src/app/kernel_modules_headers’
Hi,
Please try with v2.58.3+rev6 from https://dashboard.balena-staging.com, it has the gasket and apex modules included in the hostOS. You may need to use dma_bit_mask=32 parameter when loading the module if you encounter HIB error when running inferences.
Thank you, sadly the release is not available jet for the n30b-nano boards
I see, so you need those drivers for the jn30b-nano and not for the jetson-nano.
I added them now for jn30b , please check again for 2.58.3+rev6 in the staging dashboard.
I also have a Coral PCIe / m.2 but on a generic x86_64 device. It is running balenaOS 2.58.6+rev1 and the coral accelerator device is listed when calling lspci and shows up as /dev/apex_0. However when trying to use the device I get the following error:
ValueError: Failed to load delegate from libedgetpu.so.1
I experienced a similar issue before when I was trying to use the coral usb stick and was able to solve this by enabling UDEV devices by running the entry.sh script. However switching to the PCIe / m.2 device didn’t work.
@acostach do you have any idea how this can be solved?
Hi Toby,
The x86_64 image does not include the apex driver for the coral, did you build it for this machine and load it from your app container?
Hello and thanks for the quick reply. I did not build it for this device. How would I go about doing that?
The driver sources are available in the Coral kernel sources: https://coral.googlesource.com/linux-imx/+/refs/heads/release-chef/drivers/staging/gasket/
An example for building a module out of tree in container is available here: https://github.com/balena-os/kernel-module-build
An example Dockerfile for installing debs is available here - please note that it’s meant to be used with the coral-dev board, so you’ll need to adapt it to use a genericx86-64 base image, set arch=amd64 for debs, etc.
1 Like
After following your instructions I tried to install the kernel module. However when I ran insmod gasket.ko for the first time it turned out it was already installed and running. The module also has the same version as the one I just build (1.1.2). So the suggested solution probably doesn’t work. Do you have another idea how to solve this?
Hi Toby,
Not sure I understand the new problem, was the module inserted by the script in the container sample app already - https://github.com/balena-os/kernel-module-build/blob/93da0fefec03692c0b5a819355d599b449b9c5c7/run.sh#L10 - or does the one that was already present come from somewhere else? You can try to rmmod and insmod both modules, gasket and apex, as they depend on one another.
Dear @acostach,
I did not insert the module by script as it was already present in the container. I recall reloading both modules and that this did not fix the issue. I’ll see if I can try again this week.
Just checked, both modules are running. I am not able to use ‘rmmod’ on either module as both are in use.
Could it possibly have something to do with conflicting kernel modules? See https://coral.ai/docs/m2/get-started/#workaround-to-disable-apex-and-gasket
Hi Toby,
Yes, the genericx86-64 has those 2 modules in the hostOS already and they are provided by the upstream kernel. And yes, it’s possible that they are the ones that cause the error. You can try blacklist them in the hostOS and try load the ones you built in container, from the coral repositories.
1 Like
Hi @acostach,
We managed to get the M.2 Coral accelerator working on v2.58.6+rev1 of the genericx86-64 image after blacklisting the in-kernel apex and gasket modules and using the ones from the Coral kernel sources.
However, we haven’t found a way to perform the blacklisting of the modules from a container. Right now we’re forced to log into the OS of a device to blacklist the modules through a modprobe config file.
I found a post in this forum that describes how to perform the blacklisting from a container. However, it assumes that you can rmmod the module to install a new one. This is not possible for the apex/gasket modules because they get loaded during boot-up and remain in use by the M.2 accelerator. I even tried to unbind the driver through sysfs but the kernel kept responding with permission denied errors.
An alternative for blacklisting the modules is through the kernel parameters of course, but it seems Balena doesn’t support changing them yet.
Do you have any suggestions for how to get the blacklisting working from a container? Or have I exhausted all avaliable options?
Thanks in advance for the help!
hi, you mention above you tried unbinding the driver through sysfs but got permission denied errors. Was this performed from the container application? Could you try to do it from the hostOS instead?
If that works, you can probably add a udev rule on device detection that calls a script to perform the unbinding. You can add custom udev rules in the config.json file as described in Configuration - Balena Documentation.
@alexgg The unbinding of the driver was performed from the hostOS. However, I think that the apex driver doesn’t support being unbound and that results in the permission denied errors.
That’s unfortunate, but at least you could have the udev-triggered script create the blacklisting config instead of doing it manually.
The other solution would be to build a custom image, but that might be too involved for your case. We intend to release a feature making this kind of modifications easier - no ETA yet, though.
1 Like