Build Out-Of-Tree Kernel Module For USB Device on CM4

I’m trying to build an out of tree module for a Pure LiFi USB dongle but I’m not having much luck.

I modified v2.0.2 of this repo GitHub - balena-os/kernel-module-build: Example project for building an OOT kernel module in balena to pull in the module to use Balena OS version 2.88.4.dev as well as copy in the module source.

The module seems to build correctly but when the container runs insmod the following error is generated
insmod: can't insert '/etc/output/linux_r16858_raspberrypicm4-ioboard_2.88.4.dev/purelifi.ko': unknown symbol in module, or unknown parameter

Here is the Docker.template file

FROM ubuntu:jammy-20220531 as kernel-build

ARG DEBIAN_FRONTEND=noninteractive
ENV TZ=Etc/UTC

RUN apt-get update && \
    apt-get install -y \
    awscli \
    bc \
    bison \
    build-essential \
    curl \
    flex \
    git \
    libelf-dev \
    libssl-dev \
    wget

WORKDIR /usr/src/app

# Clones the kernel builder repository
# RUN git clone https://github.com/balena-os/kernel-module-build.git .

# You can switch to a specific commit of the kernel builder if you want to ensure consistent builds
# git reset --hard f889851c5d00890b51b27c63eaa59a976063265a

# Copy your kernel source from your local build context to the build directory
COPY . .

# Set the name of the directory where you have copied your kernel source in the step above
ENV MOD_DIR "linux_r16858"

# Set the balena OS version you intend to use
ENV OS_VERSION '2.88.4.dev'


# Start the build
RUN BALENA_MACHINE_NAME=%%BALENA_MACHINE_NAME%% ./build.sh build --device %%BALENA_MACHINE_NAME%% --os-version "$OS_VERSION" --src "$MOD_DIR"


FROM alpine

# Set the directory where you would like your kernel source
ARG MOD_PATH=/etc/output
ENV MOD_DIR linux_r16858

# Required for access when the container starts
ENV MOD_PATH="$MOD_PATH"

# Copy the built kernel module into your app
COPY --from=kernel-build /usr/src/app/output/ "$MOD_PATH"

# Copy the startup script for loading the modules
COPY --from=kernel-build /usr/src/app/run.sh /usr/src/app/run.sh

# Start the script that loads the modules.
ENTRYPOINT ["sh", "/usr/src/app/run.sh"]

# Run your usual service with CMD
CMD exec /bin/sh -c "trap : TERM INT; sleep 9999999999d & wait"

Unfortunately I don’t have much experience working with out of tree modules so I’m not sure what I should try next.

I can forward the module drivers if that would help but in the interim here is the module Makefile.

# ===========================================================================
#  File          :    Makefile
#  Entity        :    linux
#  Purpose       :    Makefile of purelifi driver
# ===========================================================================
#  Design Engineer : Angelos Spanos (DE)
#  Creation Date   : 18-Jan-2016
# ===========================================================================

TARGET=purelifi
TARGET_MODULE=$(TARGET).ko

obj-m += $(TARGET).o 

$(TARGET)-objs := $(TARGET)_chip.o $(TARGET)_mac.o $(TARGET)_usb.o
PWD = $(shell pwd)
EXTRA_CFLAGS +=-Wdeclaration-after-statement
EXTRA_CFLAGS +=-DTYPE_STA
KERNEL_PATH=/lib/modules/$(shell uname -r)/build
MODULE_INSTALLATION_PATH=/lib/modules/$(shell uname -r)/kernel/drivers/net/wireless/
FIRMWARE_BASE_PATH_GENERIC=/lib/firmware
FIRMWARE_BASE_PATH_KERNEL=$(FIRMWARE_BASE_PATH_GENERIC)/$(shell uname -r)/$(TARGET)
FIRMWARE_INSTALLATION_PATH=purelifi/li_fi_x
FIRMWARE_SOURCE_PATH=firmware
FIRMWARE_SOURCE_BINARY={fpga.bit,usb_slave_fw.hex}
RM_FLAGS=-rf
SRC+=$(wildcard *.c)
SRC+=$(wildcard *.h)
MODULE_BOOT_LAOD_FILE=/etc/modules
all: $(TARGET_MODULE)
ifneq ($(wildcard /etc/pm/config.d/config),)
ifeq ($(shell grep SUSPEND_MODULES /etc/pm/config.d/config),)
install:$(TARGET_MODULE) add_suspend_module
else
ifeq ($(shell grep SUSPEND_MODULES /etc/pm/config.d/config | grep purelifi),)
install:$(TARGET_MODULE) append_suspend_module
endif
endif
endif

$(TARGET_MODULE):$(SRC)
	@if [ ! -d "$(KERNEL_PATH)" ]; then echo "$(KERNEL_PATH) doesn't exist. Please install the kernel headers first."; false; fi
	make -C $(KERNEL_PATH) M=$(PWD) EXTRA_CFLAGS="$(EXTRA_CFLAGS)" modules

install:
	@if [ ! -d "$(MODULE_INSTALLATION_PATH)" ]; then mkdir -p $(MODULE_INSTALLATION_PATH); fi
	cp -f $(TARGET_MODULE) $(MODULE_INSTALLATION_PATH)
	@if [ "$(shell grep $(TARGET) $(MODULE_BOOT_LAOD_FILE))" != "$(TARGET)" ]; then echo $(TARGET) >> $(MODULE_BOOT_LAOD_FILE); fi
	@if [ -d "$(FIRMWARE_BASE_PATH_KERNEL)" ]; then \
		mkdir -p $(FIRMWARE_BASE_PATH_KERNEL)/$(FIRMWARE_INSTALLATION_PATH)/ ; \
	fi
	mkdir -p $(FIRMWARE_BASE_PATH_GENERIC)/$(FIRMWARE_INSTALLATION_PATH)/
	$(shell find $(FIRMWARE_BASE_PATH_GENERIC)/$(FIRMWARE_INSTALLATION_PATH) -type l -name fpga.bit -exec rm {} \;)
	cp $(FIRMWARE_SOURCE_PATH)/* $(FIRMWARE_BASE_PATH_GENERIC)/$(FIRMWARE_INSTALLATION_PATH)/
	depmod
	modprobe -r $(TARGET)
	modprobe $(TARGET)

add_suspend_module:
	echo 'SUSPEND_MODULES="purelifi"' >> /etc/pm/config.d/config

append_suspend_module:
	$(shell sed -i 's/\(SUSPEND_MODULES *= *"\)\(.*\)/\1$(TARGET) \2/g' /etc/pm/config.d/config)

tags:
	ctags -R ./ ../../mac80211/ $(KERNEL_PATH)

clean:
	@$(RM) $(RM_FLAGS) $(TARGET_MODULE) *.o* .*.cmd .*.mk Module.symvers modules.order $(TARGET).mod.c .tmp_versions *.ko*

Thanks,

Any thoughts on how best to troubleshoot the module load failure?

Unknown symbol errors could indicate that it has built the kernel modules for the wrong OS.

I assume the hashed out line for the git clone (# RUN git clone https://github.com/balena-os/kernel-module-build.git .) isn’t what is in the Dockerfile or I suspect you wouldn’t have got this far?

Is there any particular reason you are trying 2.88.4.dev rather than the latest CM4 OS?

I don’t see why it would make a difference in terms of this error, but I don’t think you will need to specify .dev for the OS version as the production and dev images are going to be the same in terms of kernel.

You could also take a look in /etc/output on the device. There will probably be a number of folders in there. You could go inside each one and try running insmod purelifi.ko in each to see if any of the others built in a way that is compatible.

@maggie0002

I have cloned kernel-module-build repo locally and replaced that line with the COPY . . line below. I’m using version 2.88.4 due to some issues I was having with newer versions of the OS. The .dev was added because it was pulling both a .prod and .dev kernel files and I only needed the dev ones as that’s what I’m running.

As for trying other *.ko files there is only one in the build folder. Is there no way to get more insight into the failure? Should I try a more complete image for the second stage of the build?

Thanks,

Are there any more errors in dmesg right after executing the insmod command?

What were the issues with the latest OS by the way? It would be great to be able to take a look at that too.

@maggie0002

I have gotten a lot further by looking at dmesg. It was missing the mac80211 module, I changed the run.sh script to modprobe mac80211 and then running insmod.

Now the issue I’m having is the firmware for the module can’t be found. I tried adding the firmware to the container using the following commands in the Docker file but that doesn’t seem to work.

RUN mkdir -p /lib/firmware/purelifi/li_fi_x

COPY --from=kernel-build /usr/src/app/linux_r16858/firmware/ /lib/firmware/purelifi/li_fi_x/

So now I’m stuck trying to figure out how to get the firmware to load.

Thoughts as how to move forward?

Thanks,

@maggie0002

I made a lot of additional progress after my previous post yesterday. I switched to using a docker-compose style of build and followed this Balena post how-to-mount-lib-firmware-rw and I was able to have the module find the firmware and load successfully.

This provided a new network interface named wlp1s0u1u3 as seen in ifconfig. So now I think I need to follow this doc Network Setup on balenaOS - Balena Documentation to add the correct SSID.

One question I have is how do I tell the network config which WiFi interface to use?

Thanks,

Great news on the progress. It seems you came across the io.balena.features.firmware: '1' and io.balena.features.kernel-modules: '1' labels that did the trick on that second problem? Is there anything you think we should add to the kernel-module-build docs that would have made any of this easier?

One way to interact with network interfaces is through network-manager. Depending on the container you are using the package may be nmcli or network-manager. You can mount the host dbus in to the container with the io.balena.features.dbus: '1' label in your docker-compose file then use network manager to interact with interfaces.

@maggie0002

Thank you for the help! I was able to get this working earlier this week.

Cheers,