Has anyone tried using PCIe expansion cards with the CM4 via the CM4 IO board? I’m trying to use a PCIe to mini PCIe adapter with an WPEA-121N/W (Atheros chipset) mini PCI card and the board hangs for a while on boot, then reboots.
Boots up fine with the PCIe to mini PCIe adapter, but adding the WiFi card causes it to hang.
Appreciate any advice for further troubleshooting…
FWIW, Some text flashes briefly at the bottom of the screen (HDMI0) on boot, but no Balena splash screen or meaningful logs.
Update: Boots fine with WiFi card installed, with Raspberry Pi OS (lite). Balena only boots on the board if the PCIe WiFi card is not present.
Hi Ash,
PCIe compatibility can be tricky on the Pi 4 compute module. Could you try connecting a serial UART cable to your carrier board to get the boot logs so we can see what’s going on here?
Sure thing! I’ll get that for you ASAP
I’m not getting anything on the serial console with the card in the board. It just hangs with the i/o light on solid. I’ve imaged the disk andI’m digging for any sort of logs which might indicate where it stopped. Any advice on where coredumps or crash logs are located?
Is this a debug image? Serial console output is disabled on production images, so you’ll want to flash the debug image for troubleshooting.
Hey @jakogut Looks like it hangs on starting the kernel:
DRAM: 2 GiB
RPI Compute Module 4 (0xb03140)
MMC: mmcnr@7e300000: 1, emmc2@7e340000: 0
Loading Environment from FAT... *** Warning - bad CRC, using default environment
In: serial
Err: serial
Net: eth0: ethernet@7d580000
PCIe BRCM: link up, 2.5 Gbps x1 (SSC)
starting USB...
No working controllers found
switch to partitions #0, OK
mc0(part 0) is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr
516 bytes read in 10 ms (49.8 KiB/s)
## Executing script at 0240000
Scanning mmc usb devices 0 1 2
24 bytes read in 13 ms (1000 Bytes/s)
Found resin image on mmc 0
Loading resinOS_uEnv.txt from mmc device0 partition 1
Failed to load 'resinOS_uEnv.txt'
Loading extra_uEnv.txt from mmc device 0 partition 1
0 bytes read in 11 ms (0 Bytes/s)
mport extra_uEnv.txt in environment
Loading bootcount.env from mmc device 0 partition 1
Failed to load 'bootcount.env'
No bootcunt.env file. Setting bootcount=0 in environment
12469316 bytes read in 558 ms (21.3 MiB/s)
Uncompressed size: 28365312 = 0x1B0D200
## Fattened Device Tree blob at 2eff4100
Booting using the fdt blob at 0x2eff4100
Using Device Tree in place at 00000002eff4100, end 000000002f002f6d
Starting kernel ...
Thanks for the update, Ash. What about the same image on the same device without the wireless card installed?
Hi @jakogut here are the configurations I’ve tested:
Pi CM4 with EMMC, without built-in wireless:
Starting Kernel
Ableconn Mini PCI-E to PCI-E adapter: https://www.amazon.com/gp/product/B07JBCL1CJ/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1
WiFi card: Atheros WPEA-121N/W
I can spare a WiFi card for testing, if you’d like for me to send you one. I realize that the CM4 IO board support is a new development, and the PCI-E bit is probably tough to test completely in software.
Hello Ash,
I suspect this issue is related to the Linux kernel version and we might have to update the Linux kernel to 5.10 to solve it.
Could you please run this command hostnamectl on both setups and tell me the Linux kernel version?
You’ll have to remove Mini Pci-E to PCI-E adapter + WiFi card from the balenaOS setup before you do that.
Here’s the output from Raspios
pi@raspberrypi:~$ hostnamectl
Static hostname: raspberrypi
Icon name: computer
Machine ID: 4e71607db41d4067bdf288279120f74b
Boot ID: 6b19a33cc710436dafdb9f6b994b9be1
Operating System: Raspbian GNU/Linux 10 (buster)
Kernel: Linux 5.4.83-v7l+
Architecture: arm
And from Balena
root@9cb85ee:~# hostnamectl
Static hostname: 9cb85ee
Icon name: computer
Machine ID: 9b083e06f82e4bf896d214fbcb9e50b7
Boot ID: 0b9fbd67fae44ac885bb30e8b996e021
Operating System: balenaOS 2.71.3+rev5
Kernel: Linux 5.4.83-v8
Architecture: arm64
I must have grabbed the Raspios image right before the new version dropped
Hi @MTolba is there any more information I can gather to make the troubleshooting process easier?
Hey Ash! We are still investigating the cause of the issue, as Mahmoud mentioned, we thought it might have to do with the kernel version since there is a known issue with CM4 and PCie in kernel 5.4, but you have 5.4.83 on both (however, Raspbian appears to be the 32-bit variant, whereas ours is the 64-bit build). We are trying to replicate on our end though, and will keep you posted on what we find.
Hi, could you try using the latest boot firmware? You can grab it from here https://github.com/raspberrypi/firmware/boot
It needs to be placed in the FAT formatted partition of the SD card. Do not copy the .dtbs from there, just overwrite the rest of the files that are on the SD card with those from the github repo.
Let us know if that helps
Hi @floion
Thanks for the advice- unfortunately, it’s still not working. Specifically, what I did was to replace every file that was present in both places (except for .dtb files and overlays dir) with the ones found in the master branch of the rpi firmware repo’s boot directory.
I also tried the same process, and additionally replaced the overlays directory with what’s in the repo. Same result with both attempts.
Hi, can you try again with a fresh balenaOS image in which you would only overwrite bootcode.bin, the fixup* files and the start* files from firmware/boot at 70f1581eec2c036b7e9309f1af41c651fb125447 · raspberrypi/firmware · GitHub ?
Thanks @floion , I’ll give that a try this weekend!
Hi @floion
It’s still locking up when starting the kernel. Using 2.71.3+rev5-dev-v12.3.5, and copying the files listed above, this is what I get on the serial console when the card is installed:
U-Boo 2021.04-rc3 (Mar 03 2021 - 15:10:34 +0000)
DRAM: 2 GiB
RPI Compute Module 4 (0xb03140)
MMC: mmcnr@7e300000: 1, emmc2@7e340000: 0
Loading Environment from FAT... *** Warning - bad CRC, using default environment
In: serial
Err: serial
Net: eth0: ethernet@7d580000
PCIe BRCM: link up, 2.5 Gbps x1 (SSC)
starting USB...
No working controllers found
switch to partitions #0, OK
mc0(part 0) is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr
516 bytes read in 10 ms (49.8 KiB/s)
## Executing script at 0240000
Scanning mmc usb devices 0 1 2
24 bytes read in 13 ms (1000 Bytes/s)
Found resin image on mmc 0
Loading resinOS_uEnv.txt from mmc device0 partition 1
Failed to load 'resinOS_uEnv.txt'
Loading extra_uEnv.txt from mmc device 0 partition 1
0 bytes read in 11 ms (0 Bytes/s)
mport extra_uEnv.txt in environment
Loading bootcount.env from mmc device 0 partition 1
Failed to load 'bootcount.env'
No bootcunt.env file. Setting bootcount=0 in environment
12469316 bytes read in 558 ms (21.3 MiB/s)
Uncompressed size: 28365312 = 0x1B0D200
## Fattened Device Tree blob at 2eff4100
Booting using the fdt blob at 0x2eff4100
Using Device Tree in place at 00000002eff4100, end 000000002f002f6d
Starting kernel ...
With the card removed, it gets all the way to the login prompt.