Raspberry Pi CM4 Not Booting with PCIe Device

Product support balenaOS
1

Hi,

I have a custom board that has a PCIe-USB3 chip on board and works properly (confirmed with Raspberry Pi OS with 5.10 kernel). When I try to boot using Balena OS (newly built) the device doesn’t boot, nothing comes out of the console after “Starting kernel …”. When I take the exact same CM4 and put on any other board without a PCIe device, it boots properly. I’ve even tried with the under development 5.10 branch of raspberrypi balenaOS, same result. Here are the boot logs:

Working:

U-Boot 2021.04-rc3 (Mar 03 2021 - 15:10:4 +0000)

DRAM:  2 GiB
RPI Compute Module 4 (0xb03140)
MMC:   mmcnr@7e300000: 1, emmc2@7e340000: 0
Loading Environment from FAT... *** Warning - bad CRC, using default environment

In:    serial
Err:   serial
Net:   eth0: ethernet@7d580000
PCIe BRCM: link down
starting USB...
No working controllers found
switch to partitions #0, OK
mmc0(part ) is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr
516 bytes read in 10 ms (49.8 KiB/s)
## Executing script at 02400000
Scanning mc usb devices 0 1 2
24 bytes read in 13 ms (1000 Bytes/s)
Found resin image on mmc 0
Loading resinOS_uEnv.txt from mmc device 0 partiion 1
Failed to load 'resinOS_uEnv.txt'
Loading extra_uEnv.txt from mmc device 0 partition 1
0 bytes read in 11 ms (0 Bytes/s)
Import etra_uEnv.txt in environment
Loading bootcount.env from mmc device 0 partition 1
Failed to load 'bootcount.env'
No bootcount.envfile. Setting bootcount=0 in environment
12469316 bytes read in 558 ms (21.3 MiB/s)
Uncompressed size: 28365312 = 0x1B0D200
## FlattenedDevice Tree blob at 2eff4100
   Booting using the fdt blob at 0x2eff4100
   Using Device Tree in place at 000000002eff410, end 000000002f002f6d

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[    0.000000] Linux version 5.4.83-v8 (oe-user@oe-host) (gcc version 9.3.0 (GCC)) #1 SMP PREEMPT Mon Dec 14 12:43:54 UTC 2020
[    0.000000] Machine model: Raspberry Pi Compute Module 4 Rev 1.0
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] efi: UEFI not found.
...

Not working:

U-Boo 2021.04-rc3 (Mar 03 2021 - 15:10:34 +0000)

DRAM:  2 GiB
RPI Compute Module 4 (0xb03140)
MMC:   mmcnr@7e300000: 1, emmc2@7e340000: 0
Loading Environment from FAT... *** Warning - bad CRC, using default environment

In:    serial
Err:   serial
Net:   eth0: ethernet@7d580000
PCIe BRCM: link up, 5.0 Gbps x1 (SSC)
starting USB...
Bus xhci_pci: Register 4000840 NbrPorts 4
Starting the contoller
USB XHCI 1.10
scanning bus xhci_pci for devices... 1 USB Device(s) found
       scanning usb for storage devices... 0 Stoage Device(s) found
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr
516 bytes read in 11 s (44.9 KiB/s)
## Executing script at 02400000
Scanning mmc usb devices 0 1 2
24 bytes read in 14 ms (1000 Bytes/s)
Found resin image on mc 0
Loading resinOS_uEnv.txt from mmc device 0 partition 1
Failed to load 'resinOS_uEnv.txt'
Loading extra_uEnv.txt from mmc devic 0 partition 1
0 bytes read in 12 ms (0 Bytes/s)
Import extra_uEnv.txt in environment
Loading bootcount.env from mmc device 0 patition 1
Failed to load 'bootcount.env'
No bootcount.env file. Setting bootcount=0 in environment
12469316 bytes read in 558 ms (21.3 MB/s)
Uncompressed size: 28365312 = 0x1B0D200
## Flattened Device Tree blob at 2eff4100
   Booting using the fdt blob at 0x2eff100
   Using Device Tree in place at 000000002eff4100, end 000000002f002f6d

Starting kernel ...

Any suggestions on places to look? Thanks.

3

Hello! This pretty weird. Some questions / ideas:

  • Are you using a production or development image of balenaOS? Could you try using (if you aren’t already) an un-managed image directly from our OS website?
  • could you try using the latest boot firmware? You can (grab it from here)[ firmware/boot at master · raspberrypi/firmware · GitHub]. It needs to be placed in the FAT formatted partition of the SD card. Do not copy the .dtbs from there, just overwrite the rest of the files that are on the SD card with those from the github repo. Let us know if that helps.
  • Can you provide any more information about the specs of that custom board, especially the PCIe-USB chip that seems to be in question?
7

Hi @toochevere, thanks for the suggestions.

Yes, I just tried a development image from the dashboard for the CM4-IOBoard (balenaOS 2.71.3+rev5). That does the same thing, stops at “Starting kiernel …”. Works if I put it on a board without the PCIe device.

  • could you try using the latest boot firmware? You can (grab it from here)[ firmware/boot at master · raspberrypi/firmware · GitHub]. It needs to be placed in the FAT formatted partition of the SD card. Do not copy the .dtbs from there, just overwrite the rest of the files that are on the SD card with those from the github repo. Let us know if that helps.

First I copied all the *.dat and *.elf files, that didn’t help. Next I copied over the bcm2711-rpi-cm4.dtb file, now the board boots. The driver for the PCIe->USB3 chip doesn’t load but that was also the case with Raspi OS 5.4, I needed to upgrade to 5.10 to get the driver to load.

  • Can you provide any more information about the specs of that custom board, especially the PCIe-USB chip that seems to be in question?

The board is a custom one built in Upverter, uses a CM4 and a ASMedia ASM1142 PCIe to USB3 controller. Everything works in Raspi OS 5.10 with the added overlay “dtoverlay=pcie-32bit-dma”.

So it looks like all is good as long as the updated cm4 dtb (and possibly the dat and elf files) are copied over. Thanks for the help.

8

ok, thanks for the info. If you need info on how to control dtoverlay in balenaOS, see this link. Advanced boot settings - Balena Documentation

10

Following up on this. With the updated boot firmware on a stock dev image from the dashboard, I can boot the device but now I’m seeing that the ethernet device is no longer registered (no eth0 device). When I try with kernel 5.10 the ethernet isn’t working and overlays don’t seem to be applied (i.e. adding the overlay “uart4” on the device configuration panel doesn’t result in a new ttyACA* device and GPIO pins 8 & 9 are not muxed properly, as shown by raspi-gpio). It all works properly using the Raspberry Pi OS. Any suggestions on things to try or places to look?

11

I’ll ping our devices team and see if they have any suggestions, but I suspect that there won’t be much more to add without physically having a device with that configuration to look at. Let’s see.

13

Hi, on the stock Raspberry Pi OS what kernel version are you running exactly?

16

Hi @floion , I’m running 5.10.17 on one device and another I’ve update to 5.10.31 I believe (basically used current 5.10 head).