CM4 Won't Reboot Running balenaOS 2.94.4+rev1 Running On Custom Hardware

Product support balenaOS
1

I have been testing our custom CM4 IO board and yesterday I noticed that BalenaOS wouldn’t startup after using the Reboot button on the Dashboard or after a dtoverlay was changed. I confirmed this on 2 custom boards. They were running balenaOS 2.94.4+rev1. I removed all peripherals and this still didn’t help. What is interesting the custom board would boot from a power up.

I created a new device in the Fleet running balenaOS 2.88.4 and everything works fine, the device reboots without issue.

I am using Development builds in both cases. Also something else to note is 2.94.4+rev1 showed no
U-Boot details on the serial port, only a login prompt.

Thoughts on how to troubleshoot this?

I have the 2.94.4+rev1 available on an SDCard ready to test with.

Thanks,

2

Hi @WestCoastDaz , you can try do a local yocto build with OS_DEVELOPMENT set or, you could enable uart from the dashboard and, in the sd-card’s boot partition edit the cmdline.txt
file to replace console=null with console=tty1 console=serial0,115200 debug loglevel=7. The second method will not display u-boot logs but will quickly show kernel traces if u-boot runs fine.

4

Hi @WestCoastDaz,

I see that my colleague acostach responded above. He knows more than I do about this. Nonetheless, a few thoughts from me below.

I’m keying off this phrase, “I noticed that BalenaOS wouldn’t startup after using the Reboot button on the Dashboard”. As you surmised, getting more data about the reboot would be helpful. You noted that with balenaOS 2.94.4+rev1 you are not getting much boot data via UART. This is a sample of the output that I get with an RPIO with v2.94.4+rev1:

RPi: BOOTLOADER release VERSION:d8b208d3 DATE: 2022/02/22 TIME: 09:33:50 BOOTMODE: 0x00000006 part: 0 BUILD_TIMESTAMP=1645522430 0x11955bac 0x00b03140 0x00064e0a
PM_RSTS: 0x00001000
part 00000000 reset_info 00000000
uSD voltage 3.3V
Initialising SDRAM 'Samsung' 16Gb x1 total-size: 16 Gbit 3200

Boot mode: NVME (06) order f2541
VID 0x144d MN Samsung SSD 980 500GB                  
NVME on
MBR: 0x00002000,   81920 type: 0x0c
MBR: 0x00016000,  655360 type: 0x83
MBR: 0x000b6000,  655360 type: 0x83
MBR: 0x00156000,  458752 type: 0x0f
Trying partition: 0
type: 32 lba: 8192 oem: 'mkfs.fat' volume: ' resin-boot '
rsc 32 fat-sectors 630 c-count 80628 c-size 1
root dir cluster 2 sectors 0 entries 0
FAT32 clusters 80628
Trying partition: 0
type: 32 lba: 8192 oem: 'mkfs.fat' volume: ' resin-boot '
rsc 32 fat-sectors 630 c-count 80628 c-size 1
root dir cluster 2 sectors 0 entries 0
FAT32 clusters 80628
Read config.txt bytes    36317 hnd 0x00000000
Read start4cd.elf bytes   800028 hnd 0x00000000
Read fixup4cd.dat bytes     3145 hnd 0x00000000
Firmware: bd88f66f8952d34e4e0613a85c7a6d3da49e13e2 Jan 20 2022 13:57:04
0x00b03140 0x00000000 0x00000fff
MEM GPU: 16 ARM: 998 TOTAL: 1014
Starting start4cd.elf @ 0xff000200 partition 0
+      
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[    0.000000] Linux version 5.10.83-v8 (oe-user@oe-host) (aarch64-poky-linux-gcc (GCC) 11.2.0, GNU ld (GNU Binutils) 2.37.20210721) #1 SMP PREEMPT Tue Dec 7 13:42:57 UTC 2021
[    0.000000] random: fast init done
[    0.000000] Machine model: Raspberry Pi Compute Module 4 Rev 1.0

You may have used the same config setting that I did when doing the initial setup. But just to be sure, the process that I used is below. Particularly BOOT_UART=1, enable_uart=1, uart_2ndstage=1.

  1. Build rpiboot on Mac to update the bootloader in eeprom on the CM4
  • git clone --depth=1 https://github.com/raspberrypi/usbboot
  • cd usbboot
  • Check BOOT_ORDER in recovery/boot.conf, edit as needed
  • BOOT_UART=1
  • # First try NVMe(6), then SD/eMMC(1), USB PCIe(4), USB SoC XHC(5), net(2)
  • BOOT_ORDER=0xf25416
  • cd recovery
  • ./update-pieeprom.sh
  • cd ..
  • make

  1. Power off Pi

  2. Fit jumper on Raspberry Pi CM4 IO Board (RPIO) J2 to disable eMMC boot

  3. Connect micro-USB cable to the RPIO

  4. Update eeprom

  • Run rpiboot on the host, wait for a connection:
  • ./rpiboot
  • Power the RPIO - your Mac should see the RPIO
  • Flash the new bootloader information to the Pi (you may need to power off and power on your Pi):
  • sudo ./rpiboot -d recovery
  • You should get output such as:
  • RPIBOOT: build-date Feb 26 2022 version 20220208~181027 042cd145
  • Loading: recovery/bootcode4.bin
  • Waiting for BCM2835/6/7/2711…
  • Loading: recovery/bootcode4.bin
  • Sending bootcode.bin
  • Successful read 4 bytes
  • Waiting for BCM2835/6/7/2711…
  • Loading: recovery/bootcode4.bin
  • Second stage boot server
  • Loading: recovery/config.txt
  • File read: config.txt
  • Loading: recovery/pieeprom.bin
  • Loading: recovery/pieeprom.bin
  • Loading: recovery/pieeprom.sig
  • File read: pieeprom.sig
  • Loading: recovery/pieeprom.bin
  • File read: pieeprom.bin
  • Unknown message

  1. Remove the jumper

  2. Power off the RPIO

  3. Use Etcher to flash balenaOS to NVMe drive e.g. with the drive in a USB carrier

  4. On the NVMe drive, edit config.txt to add:

  • enable_uart=1
  • uart_2ndstage=1
  1. On the NVMe drive, edit cmdline.txt such that:
  • console=tty0 console=ttyS0,115200

  1. Add NVMe drive to PCIe carrier on the RPIO

  2. Power on RPIO

5

@rosswesleyporter @acostach

Thank you for your answers. I will try your suggestions and report back to you.