BalenaOS 2.94.4 on RPI CM4 does not boot when NVMe drive is plugged in the PCIe slot

Product support balenaOS
44

@rosswesleyporter
Yes, I fine with the workaround for the moment.
Thanks for the update.

46

Hi all,
had a look at this a little more closer and seems this is a bug in u-boot, kernel or both.
As a quick workaround I asked the rpi devs to maybe try and let the pcie initialization in place for all boot modes which would have fixed this issue but they are reluctant to do so. The discussion on this is here: Kernel panic when booting with BOOT_ORDER=0xf25641 , using u-boot and having a NVME drive on PCIe · Issue #1800 · raspberrypi/firmware · GitHub

So until there’s progress on figuring out what the actual bug is with u-boot and the kernel then the only workaround would be to have the BOOT_ORDER changed like suggested above.

47

BOOT_ORDER=0xf25416 is described briefly above. That puts NVMe (6) before eMMC (1) in the boot order, so it will boot from eMMC if a non-bootable NVMe drive is present. I have a CM4 IO Board with CM4 v1.1 on my desk doing this now. @WestCoastDaz, is this of interest to you? If so, please do test with your setup.

On a related note, Vanmada has used this same setting to boot from NVMe.

48

Please let us know if you have any questions

50

@rosswesleyporter

Thank you for the update. I’m not sure that I am following the work around. So all I need to do is have a non-bootable NVMe drive installed and have change the boot order so it tries to boot from that fails and switches to the eMCC?

We moved away from booting from the NVMe incase that wasn’t available due to a hardware issue.

Thanks,

51

Hi team,

I´m on vacation till the end the week. I can test the work around next week.

Kind regards,

David

52

@westcoastdaz, you are correct! For everyone else, given that this is a long thread, let’s rewind:

  • If you want to boot a CM4 + CM4 IO Board from eMMC and no NVMe drive is present, then everything should work as expected.
  • If you want to boot from NVMe and the CM4 has eMMC, then use BOOT_ORDER=0xf25416 or similar i.e. put NVMe (6) before eMMC (1).
  • This thread started with reports that a CM4 v1.1 (PCB R5) will not boot from eMMC if an NVMe drive is present. Balena replicated this using the default BOOT_ORDER. And using the reference carrier board - Raspberry Pi CM4 IO Board.
  • If you want to boot a CM4 v1.1 from eMMC and have a non-bootable NVMe drive present, then try BOOT_ORDER=0xf25416. It would be slightly more efficient to put eMMC (1) first. But it turns out that putting NVMe (6) first is a viable workaround. It is safe because the NVMe boot attempt will fail on a non-bootable drive and it will then boot from eMMC.
  • The hypothesis is that there is an underlying issue with PCIe initialization and boot order. Which is the territory or u-boot and/or the kernel. There is some discussion of this between balena and the RPi devs in Kernel panic when booting with BOOT_ORDER=0xf25641 , using u-boot and having a NVME drive on PCIe · Issue #1800 · raspberrypi/firmware · GitHub.
54

@rosswesleyporter

Thank you for the clear description. After reading Kernel panic when booting with BOOT_ORDER=0xf25641 , using u-boot and having a NVME drive on PCIe · Issue #1800 · raspberrypi/firmware · GitHub it seems like the only option right now is to change the boot order as a work around for now.

Thanks,