Boot from NVMe on Custom CM4 Carrier

Product support balenaOS
1

I am currently working on a CM4 carrier board that requires a large amount of storage. I would like to boot from the NVMe drive that will be mounted on the carrier board and I see this is currently possible with standard firmware Raspberry Pi Documentation - Raspberry Pi Hardware and Balena supports booting from an SSD How to boot balenaOS on an SSD: why it matters and how it works.

How can I boot from the NVMe using Balena?

Thanks,

2

Update:

I followed instructions here Raspberry Pi Documentation - Raspberry Pi Hardware on how to boot from the NVMe. It seems like the support isn’t experimental any more as the recovery/boot.conf file GitHub - raspberrypi/usbboot: Raspberry Pi USB booting code, moved from tools repository now includes NVMe in the boot order. Once I updated the CM4’s EEPROM using these steps https://dphacks.com/2021/11/21/how-to-boot-a-pi-cm4-from-nvme-ssd/ I was able to image RPi OS Bullseye onto my NVMe via a USB carrier and then run that on my CM4 IO in a PCIe carrier.

I haven’t had a chance to test with a Balena image yet.

3

I tried with Balena this evening a development version of Balena OS 2.88+rev.0 for the CM4 IO board installed on the NVMe and it didn’t work :frowning:
The CM4 didn’t boot and nothing showed up on the monitor I had connected to HDMI 0. The red and green LEDs were both on.

I removed the NVMe and replaced Balena OS with RPI OS Lite and it booted without issue.

Thoughts on how to get the NVMe to boot Balena OS?

Thanks,
Darren

8

Hi Darren,

I have a follow-up question. Which PCIe carrier (NVMe adapter) did you use? Perhaps one of these?:

9

@rosswesleyporter

I used this one Rivo M.2 PCIe PCI-E Adapter

It works when I run RPi Bullseye.

14

Hi Darren,

You are doing the right things. I have some different PCIe to M.2 adapters. So I’ll give those a try. I’m not really expecting that to solve the problem, but it may provide some additional symptoms / clues.

20

Hi Darren,

This is a quick note to say that I reproduced the issue. I did this in the same order that you did i.e. booted Pi OS from NVMe. Then attempted to boot balenaOS from NVMe, capturing UART output in the process. We are looking into this. FYI, here are key bits of the UART output:

PCIe BRCM: link up, 5.0 Gbps x1 (SSC)
starting USB...
No working controllers found
Hit any key to stop autoboot: 0 
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1...
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Card did nt respond to voltage select! : -110
Scanning disk mmcnr@7e300000.blk...
Disk mmcnr@7e300000.blk not ready
Scanning dsk mmc@7e340000.blk...
Found 3 disks
No EFI system partition
BootOrder not defined
...
22

@rosswesleyporter Thanks for testing this. It was on my list to do today or tomorrow. It looks like the Balena OS just doesn’t know where to look for the boot image. I am sure this will get solved by the team.

Thanks for looking into this for me and capturing the serial debug data.

26

@rosswesleyporter Any update that you can share on booting from NVMe?

27

Hi Darren,

An engineer is looking into this, but I don’t have any specific news to share.

BTW, I did discover, as you probably did, that balenaOS recognizes the NVMe drive. That is, I booted from eMMC and the NVMe drive was visible. But I was not able to boot from NVMe. I did not try this, but I suspect that balenaOS will boot from the NVMe-drive-in-USB-carrier.

We will keep you posted.

29

@rosswesleyporter Thank you very much for update.

I didn’t try checking if the NVMe drive was available in Balena but that is good to know. As we have talked about I really need to have a large backing store to hold my video files and I would like to do this without Samba as I am reserving that a USB backup drive. From our previous conversations Balena currently doesn’t support volumes across multiple containers.

30

@rosswesleyporter Any additional updates on this? We are really hoping to test with Balena booting and running from a NVMe.

Thanks,

32

Hi Darren,

We are still looking into it. So I don’t have any big news to share. But it looks like it will require a new release of balenaOS as the new rpi firmware is required. As you know, I’m not on the OS team. But as a very short-term measure, I tried to hack it a bit, so no luck so far. We’ll keep you posted.

35

Hello,

I’m looking into using a CM4 module with a NVMe SSD using this carrier board and M.2 adapter: Raspberry Pi CM4 - Andino Systems
I was wondering if this would work with BalenaOS.

@WestCoastDaz were you able to boot from NVMe?
Or are you booting from eMMC and then using the SSD as a second disk?

Thanks

36

HI @maartenc , booting from the NVME works on the v2.94.4+rev1 CM4 IO-Board image, I’ve tested this using the CM4 IO-Board Devkit. Note that you’ll need to set the boot order for NVME with usbboot if your eMMC/SD contains a bootable image.

1 Like
37

@maartenc

After some testing we decided to boot from eMCC as this is always known to be on the device. We are just mounting the NVME into one of our containers as a large drive.