I am currently working on a CM4 carrier board that requires a large amount of storage. I would like to boot from the NVMe drive that will be mounted on the carrier board and I see this is currently possible with standard firmware Raspberry Pi Documentation - Raspberry Pi Hardware and Balena supports booting from an SSD How to boot balenaOS on an SSD: why it matters and how it works.
How can I boot from the NVMe using Balena?
Thanks,
Update:
I followed instructions here Raspberry Pi Documentation - Raspberry Pi Hardware on how to boot from the NVMe. It seems like the support isn’t experimental any more as the recovery/boot.conf file GitHub - raspberrypi/usbboot: Raspberry Pi USB booting code, moved from tools repository now includes NVMe in the boot order. Once I updated the CM4’s EEPROM using these steps https://dphacks.com/2021/11/21/how-to-boot-a-pi-cm4-from-nvme-ssd/ I was able to image RPi OS Bullseye onto my NVMe via a USB carrier and then run that on my CM4 IO in a PCIe carrier.
I haven’t had a chance to test with a Balena image yet.
I tried with Balena this evening a development version of Balena OS 2.88+rev.0 for the CM4 IO board installed on the NVMe and it didn’t work 
The CM4 didn’t boot and nothing showed up on the monitor I had connected to HDMI 0. The red and green LEDs were both on.
I removed the NVMe and replaced Balena OS with RPI OS Lite and it booted without issue.
Thoughts on how to get the NVMe to boot Balena OS?
Thanks,
Darren
Hi Darren,
I have a follow-up question. Which PCIe carrier (NVMe adapter) did you use? Perhaps one of these?:
Hi Darren,
You are doing the right things. I have some different PCIe to M.2 adapters. So I’ll give those a try. I’m not really expecting that to solve the problem, but it may provide some additional symptoms / clues.
Hi Darren,
This is a quick note to say that I reproduced the issue. I did this in the same order that you did i.e. booted Pi OS from NVMe. Then attempted to boot balenaOS from NVMe, capturing UART output in the process. We are looking into this. FYI, here are key bits of the UART output:
PCIe BRCM: link up, 5.0 Gbps x1 (SSC)
starting USB...
No working controllers found
Hit any key to stop autoboot: 0
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1...
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Card did nt respond to voltage select! : -110
Scanning disk mmcnr@7e300000.blk...
Disk mmcnr@7e300000.blk not ready
Scanning dsk mmc@7e340000.blk...
Found 3 disks
No EFI system partition
BootOrder not defined
...
@rosswesleyporter Thanks for testing this. It was on my list to do today or tomorrow. It looks like the Balena OS just doesn’t know where to look for the boot image. I am sure this will get solved by the team.
Thanks for looking into this for me and capturing the serial debug data.
@rosswesleyporter Any update that you can share on booting from NVMe?
Hi Darren,
An engineer is looking into this, but I don’t have any specific news to share.
BTW, I did discover, as you probably did, that balenaOS recognizes the NVMe drive. That is, I booted from eMMC and the NVMe drive was visible. But I was not able to boot from NVMe. I did not try this, but I suspect that balenaOS will boot from the NVMe-drive-in-USB-carrier.
We will keep you posted.
@rosswesleyporter Thank you very much for update.
I didn’t try checking if the NVMe drive was available in Balena but that is good to know. As we have talked about I really need to have a large backing store to hold my video files and I would like to do this without Samba as I am reserving that a USB backup drive. From our previous conversations Balena currently doesn’t support volumes across multiple containers.
@rosswesleyporter Any additional updates on this? We are really hoping to test with Balena booting and running from a NVMe.
Thanks,
Hi Darren,
We are still looking into it. So I don’t have any big news to share. But it looks like it will require a new release of balenaOS as the new rpi firmware is required. As you know, I’m not on the OS team. But as a very short-term measure, I tried to hack it a bit, so no luck so far. We’ll keep you posted.
Hello,
I’m looking into using a CM4 module with a NVMe SSD using this carrier board and M.2 adapter: Raspberry Pi CM4 - Andino Systems
I was wondering if this would work with BalenaOS.
@WestCoastDaz were you able to boot from NVMe?
Or are you booting from eMMC and then using the SSD as a second disk?
Thanks
HI @maartenc , booting from the NVME works on the v2.94.4+rev1 CM4 IO-Board image, I’ve tested this using the CM4 IO-Board Devkit. Note that you’ll need to set the boot order for NVME with usbboot if your eMMC/SD contains a bootable image.
After some testing we decided to boot from eMCC as this is always known to be on the device. We are just mounting the NVME into one of our containers as a large drive.
In case anyone is looking for a solution, I am able to install BalenaOS into a nVME drive with a carrier board.
rpiboot of your carrier board.❯ sudo rpiboot
Password:
RPIBOOT: build-date pkg-version local 4dca4d92
Please fit the EMMC_DISABLE / nRPIBOOT jumper before connecting the power and USB cables to the target device.
If the device fails to connect then please see https://rpltd.co/rpiboot for debugging tips.
Waiting for BCM2835/6/7/2711/2712...
Directory not specified - trying default /usr/local/share/rpiboot/mass-storage-gadget64/
Sending bootcode.bin
Successful read 4 bytes
Waiting for BCM2835/6/7/2711/2712...
Second stage boot server
File read: mcb.bin
File read: memsys00.bin
File read: memsys01.bin
File read: memsys02.bin
File read: memsys03.bin
File read: memsys04.bin
File read: memsys05.bin
File read: memsys06.bin
File read: memsys07.bin
File read: memsys08.bin
File read: bootmain
Loading: /usr/local/share/rpiboot/mass-storage-gadget64//config.txt
File read: config.txt
Loading: /usr/local/share/rpiboot/mass-storage-gadget64//boot.img
File read: boot.img
Second stage boot server done
=============================================================
Welcome to balenaOS
=============================================================
root@9dff74c:~# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 1.7G 0 1.7G 0% /dev
tmpfs 1.9G 6.0M 1.9G 1% /run
/dev/disk/by-state/active 300M 252M 29M 90% /mnt/sysroot/active
/dev/disk/by-state/resin-state 18M 76K 16M 1% /mnt/state
overlay 300M 252M 29M 90% /
/dev/nvme0n1p6 466G 109M 442G 1% /mnt/data
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 4.0M 0 4.0M 0% /sys/fs/cgroup
tmpfs 1.9G 8.0K 1.9G 1% /tmp
/dev/nvme0n1p1 40M 8.1M 32M 21% /mnt/boot
tmpfs 1.9G 76K 1.9G 1% /var/volatile
/dev/nvme0n1p3 300M 14K 280M 1% /mnt/sysroot/inactive
root@9dff74c:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
mmcblk0 179:0 0 29.1G 0 disk
`-mmcblk0p1 179:1 0 29.1G 0 part
mmcblk0boot0 179:32 0 4M 1 disk
mmcblk0boot1 179:64 0 4M 1 disk
zram0 254:0 0 1.9G 0 disk [SWAP]
nvme0n1 259:0 0 476.9G 0 disk
|-nvme0n1p1 259:1 0 40M 0 part /mnt/boot
|-nvme0n1p2 259:2 0 320M 0 part /mnt/sysroot/active
|-nvme0n1p3 259:3 0 320M 0 part /mnt/sysroot/inactive
|-nvme0n1p4 259:4 0 1K 0 part
|-nvme0n1p5 259:5 0 20M 0 part /var/volatile/lib/bluetooth
| /var/lib/bluetooth
| /var/volatile/lib/systemd
| /var/lib/systemd
| /var/volatile/lib/chrony
| /var/lib/chrony
| /var/volatile/lib/NetworkManager
| /var/lib/NetworkManager
| /usr/share/ca-certificates/balena
| /home/root/.ssh
| /home/root/.rnd
| /home/root/.docker
| /etc/udev/rules.d
| /etc/ssh/hostkeys
| /etc/openvpn
| /etc/hostname
| /etc/docker
| /etc/balena-supervisor
| /etc/NetworkManager/system-connections
| /etc/NetworkManager/dispatcher.d
| /etc/NetworkManager/conf.d
| /etc/fake-hwclock
| /etc/machine-id
| /mnt/state
-nvme0n1p6 259:6 0 476.2G 0 part /var/volatile/lib/docker
/var/lib/docker
/resin-data
/mnt/data