I tried with Balena this evening a development version of Balena OS 2.88+rev.0 for the CM4 IO board installed on the NVMe and it didn’t work
The CM4 didn’t boot and nothing showed up on the monitor I had connected to HDMI 0. The red and green LEDs were both on.
I removed the NVMe and replaced Balena OS with RPI OS Lite and it booted without issue.
Thoughts on how to get the NVMe to boot Balena OS?
You are doing the right things. I have some different PCIe to M.2 adapters. So I’ll give those a try. I’m not really expecting that to solve the problem, but it may provide some additional symptoms / clues.
This is a quick note to say that I reproduced the issue. I did this in the same order that you did i.e. booted Pi OS from NVMe. Then attempted to boot balenaOS from NVMe, capturing UART output in the process. We are looking into this. FYI, here are key bits of the UART output:
PCIe BRCM: link up, 5.0 Gbps x1 (SSC)
starting USB...
No working controllers found
Hit any key to stop autoboot: 0
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1...
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Card did nt respond to voltage select! : -110
Scanning disk mmcnr@7e300000.blk...
Disk mmcnr@7e300000.blk not ready
Scanning dsk mmc@7e340000.blk...
Found 3 disks
No EFI system partition
BootOrder not defined
...
@rosswesleyporter Thanks for testing this. It was on my list to do today or tomorrow. It looks like the Balena OS just doesn’t know where to look for the boot image. I am sure this will get solved by the team.
Thanks for looking into this for me and capturing the serial debug data.
An engineer is looking into this, but I don’t have any specific news to share.
BTW, I did discover, as you probably did, that balenaOS recognizes the NVMe drive. That is, I booted from eMMC and the NVMe drive was visible. But I was not able to boot from NVMe. I did not try this, but I suspect that balenaOS will boot from the NVMe-drive-in-USB-carrier.
I didn’t try checking if the NVMe drive was available in Balena but that is good to know. As we have talked about I really need to have a large backing store to hold my video files and I would like to do this without Samba as I am reserving that a USB backup drive. From our previous conversations Balena currently doesn’t support volumes across multiple containers.
We are still looking into it. So I don’t have any big news to share. But it looks like it will require a new release of balenaOS as the new rpi firmware is required. As you know, I’m not on the OS team. But as a very short-term measure, I tried to hack it a bit, so no luck so far. We’ll keep you posted.
I’m looking into using a CM4 module with a NVMe SSD using this carrier board and M.2 adapter: Raspberry Pi CM4 - Andino Systems
I was wondering if this would work with BalenaOS.
@WestCoastDaz were you able to boot from NVMe?
Or are you booting from eMMC and then using the SSD as a second disk?
HI @maartenc , booting from the NVME works on the v2.94.4+rev1 CM4 IO-Board image, I’ve tested this using the CM4 IO-Board Devkit. Note that you’ll need to set the boot order for NVME with usbboot if your eMMC/SD contains a bootable image.
After some testing we decided to boot from eMCC as this is always known to be on the device. We are just mounting the NVME into one of our containers as a large drive.
In case anyone is looking for a solution, I am able to install BalenaOS into a nVME drive with a carrier board.
Follow the configuration necessary to use rpiboot of your carrier board.
❯ sudo rpiboot
Password:
RPIBOOT: build-date pkg-version local 4dca4d92
Please fit the EMMC_DISABLE / nRPIBOOT jumper before connecting the power and USB cables to the target device.
If the device fails to connect then please see https://rpltd.co/rpiboot for debugging tips.
Waiting for BCM2835/6/7/2711/2712...
Directory not specified - trying default /usr/local/share/rpiboot/mass-storage-gadget64/
Sending bootcode.bin
Successful read 4 bytes
Waiting for BCM2835/6/7/2711/2712...
Second stage boot server
File read: mcb.bin
File read: memsys00.bin
File read: memsys01.bin
File read: memsys02.bin
File read: memsys03.bin
File read: memsys04.bin
File read: memsys05.bin
File read: memsys06.bin
File read: memsys07.bin
File read: memsys08.bin
File read: bootmain
Loading: /usr/local/share/rpiboot/mass-storage-gadget64//config.txt
File read: config.txt
Loading: /usr/local/share/rpiboot/mass-storage-gadget64//boot.img
File read: boot.img
Second stage boot server done
Download the image from Balena by creating a Fleet and adding a Device