rpi5: balenaOS hangs at boot from NVMe with Phison E8 controller(ASPM issue)

Product support
1

Device / Environment

  • Device type: Raspberry Pi 5 (8GB)
  • balenaOS version: 6.x 7.x
  • NVMe adapter: Geekworm X1002 (PCIe FFC, no HAT+ EEPROM)
  • Drive: 128GB, Phison E8 controller (lspci: 1987:5008 “Phison Electronics Corporation E8 PCIe3 x2 NVMe Controller (rev 01)”, firmware AP123PG0, NVMe 1.2)
  • Bootloader EEPROM: latest stable, PCIE_PROBE=1, BOOT_ORDER=0xf416

Symptom

balenaOS flashed to this NVMe hangs at boot: solid green LED, no HDMI output, no reboot loop. Device never appears in balenaCloud.

Isolation performed

  • Same balenaOS image on a different NVMe drive: boots normally
  • Raspberry Pi OS (kernel 6.18.x) on the drive: boots normally
  • Drive passes extended fio stress testing (seq + 4k randrw, ~1h) and
    repeated power cycles under Pi OS with zero NVMe/PCIe errors in dmesg
  • Combination of balenaOS kernel + this controller is the failure

Root cause

PCIe ASPM. Appending pcie_aspm=off to cmdline.txt on the boot partition makes balenaOS boot and run normally. Without it the link appears to enter L1/L1.2 during early boot and never recovers, so the kernel hangs waiting for the root device (console=null quiet in the default cmdline hides all output, making this look like a hard hang).

Raspberry Pi OS’s newer kernel boots the same drive without any parameters, suggesting an upstream brcmstb/NVMe ASPM quirk exists that the balenaOS kernel predates.

Second problem: host OS updates revert the workaround

Confirmed on the bench: triggering a host OS update from the dashboard rewrites cmdline.txt, removing pcie_aspm=off. The device then hangs on the new kernel before the rootfs/supervisor stage, so the automatic rollback health-check never engages — the device is bricked until physically serviced. For a remote fleet this makes host updates impossible.

Requests

  1. Pick up the kernel quirk (or default ASPM policy change) for brcmstb + Phison E8-class controllers on the rpi5 device type, OR
  2. Provide a supported, update-persistent mechanism for custom kernel cmdline arguments on rpi5 (equivalent to config.txt device configuration variables).

Happy to test patched images — we’re deploying this hardware combo in quantity and can reproduce reliably.