On CM5 (balenaOS 7.4.0+rev1), the stage-1 kernel (balena_stage2 maxcpus=0) adds ~65s before the real kernel starts, and on unclean resets it intermittently enters a retry loop: ~45s dark, ~23s with DHCP networking up (ICMP answers, no services), then a silent reset and a fresh attempt. We measured 4 consecutive failures = 6m14s to a reachable device, vs 77s on a clean boot of the same unit.
Question 1: what is stage-1 doing during the ~23s networked window, and what failure triggers the reset/retry?
Question 2: is there any way to speed up stage-1, given maxcpus=0 (we verified removing it breaks SMP in the kexec’d kernel — stage-2 comes up with 1 CPU)?
The RaspberryPi 5 balenaOS boot starts with a stage2 small kernel boot that kexecs into the final kernel. As you mention, on the available kernel version kexec only works with SMP disabled, so this second stage kernel boots with maxcpus=0. This stage does add some delay to the boot, ~4.5s that will be reduced once SMP is working for this use case. The measurements I get on my RPI5 are:
Phase
Duration
Bootloader (BOOTSYS → first Booting Linux, incl. firmware, SDRAM, ATF/BL31)
My device is also not entering retry loops on unclean resets (I am just plugging the device off) - can you be more specific about how to reproduce this issue please?
Unfortunately I don’t have a reliable pattern to trigger the retry loops, it just seems intermittent. Your 28s is a lot better than mine though, that is odd, same BalenaOS version?