In my home, I have several Pi’s, currently running raspotify or RetroPie. I want to be able to set some of those devices in a multi-room setup and so I am looking at balenaSound to bridge that gap.
What I usually do to netboot is point to a TFTP server and edit my cmdline.txt to boot from to an NFS share. However, somewhere in loading the kernel the “rainbow” screen stays visible until the Pi reboots.
My question: how do I debug this properly? I’m used to getting some output on the screen, but there is none.
This is my current cmdline.txt for reference: dwc_otg.lpm_enable=0 dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/nfs nfsroot=<nfs_ip>:/public/raspberry-pi/devices/<raspi-serial>/root,vers=3 rw ip=dhcp rootwait elevator=deadline vt.global_cursor_default=0 cgroup_enable=memory
I’m happy to share my (full) config once I get this working
Update: I logged my NFS server, and it seems there is no activity there, after the kernel8.img has been transferred by TFTP. The previously mentioned thread hints toward rewriting initrd. Is this the way to netboot Balena? Seems rather complicated? Any reason why cmdline.txt would not work?
Update2: so digging into the initramfs some more (this is uncharted territory for me, so bear with me please ) and comparing it with the balena one, it looks like the raspbian initrd contains a bin/nfsmount folder, whereas the balena one does not. Is this assesment correct?
Can’t seem to edit my original post anymore, so replying instead.
So I found out Balena doesn’t use a initramfs at all, so I’m trying to set that up. So far I added initramfs initramfs8 to the config.txt and that seems to at least pick it up during TFTP phase. I think I need to resort to UART console from here, because I can’t seem to get any bootmessages to appear.
I’m creating a genuine tumbleweed here But information on this subject is scattered and scarce, so I hope this helps someone, somewhere, sometime. This is what I’ve learned so far about BalenaOS:
it is it’s own distro, built using Yocto. This means that there is no package manager.
it mounts it’s root file system Read Only, so it wouldn’t matter anyway. This is smart, because that means that there no in-flight data for the base system, creating a robust OS when the power is pulled.
it has no initramfs on the boot disk. This means that there is only the kernel (correct me if I’m wrong) at boot.
it has no builtin support for NFS, so this needs to be added somehow.
the tool chain for creating a initramfs is difficult to reproduce: requires ARM, docker and qemu tooling.
I’m trying to build a initramfs now (first time for everyting) that does support NFS mounts, so I can use my existing infrastructure.
After some more days of fiddling I give up. I think I need to recompile balena OS to add mount.nfs to the image, and that’s just too much of a hassle. I’ll stick to SD cards for Balena for now and see how it goes. I can always pull the card if I want to do something else and netboot it.
Secondly, running docker containers from an NFS share is VERY slow (at least for me) so my gut feeling tells me it’s not going to be worth it in the end.
it has no initramfs on the boot disk. This means that there is only the kernel (correct me if I’m wrong) at boot.
The OS does indeed use an initramfs, but it’s built into the kernel. You can check out the scripts that are run as part of the initramfs here. Among other things, these scripts handle filesystem checks, expanding the data partition to fit the disk, and mounting the root filesystem before running init in the persistent rootfs.
Depending on the kernel, this is probably built as a module, and not included in the initramfs. However, you may be able to build a supplemental initramfs image and load it in addition to the built-in image. This has the effect of overlaying the files from the supplemental image onto the built-in one. Using this approach, you could add the nfs kernel module, userspace utilities, and a hook to setup the network connection and mount the share. Check out this script for an example of how to do this using NBD. Note that this is only an example, and it should be only used as a reference to build a production ready netboot environment for your application.
Thanks for reaching out! I understand what you are saying and learning something new: the layering of the initramfs’s is definitely new information, that explains some of the missing pieces. I also found you script got it running and picked it apart. However, this is all very new to me, so I would need some more involved help to get this running (see my dairy above, I know not what I’m doing).
I’d be interested in diving into it regardless, it’s very interesting stuff. Would you (or someone else with comparable knowhow) be willing to give me a helping hand moving forward? I could open a repository with my code if that helps. It’ll be going towards Balena code base if it’s of any use.
Awesome. Let me see if I can adapt the nbd script to a minimal example for nfs. It’ll take some time, but ill report back when I’m either stuck or done.
Just to let you know, I’ve quickly updated the NBD example to pin the version of balenaOS, and fix a compatibility issue with the pinned Debian image in combination with newer balenaOS releases. This really shouldn’t affect your project, but it will ensure that the example continues working in the future.
But information on this subject is scattered and scarce, so I hope this helps someone, somewhere, sometime. This is what I’ve learned so far about BalenaOS: