unable to boot past initramfs - overlay test fails on new board

Hi all

I’m currently porting balenaOS to a new device and so far everything builds properly. U-Boot worked with very minor changes but I am having issues getting through the kernel boot.

The specific issue I have occurs after the switch_root from the finish script here. After that point I believe the Mobyinit is what should be run as per this doc, but there is almost immediately a kernel panic. The error shown is:

[init][INFO] Switching root to '/rootfs'... [ 671.690614] EXT4-fs (mmcblk2p2): re-mounted. Opts: (null) ERRO[0000] mkdir /balena/overlay2/check-overlayfs-support679356590: no space left on device storage-driver=overlay2 1970/01/01 00:11:11 error loading layer store:error initializing graphdriver: driver not supported

I think the issue is with the re-mount of mmcblk2p2 (resin-rootA) while resin-state is not re-mounted at that point. I dropped to the initramfs shell using the shell kernel bootarg to confirm that partitions were mounted by the initrdscripts, and resin-rootA and resin-state were both mounted correctly. Before the finish script is run, the output of mount is:

rootfs on / type rootfs (rw) proc on /proc type proc (rw,relatime) sysfs on /sys type sysfs (rw,relatime) devtmpfs on /dev type devtmpfs (rw,relatime,size=1397848k,nr_inodes=349462,mode=755) tmpfs on /rootfs/tmp type tmpfs (rw,nosuid,nodev,mode=755) /dev/disk/by-state/resin-rootA on /rootfs type ext4 (ro,relatime,data=ordered) /dev/disk/by-state/resin-state on /rootfs/mnt/state type ext4 (rw,relatime,data=ordered) /dev/disk/by-state/resin-state on /rootfs/etc/machine-id type ext4 (rw,relatime,data=ordered)

I do not see anything here that is particularly unusual - but the kernel still remounts resin-rootA after the finish script (specifically the switch_root) is executed.

More specifically, the command that seems to fail is the SupportsOverlay test in balena-engine here: https://github.com/balena-os/balena-engine/blob/master/daemon/graphdriver/overlayutils/overlayutils.go#L39. Creating the temp directory doesn’t seem to be possible as it is doing so on the read-only filesystem resin-rootA (which is remounted after switch_root). It isn’t possible to debug past this due to a kernel panic directly after the error loading layer store:error initializing graphdriver message.

Is this something that could be modified (in this case, simply removing the creation of the tempdir?) to fix the issue?

For more info - I am running the kernel passing root=LABEL=resin-rootA as an argument instead of UUID. Using UUID caused other issues with creating the by-state symlinks (not sure why, did not investigate this too much). This caused the rootfs shell script to not find the resin-rootA partition by its old UUID or the /dev/disk/by-state/active link (since it didn’t exist)

Any help is much appreciated

Hi, are you using overlay2 such as in https://github.com/balena-os/balena-jetson/blob/master/layers/meta-balena-jetson/conf/samples/local.conf.sample#L5 ?

I am using overlay2 - the kernel version is 4.14 so attempting to use aufs threw errors during compile time. For reference the kernel version is 4.14.98 (sumo) so I believe aufs is not supported.

I have actually found a way around my previous error - I had to add an extra bit of configuration in my board’s meta-balena-(board) layer to increase the size of the image rootfs. The specific lines were



Obviously 4000000 is a lot more than specified in meta-balena but I went overboard to make sure it would work. This was in the resin-image.inc file.

edit: Forgot I had changed some files in the mount section which was causing issues. That’s fixed now and BalenaOS seems to be working fine

glad to hear you were able to make it work. Please let us know if we can help with anything else.