Dirty bit set on the balenaOS boot partition causing a device not to boot?

Product support
1

Hello!

We are trying to reproduce an issue that has caused one of our devices to no longer boot. The symptoms were that the device did not boot, nothing visible on screen and also there was no more connectivity so we weren’t able to connect to it. Our first guess was a corrupted eMMC (the device is a Raspberry Pi CM4).

After mounting the disk on a different machine through an IO board we were able to establish that all of the Linux partitions were still healthy (using fsck). However, the FAT32 boot partition somehow had the dirty bit set. (As you can see in my terminal interaction that I have attached at the end of this post).

After manually clearing the dirty bit of the boot partition the device would boot up again when installed into the original hardware without any other changes. The odd thing is that when we now manually set the dirty bit to true again, and reboot, we are no longer able to reproduce the original issue.

Our assumption is that this was caused by an unclean poweroff of the device during a balenaOS update? (However, this is an assumption as we are not 100% sure we did in fact trigger a balenaOS update on this device.) We have tried to reproduce this by powering down the device in various stages of a balenaOS update but were unable to reproduce.

My question; is there any feasible scenario thinkable that cause a balenaOS device to get into this state? (Dirty bit on the boot partition set, causing it to not boot, but once manually cleared it does boot.)

Some other potentially relevant info:

  • Device is a Raspberry Pi CM4 in a custom host board
  • balenaOS version is 4.0.16
  • Command to manually set the dirty flag: echo -ne ‘\x01’ | dd of=/dev/mmcblk0p1 bs=1 seek=65 count=1 conv=notrunc

$ sudo fsck /dev/sda2
fsck from util-linux 2.39.3
e2fsck 1.47.0 (5-Feb-2023)
resin-rootA: recovering journal
resin-rootA: clean, 6435/40960 files, 285504/327680 blocks
$ sudo fsck /dev/sda3
fsck from util-linux 2.39.3
e2fsck 1.47.0 (5-Feb-2023)
resin-rootB: recovering journal
resin-rootB: clean, 6633/40960 files, 267372/327680 blocks
$ sudo fsck /dev/sda5
fsck from util-linux 2.39.3
e2fsck 1.47.0 (5-Feb-2023)
resin-state: recovering journal
resin-state: clean, 82/5112 files, 2716/20480 blocks
$ sudo fsck /dev/sda6
fsck from util-linux 2.39.3
e2fsck 1.47.0 (5-Feb-2023)
resin-data: recovering journal
resin-data: clean, 71808/2801664 files, 996614/7451648 blocks
$ fsck.vfat /dev/sda1
fsck.fat 4.2 (2021-01-31)
open: Permission denied
$ sudo fsck.vfat /dev/sda1
fsck.fat 4.2 (2021-01-31)
There are differences between boot sector and its backup.
This is mostly harmless. Differences: (offset:original/backup)
65:01/00

  1. Copy original to backup
  2. Copy backup to original
  3. No action
    [123?q]? 3
    Dirty bit is set. Fs was not properly unmounted and some data may be corrupt.
  4. Remove dirty bit
  5. No action
    [12?q]?
2

Hello @bartvanarnhem first of all thanks for the detailed message.

Let me check internally with the balenaOS team.

Did this only happen to 1 device in all your fleet?

3

Thank you @mpous for following up! This is indeed just a single device for which we have confirmed this to have happened.

4

@bartvanarnhem please let us know if this happen to another device. We might need to know how to reproduce this issue. So if you have another device failing, we can try to debug.

Could you please contact us if that happens again?

5

@mpous definitely! I will give a heads up if / when we get another case with similar symptoms.