Hello,
One of our RPI5 got stuck in the “Online (VPN only)” and I am not to sure how to recover from it (without re-flashing it). Rebooting / repowering didn’t help.
# dmesg
[ 1074.621806] EXT4-fs warning (device mmcblk0p6): ext4_dirblock_csum_verify:405: inode #23: comm balena-engine-c: No space for directory leaf checksum. Please run e2fsck -D.
[ 1074.621808] EXT4-fs error (device mmcblk0p6): __ext4_find_entry:1693: inode #23: comm balena-engine-c: checksumming directory block 0
# journalctl -n 30
Jul 22 11:04:39 60fd187 balenad[6149]: time="2024-07-22T11:04:39.122262440Z" level=error msg="containerd did not exit successfully" error="exit status 1" module=libcontainerd
Jul 22 11:04:49 60fd187 extract-balena-ca[6181]: [extract-balena-ca][INFO] The config.json file does not contain custom CA
Jul 22 11:04:49 60fd187 extract-balena-ca[6186]: The config.json file does not contain custom CA
Jul 22 11:04:52 60fd187 balenad[5627]: Parent process exited
Jul 22 11:04:54 60fd187 balenad[6149]: failed to start containerd: timeout waiting for containerd to start
Jul 22 11:04:54 60fd187 systemd[1]: balena.service: Main process exited, code=exited, status=1/FAILURE
Jul 22 11:04:54 60fd187 systemd[1]: balena.service: Failed with result 'exit-code'.
Jul 22 11:04:54 60fd187 systemd[1]: Failed to start Balena Application Container Engine.
Jul 22 11:04:54 60fd187 balena-supervisor[6188]: Cannot connect to the balenaEngine daemon at unix:///var/run/balena-engine.sock. Is the balenaEngine daemon running?
Jul 22 11:04:54 60fd187 balena-supervisor[6196]: Cannot connect to the balenaEngine daemon at unix:///var/run/balena-engine.sock. Is the balenaEngine daemon running?
Jul 22 11:04:54 60fd187 systemd[1]: balena.service: Found left-over process 5673 (exe) in control group while starting unit. Ignoring.
Jul 22 11:04:54 60fd187 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jul 22 11:04:54 60fd187 systemd[1]: balena.service: Found left-over process 5720 (exe) in control group while starting unit. Ignoring.
Jul 22 11:04:54 60fd187 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jul 22 11:04:54 60fd187 systemd[1]: balena.service: Found left-over process 5768 (exe) in control group while starting unit. Ignoring.
Jul 22 11:04:54 60fd187 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jul 22 11:04:54 60fd187 systemd[1]: balena.service: Found left-over process 5814 (exe) in control group while starting unit. Ignoring.
Jul 22 11:04:54 60fd187 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jul 22 11:04:54 60fd187 systemd[1]: balena.service: Found left-over process 5869 (exe) in control group while starting unit. Ignoring.
Jul 22 11:04:54 60fd187 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jul 22 11:04:54 60fd187 systemd[1]: balena.service: Found left-over process 5916 (exe) in control group while starting unit. Ignoring.
Jul 22 11:04:54 60fd187 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jul 22 11:04:54 60fd187 systemd[1]: balena.service: Found left-over process 5964 (exe) in control group while starting unit. Ignoring.
Jul 22 11:04:54 60fd187 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jul 22 11:04:54 60fd187 systemd[1]: balena.service: Found left-over process 6009 (exe) in control group while starting unit. Ignoring.
Jul 22 11:04:54 60fd187 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jul 22 11:04:54 60fd187 systemd[1]: balena.service: Found left-over process 6062 (exe) in control group while starting unit. Ignoring.
Jul 22 11:04:54 60fd187 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jul 22 11:04:54 60fd187 systemd[1]: balena.service: Found left-over process 6109 (exe) in control group while starting unit. Ignoring.
Jul 22 11:04:54 60fd187 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jul 22 11:04:54 60fd187 systemd[1]: balena.service: Found left-over process 6155 (exe) in control group while starting unit. Ignoring.
Jul 22 11:04:54 60fd187 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jul 22 11:04:54 60fd187 balena-supervisor[6204]: activating
Jul 22 11:04:54 60fd187 systemd[1]: balena-supervisor.service: Control process exited, code=exited, status=3/NOTIMPLEMENTED
Jul 22 11:04:54 60fd187 systemd[1]: balena-supervisor.service: Failed with result 'exit-code'.
Jul 22 11:04:54 60fd187 systemd[1]: Failed to start Balena supervisor.
Jul 22 11:04:54 60fd187 balenad[6205]: time="2024-07-22T11:04:54.269628141Z" level=info msg="Starting up"
Jul 22 11:04:54 60fd187 balenad[6205]: time="2024-07-22T11:04:54.269684141Z" level=warning msg="Running experimental build"
Jul 22 11:04:54 60fd187 balenad[6205]: time="2024-07-22T11:04:54.271225985Z" level=info msg="libcontainerd: started new balena-engine-containerd process" pid=6216
Jul 22 11:04:54 60fd187 balenad[6216]: time="2024-07-22T11:04:54Z" level=warning msg="containerd config version `1` has been deprecated and will be removed in containerd v2.0, please switch t>
Jul 22 11:04:54 60fd187 balenad[6216]: containerd: creating temp mount location: mkdir /var/lib/docker/containerd/daemon/tmpmounts: bad message
Jul 22 11:04:54 60fd187 kernel: EXT4-fs warning (device mmcblk0p6): ext4_dirblock_csum_verify:405: inode #23: comm balena-engine-c: No space for directory leaf checksum. Please run e2fsck -D.
Jul 22 11:04:54 60fd187 kernel: EXT4-fs error (device mmcblk0p6): __ext4_find_entry:1693: inode #23: comm balena-engine-c: checksumming directory block 0
Jul 22 11:04:54 60fd187 kernel: EXT4-fs warning (device mmcblk0p6): ext4_dirblock_csum_verify:405: inode #23: comm balena-engine-c: No space for directory leaf checksum. Please run e2fsck -D.
Jul 22 11:04:54 60fd187 kernel: EXT4-fs error (device mmcblk0p6): __ext4_find_entry:1693: inode #23: comm balena-engine-c: checksumming directory block 0
Jul 22 11:04:54 60fd187 kernel: EXT4-fs warning (device mmcblk0p6): ext4_dirblock_csum_verify:405: inode #23: comm balena-engine-c: No space for directory leaf checksum. Please run e2fsck -D.
Jul 22 11:04:54 60fd187 kernel: EXT4-fs error (device mmcblk0p6): __ext4_find_entry:1693: inode #23: comm balena-engine-c: checksumming directory block 0
# e2fsck -nf /dev/mmcblk0p6
e2fsck 1.46.5 (30-Dec-2021)
Warning! /dev/mmcblk0p6 is mounted.
Warning: skipping journal recovery because doing a read-only filesystem check.
Pass 1: Checking inodes, blocks, and sizes
Inode 102 has an invalid extent
(logical block 0, invalid physical block 35184372111003, len 1)
Clear? no
Inode 102 is a zero-length directory. Clear? no
Inode 102, i_size is 4096, should be 0. Fix? no
Inode 102, i_blocks is 8200, should be 0. Fix? no
Inode 102 passes checks, but checksum does not match inode. Fix? no
Inode 103, i_blocks is 8, should be 8200. Fix? no
Inode 103 passes checks, but checksum does not match inode. Fix? no
Inode 104 has an invalid extent
(logical block 0, invalid physical block 34359744157, len 16385)
Clear? no
Inode 104, i_blocks is 520, should be 0. Fix? no
Inode 104 passes checks, but checksum does not match inode. Fix? no
Inode 109 has an invalid extent
(logical block 0, invalid physical block 24885040513025, len 65537)
Clear? no
Inode 109 extent tree (at level 1) could be shorter. Optimize? no
Inode 109, i_blocks is 1073741832, should be 0. Fix? no
Inode 109 passes checks, but checksum does not match inode. Fix? no
Inode 110 is in use, but has dtime set. Fix? no
Inode 110, i_blocks is 264, should be 8. Fix? no
Inode 110 passes checks, but checksum does not match inode. Fix? no
Inode 111 is in use, but has dtime set. Fix? no
Inode 111 has a bad extended attribute block 2048. Clear? no
Extended attribute block 2048 has h_blocks > 1. Clear? no
Inode 111, i_size is 36028797018964039, should be 4096. Fix? no
Inode 111, i_blocks is 4398046511112, should be 16. Fix? no
Inode 111 passes checks, but checksum does not match inode. Fix? no
Inode 112, i_blocks is 1099512676360, should be 8. Fix? no
Inode 112 passes checks, but checksum does not match inode. Fix? no
Inode 2433 seems to contain garbage. Clear? no
Inode 2433 has a extra size (544) which is invalid
Fix? no
Inode 2433, i_size is 39832, should be 4398046552064. Fix? no
Inode 2433 passes checks, but checksum does not match inode. Fix? no
Inode 2434 seems to contain garbage. Clear? no
Inode 2434 is in use, but has dtime set. Fix? no
Inode 2434 has corrupt extent header. Clear inode? no
Inode 2434, i_blocks is 70368744177744, should be 8. Fix? no
Inode 2434 passes checks, but checksum does not match inode. Fix? no
Inode 2435 seems to contain garbage. Clear? no
Inode 2435 passes checks, but checksum does not match inode. Fix? no
Inode 2436 seems to contain garbage. Clear? no
Inode 2436 passes checks, but checksum does not match inode. Fix? no
Inode 2437 seems to contain garbage. Clear? no
Inode 2437 has corrupt extent header. Clear inode? no
Inode 2437, i_size is 1125899906907152, should be 0. Fix? no
Inode 2437, i_blocks is 128, should be 0. Fix? no
Inode 2437 passes checks, but checksum does not match inode. Fix? no
Inode 2438 seems to contain garbage. Clear? no
Inode 2438, i_size is 17592188280464, should be 139264. Fix? no
Inode 2438, i_blocks is 400, should be 272. Fix? no
Inode 2438 passes checks, but checksum does not match inode. Fix? no
Inode 2439 seems to contain garbage. Clear? no
Inode 2439 is in use, but has dtime set. Fix? no
Inode 2439 has a extra size (32800) which is invalid
Fix? no
Inode 2439 has a bad extended attribute block 4104. Clear? no
Extended attribute block 4104 has h_blocks > 1. Clear? no
Extended attribute block 4104 is corrupt (invalid name). Clear? no
Extended attribute block 4104 is corrupt (allocation collision). Clear? no
Inode 2439 has corrupt extent header. Clear inode? no
Inode 2439, i_blocks is 1610612865, should be 8. Fix? no
Inode 2439 passes checks, but checksum does not match inode. Fix? no
Inode 2440 seems to contain garbage. Clear? no
Inode 2440 has corrupt extent header. Clear inode? no
Inode 2440, i_size is 17592186235856, should be 0. Fix? no
Inode 2440, i_blocks is 67125368, should be 0. Fix? no
Inode 2440 passes checks, but checksum does not match inode. Fix? no
Inode 2441 seems to contain garbage. Clear? no
Inode 2441 is in use, but has dtime set. Fix? no
Inode 2441 has corrupt extent header. Clear inode? no
Inode 2441, i_blocks is 144, should be 0. Fix? no
Inode 2441 passes checks, but checksum does not match inode. Fix? no
Inode 2442 seems to contain garbage. Clear? no
Inode 2442 is in use, but has dtime set. Fix? no
Inode 2442 has corrupt extent header. Clear inode? no
Inode 2442, i_blocks is 536871168, should be 0. Fix? no
Inode 2442 passes checks, but checksum does not match inode. Fix? no
Inode 2443 seems to contain garbage. Clear? no
Inode 2443 is in use, but has dtime set. Fix? no
Inode 2443 has a bad extended attribute block 524288. Clear? no
Extended attribute block 524288 has h_blocks > 1. Clear? no
Inode 2443 has illegal extended attribute value inode 4294967295.
Clear? no
Inode 2443 has illegal extended attribute value inode 4294967295.
Clear? no
Inode 2443 has illegal extended attribute value inode 4294967295.
Clear? no
Inode 2443 has illegal extended attribute value inode 4294967295.
Clear? no
Inode 2443 has illegal extended attribute value inode 4294967295.
Clear? no
Inode 2443 has illegal extended attribute value inode 4294967295.
Clear? no
Inode 2443 has illegal extended attribute value inode 4294967295.
Clear? no
Inode 2443 has corrupt extent header. Clear inode? no
Inode 2443, i_blocks is 201, should be 18446744073709537288. Fix? no
Inode 2443 passes checks, but checksum does not match inode. Fix? no
Inode 2444 seems to contain garbage. Clear? no
Inode 2444 has a extra size (1056) which is invalid
Fix? no
Inode 2444 has corrupt extent header. Clear inode? no
Inode 2444, i_blocks is 272, should be 0. Fix? no
Inode 2444 passes checks, but checksum does not match inode. Fix? no
Inode 2445 seems to contain garbage. Clear? no
Inode 2445 passes checks, but checksum does not match inode. Fix? no
Inode 2446 seems to contain garbage. Clear? no
Inode 2446 has an invalid extent
(logical block 0, invalid physical block 2154759552, len 32)
Clear? no
Inode 2446, i_blocks is 4194560, should be 0. Fix? no
Inode 2446 passes checks, but checksum does not match inode. Fix? no
Inode 2447 seems to contain garbage. Clear? no
Inode 2447 has corrupt extent header. Clear inode? no
Inode 2447, i_blocks is 176, should be 0. Fix? no
Inode 2447 passes checks, but checksum does not match inode. Fix? no
Inode 2448 seems to contain garbage. Clear? no
Inode 2448 is in use, but has dtime set. Fix? no
Inode 2448 has a bad extended attribute block 1048576. Clear? no
Extended attribute block 1048576 has h_blocks > 1. Clear? no
Inode 2448 has illegal extended attribute value inode 4294967295.
Clear? no
Inode 2448 has illegal extended attribute value inode 4294967295.
Clear? no
Inode 2448 has illegal extended attribute value inode 4294967295.
Clear? no
Inode 2448 has illegal extended attribute value inode 4294967295.
Clear? no
Inode 2448 has illegal extended attribute value inode 4294967295.
Clear? no
Inode 2448 has illegal extended attribute value inode 4294967295.
Clear? no
Inode 2448 extent tree could be more shallow (64; could be <= 4)
Fix? no
Inode 2448 has an invalid extent
(logical block 0, invalid physical block 5222680232995, len 35)
Clear? no
Inode 2448 extent tree (at level 1) could be shorter. Optimize? no
Inode 2448, i_blocks is 35186519572504, should be 18446744073709539336. Fix? no
Inode 2448 passes checks, but checksum does not match inode. Fix? no
Inode 2449 seems to contain garbage. Clear? no
Inode 2449 has corrupt extent header. Clear inode? no
Inode 2449, i_blocks is 200, should be 0. Fix? no
Inode 2449 passes checks, but checksum does not match inode. Fix? no
Inode 2450 seems to contain garbage. Clear? no
Inode 2450 is in use, but has dtime set. Fix? no
Inode 2450 has corrupt extent header. Clear inode? no
Inode 2450, i_size is 2392537302116728, should be 0. Fix? no
Inode 2450, i_blocks is 152, should be 0. Fix? no
Inode 2450 passes checks, but checksum does not match inode. Fix? no
Inode 2451 seems to contain garbage. Clear? no
Inode 2451 passes checks, but checksum does not match inode. Fix? no
Inode 2452 seems to contain garbage. Clear? no
Inode 2452 is in use, but has dtime set. Fix? no
Inode 2452 passes checks, but checksum does not match inode. Fix? no
Inode 2453 seems to contain garbage. Clear? no
Inode 2453 has a bad extended attribute block 32768. Clear? no
Extended attribute block 32768 has h_blocks > 1. Clear? no
Inode 2453 has illegal extended attribute value inode 32768.
Clear? no
Extended attribute block 32768 is corrupt (invalid name). Clear? no
Extended attribute block 32768 is corrupt (allocation collision). Clear? no
Inode 2453 has an invalid extent
(logical block 0, invalid physical block 4398053788232, len 13)
Clear? no
Inode 2453, i_blocks is 4168, should be 18446744073709549576. Fix? no
Inode 2453 passes checks, but checksum does not match inode. Fix? no
Inode 2454 seems to contain garbage. Clear? no
Inode 2454, i_blocks is 8589967520, should be 160. Fix? no
Inode 2454 passes checks, but checksum does not match inode. Fix? no
Inode 2455 seems to contain garbage. Clear? no
Inode 2455 has an invalid extent
(logical block 134217728, invalid physical block 206165707345, len 1)
Clear? no
Inode 2455, i_blocks is 8, should be 0. Fix? no
Inode 2455 passes checks, but checksum does not match inode. Fix? no
Inode 2456 seems to contain garbage. Clear? no
Inode 2456 extent tree could be more shallow (8192; could be <= 4)
Fix? no
Inode 2456 has an invalid extent
(logical block 0, invalid physical block 11347303596040, len 8)
Clear? no
Inode 2456 extent tree (at level 1) could be shorter. Optimize? no
Inode 2456, i_blocks is 16777282, should be 0. Fix? no
Inode 2456 passes checks, but checksum does not match inode. Fix? no
Inode 2457 seems to contain garbage. Clear? no
Inode 2457 is in use, but has dtime set. Fix? no
Inode 2457 has corrupt extent header. Clear inode? no
Inode 2457, i_blocks is 131080, should be 0. Fix? no
Inode 2457 passes checks, but checksum does not match inode. Fix? no
Inode 2458 seems to contain garbage. Clear? no
Inode 2458 is in use, but has dtime set. Fix? no
Inode 2458 has a bad extended attribute block 49152. Clear? no
Extended attribute block 49152 has h_blocks > 1. Clear? no
Extended attribute block 49152 is corrupt (invalid name). Clear? no
Extended attribute block 49152 is corrupt (allocation collision). Clear? no
Inode 2458 has corrupt extent header. Clear inode? no
Inode 2458, i_size is 1125899906907480, should be 0. Fix? no
Inode 2458, i_blocks is 1073742208, should be 8. Fix? no
Inode 2458 passes checks, but checksum does not match inode. Fix? no
Inode 2459 seems to contain garbage. Clear? no
Inode 2459 has corrupt extent header. Clear inode? no
Inode 2459, i_blocks is 176, should be 0. Fix? no
Inode 2459 passes checks, but checksum does not match inode. Fix? no
Inode 2460 seems to contain garbage. Clear? no
Inode 2460 extent tree could be more shallow (64; could be <= 4)
Fix? no
Inode 2460 has an invalid extent
(logical block 0, invalid physical block 13211336179757, len 393517)
Clear? no
Inode 2460 extent tree (at level 1) could be shorter. Optimize? no
Inode 2460, i_blocks is 360, should be 0. Fix? no
Inode 2460 passes checks, but checksum does not match inode. Fix? no
Inode 2461 seems to contain garbage. Clear? no
Inode 2461 has corrupt extent header. Clear inode? no
Inode 2461, i_blocks is 100663576, should be 0. Fix? no
Inode 2461 passes checks, but checksum does not match inode. Fix? no
Inode 2462 seems to contain garbage. Clear? no
Inode 2462 has an invalid extent
(logical block 0, invalid physical block 15695896, len 1025)
Clear? no
Inode 2462, i_blocks is 4104, should be 0. Fix? no
Inode 2462 passes checks, but checksum does not match inode. Fix? no
Inode 2463 seems to contain garbage. Clear? no
Inode 2463 is in use, but has dtime set. Fix? no
Inode 2463, i_blocks is 140737622573072, should be 0. Fix? no
Inode 2463 passes checks, but checksum does not match inode. Fix? no
Inode 2464 seems to contain garbage. Clear? no
Inode 2464 is in use, but has dtime set. Fix? no
Inode 2464 has corrupt extent header. Clear inode? no
Inode 2464, i_blocks is 184, should be 0. Fix? no
Inode 2464 passes checks, but checksum does not match inode. Fix? no
Inode 2465 seems to contain garbage. Clear? no
Inode 2465 is in use, but has dtime set. Fix? no
Inode 2465 has corrupt extent header. Clear inode? no
Inode 2465, i_blocks is 40, should be 0. Fix? no
Inode 2465 passes checks, but checksum does not match inode. Fix? no
Inode 2466 seems to contain garbage. Clear? no
Inode 2466 is in use, but has dtime set. Fix? no
Inode 2466 has a bad extended attribute block 16. Clear? no
Extended attribute block 16 has h_blocks > 1. Clear? no
Inode 2466 has illegal extended attribute value inode 4096016.
Clear? no
Inode 2466 has corrupt extent header. Clear inode? no
Inode 2466, i_blocks is 1108101562504, should be 2944110754568. Fix? no
Inode 2466 passes checks, but checksum does not match inode. Fix? no
Inode 2467 seems to contain garbage. Clear? no
Inode 2467 has corrupt extent header. Clear inode? no
Inode 2467, i_blocks is 153, should be 0. Fix? no
Inode 2467 passes checks, but checksum does not match inode. Fix? no
Inode 2468 seems to contain garbage. Clear? no
Inode 2468 has corrupt extent header. Clear inode? no
Inode 2468, i_blocks is 16777320, should be 0. Fix? no
Inode 2468 passes checks, but checksum does not match inode. Fix? no
Inode 2469 seems to contain garbage. Clear? no
Inode 2469 passes checks, but checksum does not match inode. Fix? no
Inode 2470 seems to contain garbage. Clear? no
Inode 2470 has a extra size (160) which is invalid
Fix? no
Inode 2470 has illegal block(s). Clear? no
Illegal block #0 (1949527912) in inode 2470. IGNORED.
Illegal block #1 (1701667182) in inode 2470. IGNORED.
Inode 2470 is too big. Truncate? no
Block #4 (524288) causes symlink to be too big. IGNORED.
Illegal block #5 (8405024) in inode 2470. IGNORED.
Block #6 (256) causes symlink to be too big. IGNORED.
Illegal block #8 (285212672) in inode 2470. IGNORED.
Block #11 (524288) causes symlink to be too big. IGNORED.
Inode 2470 block 524304 conflicts with critical metadata, skipping block checks.
Too many illegal blocks in inode 2470.
Clear inode? no
Suppress messages? no
Illegal triple indirect block (33554432) in inode 2470. IGNORED.
Error while iterating over blocks in inode 2470: Illegal triply indirect block found
resin-data: ********** WARNING: Filesystem still has errors **********
e2fsck: aborted
resin-data: ********** WARNING: Filesystem still has errors **********
# df -lhi /dev/mmcblk0p6
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mmcblk0p6 2.7M 55K 2.6M 3% /mnt/data
# df -lh /dev/mmcblk0p6
Filesystem Size Used Avail Use% Mounted on
/dev/mmcblk0p6 28G 1.4G 25G 6% /mnt/data