I am using balena on different Raspberry Pi’s 3, with the OS version : balenaOS 2.95.3+rev1.
I am having lots of issues with corrupted SD cards when I unplug my boards, so I decided to try the “shutdown” option available on the balena dashboard.
This correctly shuts down the boards, but often the device doesn’t boot anymore after that (even after unplugging and re-plugging the power). Sometimes it does, but when it doesn’t the only solution seem to be to erase the SD card and re-flash a new image.
I have plugged the non-booting SD cards into my computer and the resin-boot/ folder seems to be ok…
I have no clue what is happening, but this is very problematic for my application.
Any idea or suggestion ?
Thanks in advance for your help !
Thomas
PS : I found this topic more or less related, but it didn’t help me out
[UPDATE] So I just discovered that I had a python container with a script looping through an event loop, but this script used 100% of the CPU (I added a short time.sleep() to calm it down, and it dropped back to 8%).
Can this be related ?
Hi, is your application performing disk writes? A side effect of the continuous loop is that your application will be starving systemd which is supposed to update the system watchdog. So what will happen is that the watchdog not being refreshed will reboot your board unexpectedly, and if the application is writing to the SDcard this will be interruped and the filesystem will be corrupted.
When you say the device is not booting, do you mean the hostOS is not booting or the application is not launching? Did you try connecting a serial cable and enabling the UART console to see whether the hostOS is booting? I would expect both boot and root partitions to be fine and only the data partition to be corrupted. However, if the corruption is not automatically recoverable, the fsck check on boot will fail and the system will not be able to boot.
Hi,
Thanks a lot for your help.
My application does perform some disk writes but not in my main loop, only on user input, which is almost never the case.
I haven’t seen any unexpected reboots and the code is working as expected. The only issues were really just related to switching off the Raspberry Pi.
Regarding the booting process, I really don’t know what runs or what doesn’t once the problem occurs. I have a display connected to the device and the initial balena logo isn’t showing. It just remains dark and nothing happens. The green LED flashes 3 times when I plug the power in and that is it. I have tried plugging the SD into another Raspberry Pi, and it doesn’t seem to boot either.
But this morning I have implemented a quick patch to slow my main loop down and bring the CPU usage back to ~8% (adding a simple sleep) and have shutdown the Raspberry Pi more than 25 times in different ways without errors so far… it is still a too small sample to conclude that it solved the issue, but it starts to seem like it was related.
Just to give the latest update : reducing the CPU usage seems to have done the trick. I haven’t had any issue since.
Still strange that a full CPU crashes a safe shutdown though.
