Device stuck on "Update successful, rebooting" status

Raspberry Pi 3 is stuck on the status: “Update successful, rebooting”
Cannot reboot from the webinterface, but reboot command from the Host OS terminal seems to be accepted. Unfortunately, no change in status with reboot from the terminal.

This started after I wanted to update the Host OS to the latest version. This has worked fine on some other Raspberry Pi 3 devices.

Hey @WolfSchouten could you provide the dashboard link and enable support access please?
and support access has been granted.

Thanks, we are checking it out.

Hi, @WolfSchouten, it seems the device is currently offline, has it been switched off? Is it possible the turn it back on?

The device is turned on again and reachable. Status is still the same though :frowning_face:

Expected that, just checking it out now.

Hey @WolfSchouten, the device should be now fixed up and working properly.

What we found, is that some system files were in a corrupt state, unfortunately one of them being balena, hence the supervisor / application couldn’t run. We’ve replaced that binary with the correct one, and the device started up.

As test, run another update on the device, that seems to have gone okay, but after restart I see some other files in corrupt state (checking by the fingerprints shipped with the device). So the device works now, but cannot guarantee what’s going to happen, if files getting corrupted.

This to me suggest that the SD card is in a bad state, and would recommend replacing it, as otherwise it might keep corrupting the system. What make/brand SD card are you using? We strongly recommend more robust cards such as Sandisk Extreme Pro, which hasn’t let us down yet (while a lot of common cards do have such a short life time for IoT application).

What do you think?

That batch has no specific brand of SD cards. Our current batch has Transcend Premium cards. So yes, it could be an SD card issue.

Can you point me to documentation on how to check for corruption in system files remotely? That way I can keep that device up while we ship a new SD card their way.

The device should be okay for the time being, so I think you should have time to replace the card. We haven’t had any experience with Transcend Premium, hence our recommendation of the tried & tested type above.

The integrity checks are done through checking against the fingerprint file shipped with the system. It’s found at /resinos.fingerprint in the host OS. It can be checked by running

md5sum --quiet -c /resinos.fingerprint

Please note, that some checksum fails are expected, as they are overridden at runtime by device-specific versions, thus they will fail the check, this is fine and does mean normal operation. For example on current OS versions this result would be a good outcome:

root@xxxxxxx:~# md5sum --quiet -c /resinos.fingerprint
/etc/hostname: FAILED
/etc/machine-id: FAILED
/home/root/.rnd: FAILED
md5sum: WARNING: 3 computed checksums did NOT match

There’s also checksum on the files in /mnt/boot (the boot partition), which can be checked by

cat /mnt/boot/resinos.fingerprint | sed 's/ \// \/mnt\/boot\//' | md5sum --quiet --check -

This should report a few files changed that do change in the normal course of operation (such as config.txt, cmdline.txt, etc. on the Raspberry Pi for example).

Please note, that these might change, and the fingerprints are used only for general diagnostics, as part of a whole toolset to debug devices misbehaving. We don’t really recommend relying on this in your day-to-day work with devices, and if there’s any checksum failures, don’t recommend jumping to conclusion. We just mention it here to give you some more background info. Overall we are working towards ensuring that you wouldn’t need to do these checks ever, and have a well behaved device without worrying about data integrity.

I’m experiencing a similar issue. The notable difference being that I can not reboot due to a socket issue. I also do not have host OS access because i’m updating the OS from a version that doesn’t support it.

Any suggestions on how to proceed?

@sam - just to close the loop on this, we resolved this issue by manually updating the Supervisor running on your device. The OS version 2.13.6+rev1 did not match the Supervisor version that was installed. OS version 2.13.6+rev2 contains an updated Supervisor, and it is recommended that if you are using version 2.13.6 to ensure all devices have rev2 installed.