Device hung in "Stopping" state

Hi there, a device hung in “Stopping” state.

https://dashboard.resin.io/apps/273906/devices/468569

Could you please see if you can un-brick it from your end or advise if there is anything we can do from here…

Cheers!

Hey @ab1, I’ve fixed the device for you for now, but if you use the ‘ResinOS Update’ option under the device’s actions page to update to 1.26.0 then that version contains a fix that will prevent the issue occurring in the future

Awesome, thank you. I’ll get all the device upgraded.

Looks like a device upgrade to v1.26 failed with “ResinOS: Update failed.”

It’s currently online, but not responding.

Are you able to kick in back to life from you end or will it require a rebuild?

Hey, we should be able to get it back, the not responding means that the supervisor was stopped as it is during the course of the update, but something didn’t work out along the way (likely some checks didn’t finish fine, not “failure” during the update). There are logs generated by the update to see what’s going on (why it did not succeed).

But it seems like the device does not exists anymore?
Can you tell us what was the original resinOS version?

I think the original was 1.24.x

This is the device URL:
https://dashboard.resin.io/apps/273906/devices/468569/

Seems to be in the dashboard at least, but I am not certain whether it is contactable…

Hey, found it. :tada: Rebooted the device and should be back in service now.

:pencil2: The issue during the update seems to be indeed in the validation step. The validation of /etc/hostname has failed (it has a different fingerprint than stored), and thus the update bailed out as expected. That file should really be whitelisted, so this was a bug. Just filing an issue for it and updating resinHUP, so the update can be applied properly.

Great stuff, should we try upgrading again now?

Not yet, as the change hasn’t been merged yet into the updater, but soon!

Hi guys, another device with the same problem:
apps/273906/devices/468569

Are you able to recover it?

Cheers!

It should be OK now. Sorry about that.

Awesome, thank you! Did you manage to fix the updater to exclude the problematic files so I can try updating to 1.26?

:speech_balloon:the update is done, but it’s not yet deployed, sorry, should be good soon!

Apologies, could you please see if you can recover apps/273905/devices/476891 as well? Also stuck “Stopping”…

Thank you!

1 Like

Hi guys, can you please see if you can recover 273905/devices/589705 as well?

Probably the same issue as above!

Cheers!

Is this device in a bad network? It seems it’s struggling to get the update but the connection is reset

Thank you, it does seem to be struggling, but at least the supervisor link is up.

I’ll see what I can do from my end about the network quality…

– ab1

Hey, been checking it out, and seems like the device experiences some rough networks indeed, something like ~10% packet loss. The deltas are using rsync, set up with 5 minutes timeout (timing out if not receiving data for that long), and that seems to be some of the error messages (that’s what’s code 30 is).

Trying to see if we can do anything from this end. Was this device having issues before, or just now?

Hi guys, are you able to recover another one for me please from the “Stopping” state:

apps/273905/devices/613678

Cheers!