How to detect restart loops?

Hi!

I was wondering if there is any way to detect restart loops that can happen on BalenaOS? Preferably by polling the supervisor for this from another BalenaOS device using the API. Any information about this?

Hi, do you refer to application containers restart loops or services running in the hostOS?

This would be application containers being restarted by the supervisor :slight_smile:

Thanks, there is no way to detect this through the API at the moment. However, the device diagnostic in the dashboard will tell you whether there are application containers restarting. The way this works is through a connection to the balena engine running in the host, and a balena inspect looking for the RestartCount attribute. You could use this approach for example sshing from another BalenaOS device with the appropriate keys in place.

Alright! So say that the goal here is to reboot the device if an irreparable error is detected (say a restart loop of the application containers). What would be the best way of doing this? Would it be possible to detect these loops from a container on the device itself, and then issue a local reboot command? Is there a restart counter per application container?

Hey, as Alex mentioned in the previous post we don’t expose this information directly, but we have plans to eventually surface it to the balenaCloud API. For the time being what you suggest sounds like a good approach. You can start a container with the io.balena.features.balena-socketlabel which will expose the balena-engine socket to your container and allow you to query for the state of the containers on the host via inspect (to extract the restart count specifically you can try balena-engine inspect -f "{{ .RestartCount }}” [ID] where [ID] is the container ID that can be obtained from the supervisor API from the status endpoint. Let me know if that makes sense and works for you

@nazrhom Would it be possible to enable that particular container always start with io.balena.features.balena-socket, even on first deploy? If so, how?

Hey, you can see how to apply the labels in your docker compose here: https://www.balena.io/docs/reference/supervisor/docker-compose/#labels, and those apply on the first deploy, yes.

Awesome! Thanks alot! :slight_smile:

No worries, let us know if you’re having issues with it and how it goes.