Suggestions for acting on container healthcheck

I am interesting in interacting with other services in based on balena container healthchecks. E.g. creating an alarm in opsgenie if a service is unhealthy

I have seen some healthcheck examples

And I’m wondering if anyone has experience for inspecting the health check through a balena interface (e.g. cli, api)

E.g. if the healthcheck status for each device service is available via the api, we could poll the api and create an alarm in our system for an unhealthy service

We could also implement this in the healthcheck itself. So far, that looks like the only option to me, but it’s not my first choice.

Any suggestions?

i think we will just run a sensu-client container in our balena deloyment and mount the docker socket into that container

the docker sensu plugins can easily be extended to alarm on healthcheck failures

and it fits with the stack we already have

in the future we will consider something like prometheus/datadog

there are several sample projects that explain how to set up monitoring on your containers

Maybe you should considered adapting one of them for your purposes.
Using the docker healthcheck will be rather problematic as this information lives in the host OS where it is not accessible to user applications.

To add to Stevche’s message; something else to keep in mind is that in balenaOS devices, one of balenaEngine’s features is that any container that is marked as unhealthy will be restarted. So services won’t stay in an unhealthy state for long - if you’re worried about them being stuck in a restart cycle (i.e. if every time they restart they get unhealthy), it might make sense to add something like Sentry to notify you when something goes wrong.

But yeah, mounting the docker socket to use a standard container monitoring tool is a great solution too - just keep in mind mounting this socket gives the container the equivalent of full system access, so it’s something to treat with care.

Many thanks for the feedback and info!