Suggestions for acting on container healthcheck

[jotham]

I am interesting in interacting with other services in based on balena container healthchecks. E.g. creating an alarm in opsgenie if a service is unhealthy

I have seen some healthcheck examples

Demo of Docker HEALTHCHECK for a service

Hey, I’ve put together a little demo showing of the Docker HEALTHCHECK functionality, which can help keep your services running fine. Docker can periodically run a command in your container and depending on the command’s results restart it. In this demo, a small web service is implemented, which has a state endpoint. Ona a given trigger, it will go into a bad state, and starts to return 503 errors on that endpoint. The healthcheck picks that up, and restarts the container. Of course this is …

And I’m wondering if anyone has experience for inspecting the health check through a balena interface (e.g. cli, api)

E.g. if the healthcheck status for each device service is available via the api, we could poll the api and create an alarm in our system for an unhealthy service

We could also implement this in the healthcheck itself. So far, that looks like the only option to me, but it’s not my first choice.

Any suggestions?
Thanks!

[jotham]

i think we will just run a sensu-client container in our balena deloyment and mount the docker socket into that container

the docker sensu plugins can easily be extended to alarm on healthcheck failures

and it fits with the stack we already have

in the future we will consider something like prometheus/datadog

[sradevski]

Hi
there are several sample projects that explain how to set up monitoring on your containers

Monitoring your balena devices with Datadog

Fleet-wide Machine Metrics Monitoring in 20mins

Monitoring linux stats with Prometheus.io

Maybe you should considered adapting one of them for your purposes.
Using the docker healthcheck will be rather problematic as this information lives in the host OS where it is not accessible to user applications.

[pcarranzav]

To add to Stevche’s message; something else to keep in mind is that in balenaOS devices, one of balenaEngine’s features is that any container that is marked as unhealthy will be restarted. So services won’t stay in an unhealthy state for long - if you’re worried about them being stuck in a restart cycle (i.e. if every time they restart they get unhealthy), it might make sense to add something like Sentry to notify you when something goes wrong.

But yeah, mounting the docker socket to use a standard container monitoring tool is a great solution too - just keep in mind mounting this socket gives the container the equivalent of full system access, so it’s something to treat with care.

[jotham]

Many thanks for the feedback and info!