Relaying host and container logs & console to logstash or logagent

Product support balenaOS
1

Hello everyone - can anyone help me get logs and stdout & stderr output from containers running under BalenaOS into a log shipper like logstash or logagent. I am using Sematext to collect and analyse logs from my application but would also like to gather information about the containers and host that are running these applications.

As a start it would be great to gather everything that is printed by the applications in the containers in the Balena Dashboard logs window. This output tends to show everything I need right now. I have been investigating with the balena logs command but do not know how to install something that would push all logs out to a remote collection tool.

This will be used on a production system to monitor it for reliability and generate alerts when bad things happen. Sematext have a logagent docker that can ship the logs remotely, but how to get the logs to it?

4

Hey there @Charry2014

I think the best thing for you to start looking into for this would be receiving the Journald logs from the supervisor API: https://www.balena.io/docs/reference/supervisor/supervisor-api/#journald-logs

We also have an example here that uses Loggly, but you may be able to modify it for your use case: https://github.com/balena-io-examples/device-cloud-logging

Let us know if you need any further information!

6

Hi chrisys - many thanks for the tips. I did some playing and have understood what is going on here.

Added in docker-compose for the services that need to log:

labels:
  io.balena.features.supervisor-api: '1'
  io.balena.features.balena-api: '1'
  io.balena.features.balena-socket: '1'

Then:

curl -X POST -H "Content-Type: application/json" --data '{"follow":true,"all":true}' "$BALENA_SUPERVISOR_ADDRESS/v2/journal-logs?apikey=$BALENA_SUPERVISOR_API_KEY"

works a little like tail on the logs that are generated. So far so good. Thank you.

1 Like