Hi all, I’m working on trying to standardize our logging setup with some best practices and would love some input. Currently I tooled up some custom, app level logging in python to ship logs directly to Datadog (using daiquiri in python), but I’m already starting to see the complexity that per app log configurations will cause as new services are introduced. This approach also doesn’t address log management for third party apps which have their own methods.
So in the interest of simplicity, rather than configure logs one by one, I’d like to instead have all balena logs, container, and app logs (first party or third party) send to stdout and use fluentbit or similar to handle the complex work of aggregating, parsing, and shipping. The upside is that it also makes switching between or combing vendors like Datadog, Elastic, Grafana/Loki, Cloudwatch, etc much, much easier. I’ve done some quick tests and easily shipped logs to Datadog and Newrelic and S3 all at the same time.
I’ve been testing:
- FluentBit Seems like a fantastic fit. Well established based on older brother Fluentd and tailored to be small and lightweight which is perfect for embedded.
- Vector Also excellent. Very powerful and fast growing community, but its a bit of a resource hog compared to Fluentbit and the docs are a little confusing. Upside, it handles metrics quite well too.
I’ll document my progress in this thread if anyone else wants to give input.