Balena logging aggregator/shipper

Hi all, I’m working on trying to standardize our logging setup with some best practices and would love some input. Currently I tooled up some custom, app level logging in python to ship logs directly to Datadog (using daiquiri in python), but I’m already starting to see the complexity that per app log configurations will cause as new services are introduced. This approach also doesn’t address log management for third party apps which have their own methods.

So in the interest of simplicity, rather than configure logs one by one, I’d like to instead have all balena logs, container, and app logs (first party or third party) send to stdout and use fluentbit or similar to handle the complex work of aggregating, parsing, and shipping. The upside is that it also makes switching between or combing vendors like Datadog, Elastic, Grafana/Loki, Cloudwatch, etc much, much easier. I’ve done some quick tests and easily shipped logs to Datadog and Newrelic and S3 all at the same time.

I’ve been testing:

  • FluentBit Seems like a fantastic fit. Well established based on older brother Fluentd and tailored to be small and lightweight which is perfect for embedded.
  • Vector Also excellent. Very powerful and fast growing community, but its a bit of a resource hog compared to Fluentbit and the docs are a little confusing. Upside, it handles metrics quite well too.

I’ll document my progress in this thread if anyone else wants to give input.

Example for Fluentbit which simply generates some dummy log data and outputs it to terminal and NewRelic. It plays nicely with variables too, so in Balena I set an NR_API_KEY variable and it just works.

docker-compose.yml

fluent-bit:
    build: fluent-bit
    privileged: true
    restart: always
    network_mode: host
    volumes:
        - 'persistent-data:/persistent-data'
    labels: 
        io.balena.features.balena-socket: '1'
        io.balena.features.journal-logs: '1'
    ports:
        - '2020:2020'

Dockerfile

FROM fluent/fluent-bit:1.8

COPY fluent-bit.conf fluent-bit/etc/fluent-bit.conf

COPY plugins.conf fluent-bit/etc/plugins.conf

# Use ADD to get the remote NewRelic plugin, rename it, and copy it to the plugins folder

ADD https://github.com/newrelic/newrelic-fluent-bit-output/releases/download/v1.7.0/out_newrelic-linux-arm64-1.7.0.so /fluent-bit/plugins/out_newrelic.so

CMD /fluent-bit/bin/fluent-bit -c /fluent-bit/etc/fluent-bit.conf

plugins.conf

[PLUGINS]
    Path    /fluent-bit/plugins/out_newrelic.so

fluent-bit.conf

# More info: https://kevcodez.de/posts/2019-08-10-fluent-bit-docker-logging-driver-elasticsearch/

##########################
# Configure Service
##########################
[SERVICE]
# This is the main configuration block for fluent bit.
# Use http server to get prometheus metrics for fluentbit itelf!
# ie. curl -s http://127.0.0.1:2020/api/v1/metrics/prometheus

    Plugins_File    /fluent-bit/plugins/plugins.conf
    HTTP_Server     On
    # HTTP_Listen     0.0.0.0 # May cause a problem with balena https://www.balena.io/docs/learn/develop/runtime/#using-dns-resolvers-in-your-container
    HTTP_Listen     127.0.0.1
    HTTP_PORT       2020
    log_level       debug

##########################
# Configure Inputs
# https://docs.fluentbit.io/manual/pipeline/inputs
##########################
[INPUT]
    Name        dummy
    Tag         dummy.log


##########################
# Configure Parsers
# https://docs.fluentbit.io/manual/pipeline/parsers
##########################


##########################
# Configure Filters
# https://docs.fluentbit.io/manual/pipeline/filters
##########################


##########################
# Configure Outputs
# https://docs.fluentbit.io/manual/pipeline/outputs
##########################
[OUTPUT]
    Name        stdout
    Match       **

[OUTPUT]
    # Dont forget to install NR's plugin.
    # More info: https://docs.newrelic.com/docs/logs/enable-log-management-new-relic/enable-log-monitoring-new-relic/fluent-bit-plugin-log-forwarding/#fluentbit-plugin
    Name        newrelic
    Match       *
    licenseKey  ${NR_API_KEY}
    endpoint    https://log-api.newrelic.com/log/v1

# [OUTPUT]
#     # No extra plugin required like NewRelic.
#     Name          datadog
#     Match         *
#     Host          http-intake.logs.datadoghq.com
#     TLS           on
#     compress      gzip
#     apikey        ${DD_API_KEY}
#     dd_service    test
#     dd_source     dummy
#     dd_tags       environment:test

One thing I haven’t figured out yet is how to best use dockers logging driver. In theory you could set each service in your docker-compose to use fluentd as a logging driver and have fluentbit automatically capture and handle it.

For example in docker-compose.yml

my-service:
        build: my-service
        privileged: true
        restart: always
        network_mode: host
        logging:
            driver: fluentd
            options:
              tag: my-service

Thanks @barryjump for the context here. If I understand this correctly, you are looking to see if we can use the logging keyword for compose directly with balena-engine.

Looking over our balena-engine code and cli, I think that drop-in usage should be possible without an issue (ref: balena-engine-cli/plugins_logging.md at master · balena-os/balena-engine-cli · GitHub - our base for logging is same as docker). Let us know how it goes with use of Fluentbit.

Regards,
N

Thanks @nitish thats good news. Yeah, thats roughly the goal.

Here’s a simple diagram for example of what I’m hoping to do. Yellow boxes are individual containers.

Thanks @barryjump for sharing the overall view (“big picture”). This looks great and sounds like a great blog post once done!