Observability solutions

Project help
1

Hello,

What are the recommendations for observability?

Some of the things I want to achieve:

  • centralised searchable logs (ELK style)
  • granular metrics (host & container)
  • alarms on the above

I found that sumologic has a good pricing model for me. But is there some already integrated options?

2

Hello @zukoo i donā€™t have any recommendation but did you read this?

In the meantime, I will ask internally if anyone from the team can help you!

3

Here are a couple more blog posts:

Jason Dixon writes a weekly newsletter on monitoring. See the issue archive for lots of use cases and tools.

1 Like
5

Thanks for the responses,

Iā€™m now considering more seriously Grafana cloud and Datadog.

Iā€™ve tried to setup up both with the help from the above blog posts, Datadog works for the system metrics but I can see two things missing: no logs and no containers ( I can see the images but none of them shows as ā€œRunningā€).

logs_enabled: true
listeners:
  - name: docker
config_providers:
  - name: docker
    polling: true
logs_config:
    container_collect_all: true
process_config:
  process_collection:
    enabled: true
apm_config:
  enabled: false # disable APM
site: us3.datadoghq.com

Not sure if itā€™s related but i see those errors:

image
image1974Ɨ492 246 KB

6

Hello @zukoo iā€™m not sure if the Datadog blogpost is too old. Did you try this?

itā€™s old as well and then iā€™m checking the Pull Requests existing to the associated repo ā†’ Pull requests Ā· balena-io-examples/balena-datadog Ā· GitHub

What did you try?

7

@mpous I tried all three blogs above, for graphana i couldnā€™t make it work. And for datadog both the IOT and the normal client (i had to modify the dockerfile to get it to work and use the latest version) had the same issue.

This is an example on the IOT client where metrics work (even the docker ones) but no logs and no containers:

image
image1741Ɨ640 53.3 KB

image
image1741Ɨ640 28.6 KB

image
image1741Ɨ640 42.4 KB

8

@zukoo i never worked with Datadog! Maybe @kb2ma has some ideas?

on the other hand, what issues do you have with Grafana?

9

Itā€™s not sending any data, the logs of the collector are filled with ā€˜Permanent error: Permanent error: Post "https://1480699:***@https//prometheus-prod-37-prod-ap-southeast-1.grafana.net/api/prom/push\": dial tcp: lookup https on 10.114.102.1:53: no such hostā€™

image
image1801Ɨ250 88.5 KB

EDIT: well Iā€™m sorry after writing this i realize the issue with URL which shouldnt have the https, Iā€™m trying without it now

1 Like
10

Thanks @mpous, so my Grafana setup is now on par with the datadog one. I donā€™t see anything in the blog about logs. Do you recommend any way/exporter to upload the stdout/stderr of my containers to grafana?

11

So i didnā€™t manage to get the agent to tail the containers output, but at least temporally i got the logs out to Datadog directly from my application code using: GitHub - DataDog/datadog-api-client-python: Python client for the Datadog API

1 Like
12

That sounds good! let us know if this is your latest configuration!

Let us know if we can help you more!

13

Iā€™m heading down this path myself, I need to set something up. @zukoo any updates to your setup and testing?

14

Hey @philletourneau,

Currently i use both IoT fleet monitoring with Datadog and balenaCloud: How small agent containers make a big impact - balena Blog to get host metrics, and the datadog SDK GitHub - DataDog/datadog-api-client-python: Python client for the Datadog API to stream logs.

If you find a way to use the agent to stream logs Iā€™ll be interested to know how.

1 Like
15

Thanks for the update. Iā€™d love to get logs and host metrics all-in-one too! I havenā€™t set anything up yet, but GrafanaCloud looks very tempting because of the pricing, though I donā€™t know if I can figure out how to set that up, theyā€™re moving to something new now?

1 Like
16

@philletourneau @zukoo We started using GCP Monitoring to collect both (logs and metrics), published by an OTEL collector: opentelemetry-collector-contrib/exporter/googlecloudexporter/README.md at main Ā· open-telemetry/opentelemetry-collector-contrib Ā· GitHub

So as long as you get stuff into the collector, everything ends up in a GCP dashboard.

@mpous To get logs and metrics up, having an OTEL collector as a service is obviously easy. For metrics you have to deal with each service, makes sense, itā€™s very specific. But what would be a tremendous simplification is to have Balena export the console logs of all services to the collector automatically. In that case we donā€™t have to instrumentalise each service independently. Our devices run 5 to 10 different services, and some are 3rd party.

2 Likes
17

@ada this is a really interesting feedback!

Do you think you can introduce this in our public roadmap tool so our team can think about it?

Thanks

18

:+1: done: Publish all console logs to OpenTelemetry Collector Ā· Balena Roadmap

1 Like
19

Thanks for additional ideas and feedback folks!

I just started experimenting with a new solution in beta, Pydantic Logfire | Uncomplicated observability so far itā€™s really great if youā€™re running Python services.

3 Likes