Netdata IoT monitoring

Project help
1

netdata is a system for distributed real-time performance and health monitoring. It provides unparalleled insights, in real-time, of everything happening on the system it runs (including applications such as web and database servers), using modern interactive web dashboards.
netdata is fast and efficient, designed to permanently run on all systems (physical & virtual servers, containers, IoT devices), without disrupting their core function.

I already deployed it on RPi 1 (!) and the performance is awesome even with default settings, but there are also some IoT perf. tweaks I will be adding with ansible:

There’s also a sensors library, so it’s a great way to keep track of data of them in real time.
It’s also a good way to check performance and resource usage of existing projects with minimal footprint on them.

1 Like
2

Hey, very interesting! :slight_smile:
Just been reading the Dockerfile.template, and seen that ansible is run during the build (with a RUN step), is that intended? Can you give a nutshell info what would it do? I was wondering if it gives info on the local host, it might be an issue when the builder and the machine that runs it are not the same machine. Is it deployed on the standalone resinOS or through resin.io? (or maybe I misunderstand something?)

3

Since Dockerfile is not a very flexible and powerful way of installing and configuring stuff, I use ansible to deploy the netdata app.
Ansible, in this case, is installed locally in the docker container and runs locally on the container itself too. As it is a very lightweight app installed by pip it’s not a big resource consumer.
This way the playbook.yml can be extended and updated easily without dealing with Dockerfiles or bash scripts.

The colorized Ansible output is then served by http in the last steps.

As a example, the step:

  tasks:
    - name: Display all variables/facts known for a host
      debug:
        var: hostvars

collects facts from the container and also the host itself.

The netdata app runs on port 19999 by default, so it can be accessed through local network ip.
As an example I exposed it here (changing the port):
https://c26341b6b3808824931df24b2956c702660959c662b4eeadbf2ac3a76a9e2a.resindevice.io/

4

oops now I see the issue, the docker image is actually built outside, so the facts are not from the device, I’ll remove that dummy step :grin:

5

Hi @imrehg
To be clearer, I’ve been using ansible as an experiment here.
In this project it may seem redundant and/or an overkill, but it could be a way to have more modular projects without combining them in a single Dockerfile or using docker-in-docker, etc.

Let’s say I want a monero node with netdata monitoring, for example :slight_smile:

There’s one caveat though: using ansible locally in the container, you lose the benefits of the cache, as it runs all in a single RUN step.
I’ve been trying to disable the cache for those steps but I couldn’t.
Is there a way to disable the cache in build time?

6

Running locally means using the rdt (resin-device-toolbox), right? It seems to me that there’s no way to bust the cache, added an issue on github (can follow the results there). In the meantime you can work around it by adding an ENV at the point in the Dockerfile after which you want to bust the cache, and change the value to trigger a rebuild for all the layers afterwards. Eg.

# Change the value of this env var to rerun the following commands without cache
ENV CACHE 1

RUN ansible-galaxy ...
...

It’s not very nice, but works. In general a nicer way is to version anything that’s possible (version your ansible config I guess, etc), as version number change implies rebuild too, otherwise Docker pretty much rightly things that things haven’t changed and no need for rebuild. (It’s good practice whenever possible, but it’s understandably extra work to organize)

In hosted builds (on resin.io) you can push to the resin-nocache branch to build without caching (so for example git push resin master:resin-nocache).

I don’t know ansible well enough to see the modularity, so if you have any other examples, would be happy to hear.

7

I’ve been using hosted builds actually, but I didn’t find anything related with cache busting in the docs, I think it’s worth adding the resin-nocache feature.

Another alternative would be build time arguments, so the cache can be busted at a specific point always, like described here: http://dev.im-bot.com/docker-select-caching/

Thanks for the help!

1 Like
8

Yeah, the resin-nocache is an experimental feature, but been used long enough to warrant the docs, totally with you :pencil2:

The build time arguments (and settings) is something that we are actively exploring, I know @cameron has some plans and discussions for the proper implementation, as the backend needs some changes to make it possible. :wink:

Thanks for the feedback, all these are making resin.io better :sparkles: