I would like to create a stack to capture and display my home environment metrics. I am also playing with ideas to incorporate Home Assistant, not that I have any ideas yet, but in case it drives the direction of any response.
I have read the BalenaSense project, and that looks suitable. More specifically the guide to “aggregate data from a fleet of sensors”. The twist here is I would like to use lower-power microcontrollers; something like the ESP8266. I have had this working with a vanilla TICK stack on a Pi, that had the ESP8266 write directly to InfluxDB. With the way Balena Sense is setup, should messages instead be handled by Telegraf and MQTT?
This is all a learning exercise for me too, and thank you in advance for guidance.
Welcome to balena forums,
We are happy to know about your project. As you mentioned you the messages should be handled by MQTT. There are a lot of resources on the internet describing how to implement that. http://nilhcem.com/iot/home-monitoring-with-mqtt-influxdb-grafana
Please don’t forget to share your project with us.
Wish you happy tinkering!
I have a balena-sense stack built, running mqtt, telegraf, influxdb, grafana.
I want to send data (from various network attached sources) to the mqtt broker, but that action is failing.
As a basic test (I don’t know if it’s a valid one) I try to telnet to port 1883, both locally and over a network, neither connect. I have ensured 1883 is listening, and have exposed the port in docker-compose.yml, but still I can not connect.
I want to say it’s a firewall issue, it reacts like one, but I also don’t know enough about how mqtt works to know the difference.
That works on the local host, thanks for the suggestion. This prompted me to further check the mosquitto configuration and I have made good progress!
I noticed mqtt is running in local host mode only:
indent preformatted text by 4 spaces/ # mosquitto -v
1615902985: mosquitto version 2.0.9 starting
1615902985: Using default config.
1615902985: Starting in local only mode. Connections will only be possible from clients running on
this machine.
1615902985: Create a configuration file which defines a listener to allow remote access.
Which was confirmed in this breaking change with the release of mosquitto version 2.0:
When the Mosquitto broker is run without configuring any listeners it will now bind to the loopback interfaces 127.0.0.1 and/or ::1. This means that only connections from the local host will be possible.
I got around that by adding the following settings to mosquitto.conf:
bind_address 172.17.0.5
allow_anonymous true
Then running the service again:
/ # mosquitto -c /mosquitto/config/mosquitto.conf -d
1615904774: The 'bind_address' option is now deprecated and will be removed in a future version.
The behaviour will default to true.
As can be seen, there is a deprecation warning using bind_address but at least MQTT is now working across a local container network.
I will test and confirm the correct setup and operation of the ESP8266 part of this too before marking as resolved.
I’m still struggling to get data from the esp to the pi stack. As a reminder here is the stack:
esp8266 w/ temperature sensor
balena-sense: influx, telegraf, mqtt, grafana – with sensor service removed
I configured telegraf to run in INTERNAL mode which means it’s allowed remote connections. This part works, as I see hits land on the telegraf logs, which leads me nicely on to the problem.
From the telegraf logs I get this error:
telegraf 2021-03-19T21:45:47Z E! [inputs.mqtt_consumer] Error in plugin: metric parse error: expected tag at 1:16: "esp8266_sensor, sensor=temperature temperature=23.1"
I’m not sure where the problem lays, whether it’s message syntax for mqtt or what influxdb is expecting. I’ll post some snippets that someone can hopefully then assist me with.
Aside from the balena-sense database there are no series entries.
Really need some help on this one, been starring at it for too long, can’t now see wood for trees!!! Thanks in advance. Happy to post more config if required.
Rereading my previous posts, I can tell now this was clearly a learning exercise!
I have this working in principle as I would like now. It was a combination of mqtt topic, telegraf input modification and grafana dashboard creation. I doubt further details add any value to this post but it was nice to finally get it working.
As it is only a minor derivative of balena-sense, is it worth posting as a finished balena project when it is?
Thank you for getting back to us! I’m glad to see you have this working now. It would be great if you could write up your project, so others can follow along with this (and future ) projects.
Also, check out our post on balenaBlocks: Introducing balenaBlocks: jumpstart your IoT app development
Specifically the connector block. It does what you are doing here - just in case it would make things easier for future ventures. One of my colleagues is currently re-implementing balenaSense using blocks, so keep an eye on the blog.
I’m having the same issue. It looks the sensor data is not getting sent or is being blocked somewhere. Like n1md4 says, it seems to be an mqtt broker issue
There are 2 configuration lines that need adding to /mosquitto/config/mosquitto.conf.
Listener: By default Mosquitto does not allow external connections, should they be required you need to add a listener listener 1883 <host-ip>.
Authentication: You have two options here, either you have set a user name and password, in which case you need to add those details to mosquitto.conf, else you allow anonymous connections. I have the latter for now allow_anonymous true.
Add those 2 lines then run mosquitto -c /mosquitto/config/mosquitto.conf -d then you can logout. At which point you should see log entries within Balena Cloud UI.
Note: There will be a more elegant solution to add listeners and authentication details as variables and requires adding application build details, as yet I’ve not looked into that though. This means that each time you push any updates you will need to apply this hack to make it work again.
Once you have that working, you will need to add a new dashboard to grafana and all the panels you wish. Once you have a config you like you can export that in json and add it to your balena sense branch. Again, there’s a bit of reworking required to change some of the default grafana bits so it instead displays your custom data, and again something else I’m yet to tidy up!
There may be one or two more configurations required. Let me know if you get stuck, and if I had a fix I’ll happily share it with you. One thing I found very useful during testing was to install mqttbox. Good luck and happy hacking
) projects. 