I have this Lora base Rpi device which is offline most of the time, but I still want to examine the logs.
Which are the config files that I need to change to enable local logging with the balena docker engine?
Hi,
I think what you are looking for is persistent logging. You can set persistent logging in the device configuration page.
Yep, that is it, thanks a bunch!
ok now I have it enabled, but as soon as the Rpi looses internet access and no logs are sae anywhere.
How is this suppose to work?
What I did:
disconnected the Rpi from the internet
connected it again after few minutes.
checked the logs from the web gui - connected from the web terminal
balena logs containerID
it shows only the logs while it was connected to the internet.
Device logs do not appear in the Dashboard. When the device comes back online you can SSH into the host OS via the Dashboard terminal (or CLI) and read them through journald. Your application logs should also show up there. Persistent logs also persist across reboots.
Which is the unit in journals that would have the logs?
I tried balena and balena-angine, but none hold any container logs.
also balena logs containerID has a gap in he logs for when it was not connected to the internet.
how do I check if persistent logging is actually enabled by this env variable?
Just very quick background. We run balenaEngine with logging driver set to journald. As a result, all container logs are accessible then with journalctl or with balena-engine logs <containerId>. The latter gives access only to recent logs kept by the engine itself - they are not stored across reboots. The persisted logging flag you set in device configuration enables storing journald across device reboots.
Meta data stored in journald is documented here:
And it has a paragraph about retrieving logs from journalctl. For instance, you could run
journalctl CONTAINER_ID=<ID from balena-engine ps>
Let us know if you’ve managed to access the logs.
That was very useful, thanks, I got the logs. for some reason it only shows the last 2 days although the container was created 3 days ago, but that should be enough logs.
btw a bit unrelated question: With the persistent logging disabled does the engine make any writes to the SD card or keeps the logs in memory to extend the SD card life?
There is a hard limit of 8MB for persisted logs (not configurable atm). Hence, older logs could be deleted because of this. With persistent logging, the engine does not write to SD card, but journald does to keep the logs across reboots.
Thanks for the answers.
My question was whether any SD writes are done with the persistent logging switched OFF.
Ah, sorry, I misread the question. No, when persistent logging is off, logs are kept in memory to preserve the SD card life.
Why is that? Some hard coded value in custom log rotator or something else?
The persisted logs currently live on the state partition, which is only 16MB total, that’s why they are limited to 8MB atm. But we are looking into increasing this limit by moving the logs: https://github.com/balena-os/meta-balena/issues/1646
Thanks I will track that issue!
Will this change also make the journalctl logs available to the other services? So far I have only been able to use copy/paste from the dashboard to export log information for analysis.
Hi Jason. the change won’t affect where the logs will be available, they would still need to be grabbed via webterminal. If you are looking to shift logs out to a third party service or something, i would recommend using the journal logs API in the supervisor (documentation here: https://www.balena.io/docs/reference/supervisor/supervisor-api/#journald-logs ) and there is a demo repo here: https://github.com/balena-io-playground/device-cloud-logging which shows how to log locally to container or export to loggly
Thanks Shaun!