Service logs retention

[mathroc]

Hi,

I’d like to know more about how log retention works. Most of the time I can’t find the logs I’m looking for if it’s not something that happened very recently.

On the device logs are available a bit longer but they disappear when a container is recreated (eg: new deploy, service is restarted manualy or because it crashed). Sometime I’m able to find more logs in journald but it’s not really handy

On balena-cloud.com logs are disappearing after some time limit I think but I’m not sure

The most important thing for me would be to have reliable access to the logs from balena-cloud.com for at least a few days.

thank you

[vipulgupta2048]

Hello
Thanks for reaching out, have you tried using persistent logging. Here’s more information about it, Configuration - Balena Documentation
You can toggle persistent logging from the dashboard under the device/fleet configuration menu. Also, you can try downloading the logs to get all the logs available. Hopefully these help, let me know if these solve your problem.

[mathroc]

Hi @vipulgupta2048,

yes, persistent logging is already enabled at the fleet level. I tried downloading the logs but I only ha ve a few minutes of history (it’s even worst than I expected )

I downloaded the logs a few times, I notice that they are always 999 lines long. Is this the limit I’m hitting ?

[vipulgupta2048]

Hello,
Thanks for reaching back. Here’s some information regarding logs retention that might be helpful context.

• Persistent logging is designed for it be a debugging feature, and not a way to store logs on device permanently.

• If you are dealing with lot of logs over the course of days, then persistent logging might not be a good long term solution as it will wear your storage media incrementally. What you could try is streaming the logs using the supervisor API. Check: Interacting with the balena Supervisor - Balena Documentation. You can stream logs using this endpoint to whatever platform you use to store logs reliably over the course of time. For example, Datadog: IoT fleet monitoring with Datadog and balenaCloud: How small agent containers make a big impact

• The persistent logging when enabled stores logs inside the device and makes sure logs persist between reboots. As part of the data partition on the device either on SD card, eMMC, hard-disk, etc. The limit you are hitting here might be the RuntimeMaxUse= in /etc/systemd/journald.conf which can be much larger than the default. You can increase the limit to potentially increase the logs being stored. Let us know if this works out. We have an issue open already to make this configurable using a variable here Allow users to configure the retention limit on persistent logs for OS versions where persistent logs are on the data partition · Issue #1310 · balena-os/balena-supervisor · GitHub

SystemMaxUse= and RuntimeMaxUse= control how much disk space the journal may use up at most. SystemKeepFree= and RuntimeKeepFree= control how much disk space systemd-journald shall leave free for other uses. systemd-journald will respect both limits and use the smaller of the two values. From, journald.conf

• The limit is indeed 1000 lines when the log file is downloaded. As that is the number of logs stored in the API, and viewable from the dashboard. The reason is similar as the current solution isn’t meant to be used for long-term, reliable storage of logs. It’s designed to provide the latest logs from the device for direct debugging purposes. We do plan on expanding our logging solution to offer long-term storage and search.

Thanks for bringing this up, let us know if this helps you out or what worked for you. It will go on to help other folks on the forums!

[mathroc]

ok I understand the situation much better now.

Indeed I’ll switch back off the persistance logs to avoid damagind the sd cards. For now I’ll try to reduce the log output. It’s something I should have done either way

We do plan on expanding our logging solution to offer long-term storage and search.

that would be amazing !

thank you @vipulgupta2048

[fisehara]

Hi, just a note to inform you that we now have some documentation about device logs at Device Logs - Balena Documentation.