Service logs retention

Hi,

I’d like to know more about how log retention works. Most of the time I can’t find the logs I’m looking for if it’s not something that happened very recently.

On the device logs are available a bit longer but they disappear when a container is recreated (eg: new deploy, service is restarted manualy or because it crashed). Sometime I’m able to find more logs in journald but it’s not really handy

On balena-cloud.com logs are disappearing after some time limit I think but I’m not sure

The most important thing for me would be to have reliable access to the logs from balena-cloud.com for at least a few days.

thank you

Hello
Thanks for reaching out, have you tried using persistent logging. Here’s more information about it, Configuration - Balena Documentation
You can toggle persistent logging from the dashboard under the device/fleet configuration menu. Also, you can try downloading the logs to get all the logs available. Hopefully these help, let me know if these solve your problem.

Hi @vipulgupta2048,

yes, persistent logging is already enabled at the fleet level. I tried downloading the logs but I only ha ve a few minutes of history (it’s even worst than I expected :laughing: )

I downloaded the logs a few times, I notice that they are always 999 lines long. Is this the limit I’m hitting ?

Hello,
Thanks for reaching back. Here’s some information regarding logs retention that might be helpful context.

SystemMaxUse= and RuntimeMaxUse= control how much disk space the journal may use up at most. SystemKeepFree= and RuntimeKeepFree= control how much disk space systemd-journald shall leave free for other uses. systemd-journald will respect both limits and use the smaller of the two values. From, journald.conf

  • The limit is indeed 1000 lines when the log file is downloaded. As that is the number of logs stored in the API, and viewable from the dashboard. The reason is similar as the current solution isn’t meant to be used for long-term, reliable storage of logs. It’s designed to provide the latest logs from the device for direct debugging purposes. We do plan on expanding our logging solution to offer long-term storage and search.

Thanks for bringing this up, let us know if this helps you out or what worked for you. It will go on to help other folks on the forums!

ok I understand the situation much better now.

Indeed I’ll switch back off the persistance logs to avoid damagind the sd cards. For now I’ll try to reduce the log output. It’s something I should have done either way

We do plan on expanding our logging solution to offer long-term storage and search.

that would be amazing ! :slight_smile:

thank you @vipulgupta2048