Increase journal size when persistent logging is enabled?

Hi,

Is there any way we can configure the amount of space allocated to the journal when persistent logging is enabled? We’re having a hard time tracking down some issues in the wild because the journal gets filled with console output after the issue occurs but before the user notices and stops the application, and any relevant information is then lost. We have plenty of storage on our device, so limiting to just 32 MB is not necessary for us.

Thanks,

Adam

Hi.

Currently, the 32 MB limit is not adjustable, see this thread for a related discussion.

In the meantime, if your applications are quite verbose, you might want to try and cut down the logging output in order to make the limit work with persistent logging.

Hi Pranas,

Unfortunately, the logging output from our application is as reduced as we can make it at the moment while allowing us to see that the condition we’re looking for has happened and hopefully figure out what went wrong just before that. This is particularly true since the journal is full of lots of network change/device prints from the balena engine and supervisor (discussed here) that take up space and may or may not be relevant to the issue we’re trying to track down. This issue is pretty difficult to detect, and when it issue happens in the wild to our customers, they haven’t noticed some time before reporting it, and by then the information is usually gone.

Since the journal is stored on a fixed-size partition, it makes sense that it can’t be changed trivially. It would be really great if there was a user control or something that could redirect the journal location to instead be stored on the larger persistent storage partition where the Docker shared volume is located. We have tons of available space and it would be really nice to take advantage of that, at least for debugging efforts.

Just a thought.

I took a look at https://github.com/balena-io-examples/device-cloud-logging and it looks like maybe that could be used to redirect a copy of the log data to the shared volume. We can take a look at that as a workaround.

That’s exactly what I wanted to mention. While not ideal, it could help your case by offloading the logs elsewhere. Let us know how it goes!

Is there any way to specify arguments to journalctl using the supervisor /v2/journal-logs query? https://www.balena.io/docs/reference/supervisor/supervisor-api/#journald-logs supports a few (--line, etc.), but we would ideally like to use --since=. That way, whenever this logging service restarts it can pick up where it left off.

Alternatively we could maybe query the full journal and prune everything out from the start, but since journalctl already supports exactly this feature it would be nice to take advantage of that if possible. It would also be nice to use some of the other features like --priority and --boot. Is there any way /v2/journal-logs could have an extra argument that is just “pass the following arguments to journalctl” so we could use all of the native features of journalctl?

Hello,

Unfortunately --since is not supported.
All supported options are here: https://github.com/balena-io/balena-supervisor/blob/master/src/device-api/v2.ts#L593-L597

There is an open issue for this https://github.com/balena-io/balena-supervisor/issues/1062#issuecomment-521636406

I’ll ping the supervisor team about this.