Hi,
Is there any way we can configure the amount of space allocated to the journal when persistent logging is enabled? We’re having a hard time tracking down some issues in the wild because the journal gets filled with console output after the issue occurs but before the user notices and stops the application, and any relevant information is then lost. We have plenty of storage on our device, so limiting to just 32 MB is not necessary for us.
Thanks,
Adam
Hi.
Currently, the 32 MB limit is not adjustable, see this thread for a related discussion.
In the meantime, if your applications are quite verbose, you might want to try and cut down the logging output in order to make the limit work with persistent logging.
Hi Pranas,
Unfortunately, the logging output from our application is as reduced as we can make it at the moment while allowing us to see that the condition we’re looking for has happened and hopefully figure out what went wrong just before that. This is particularly true since the journal is full of lots of network change/device prints from the balena engine and supervisor (discussed here) that take up space and may or may not be relevant to the issue we’re trying to track down. This issue is pretty difficult to detect, and when it issue happens in the wild to our customers, they haven’t noticed some time before reporting it, and by then the information is usually gone.
Since the journal is stored on a fixed-size partition, it makes sense that it can’t be changed trivially. It would be really great if there was a user control or something that could redirect the journal location to instead be stored on the larger persistent storage partition where the Docker shared volume is located. We have tons of available space and it would be really nice to take advantage of that, at least for debugging efforts.
Just a thought.
I took a look at https://github.com/balena-io-examples/device-cloud-logging and it looks like maybe that could be used to redirect a copy of the log data to the shared volume. We can take a look at that as a workaround.
That’s exactly what I wanted to mention. While not ideal, it could help your case by offloading the logs elsewhere. Let us know how it goes!
Is there any way to specify arguments to journalctl using the supervisor /v2/journal-logs query? https://www.balena.io/docs/reference/supervisor/supervisor-api/#journald-logs supports a few (--line, etc.), but we would ideally like to use --since=. That way, whenever this logging service restarts it can pick up where it left off.
Alternatively we could maybe query the full journal and prune everything out from the start, but since journalctl already supports exactly this feature it would be nice to take advantage of that if possible. It would also be nice to use some of the other features like --priority and --boot. Is there any way /v2/journal-logs could have an extra argument that is just “pass the following arguments to journalctl” so we could use all of the native features of journalctl?
Hello,
Unfortunately --since is not supported.
All supported options are here: https://github.com/balena-io/balena-supervisor/blob/master/src/device-api/v2.ts#L593-L597
There is an open issue for this https://github.com/balena-io/balena-supervisor/issues/1062#issuecomment-521636406
I’ll ping the supervisor team about this.
since is supported now, see
Yes, the amount of space allocated to the journal when persistent logging is enabled can usually be configured in the settings of the specific logging system or database management system being used, and for more info visit.
For example, in the case of the MongoDB database management system, the size of the journal can be configured using the “storage.journal.volume.size” parameter in the configuration file. This parameter specifies the maximum size, in megabytes, of the journal volume, and can be set to a value that meets the specific needs of the application.
Configuring the journal size empowers developers to tailor the logging system to their specific needs. Organizations with sufficient storage capacity can benefit from adjusting the journal size to strike a balance between retaining enough log information for analysis and avoiding excessive disk space consumption.
The systemd journald service uses two settings to control the amount of disk space that is used for persistent logging: SystemMaxUse and RuntimeMaxUse. The default values for these settings are 32 MB and 16 MB, respectively. You can increase the value of either of these settings to increase the amount of disk space that is allocated to the journal.
You can configure the amount of space allocated to the journal when persistent logging is enabled. The SystemMaxUse option in the journald.conf file controls the maximum size of the journal. The default value for this option is 32 MB. To increase the size, you need to change the value to a larger number.
The ability to configure the amount of space allocated to the journal when persistent logging is enabled depends on the specific logging system or tool you are using. Since you mentioned the “journal” and “console output,” I assume you are referring to the systemd journal on a Linux system.
In systemd journal, the storage limits are defined in the “journald.conf” configuration file. To increase the amount of space allocated to the journal, you can modify the SystemMaxUse parameter. By default, it is set to a conservative value like 10% of the total available disk space or a fixed value (e.g., 4G, 32M, etc.).