how exactly balena device logs work on OpenBalena

openBalena
1

Hi everyone, i have some questions about device logs on openbalena.

if persistent logging is not enabled, whats the storage limit for local logs?
are those logs uploaded to the server, if so, for how much time are they persisted?
if persistent logging is enabled, are the logs limited to 32MB on the device? do they get uploaded to the server?

does the balena-cli grab the logs from the server or directly from the decive using the vpn?
the docs specify that the logs are deleted from the device after every restart, but i can keep seeing logs from past restarts on my device using the cli.
thanks in advance.

2

From BalenaOS 2.x docs:

In balenaOS, logs are written to an 8 MB journald RAM buffer in order to avoid wear on the flash storage used by most of the supported boards.

This appears to be shared between all containers and system logs. I checked balena logs for the main service on one long-running device and found slightly over 0.5 MB there. The same link covers the persistent size limits too (32 on everything since 2.45).

And yes, I have seen BalenaOS send log stream requests to a server, though I am not sure whether all devices do this constantly, or only ones that have been asked for logs via the CLI at some point since booting. Probably not affected by persistence settings. Logs are definitely deleted without persistence, so any longer history that you see must be from the server.

3

Hello @jose_barrero,

thanks for your questions.

@MBer thanks a lot for sharing the details about on device logging.

Each device has a configuration variable: SUPERVISOR_LOG_CONTROL in the set of configuration variables (Configuration List for Raspberry Pi 4 - Balena Documentation)

This control variable will enable or disable if the supervisor will send logging data to an API log endpoint. This logging data will be sent to a redis cache (thus it’s not persistent). From there the API serves on an read endpoint the cached log entries. For reference here the open-balena-api entrypoints: open-balena-api/index.ts at f112f4d3b72de599b77165c5aea54e9d60da8716 · balena-io/open-balena-api · GitHub

As long as the redis cache is not invalidated, eg. by restarting, the old logs may be accessible from the read endpoint. There is a default configuration of last 1000 entries are accessible from the read endpoint. Please be aware that only service logs are send and read from the API. The device level logging has more detail and logs also all systemd services running on balenaOS. The logs that are send to the API are directly captured by the balena supervisor from the balena engine.

The balena cli will grab the logs from server, except when the device is running in local mode. Then the balena CLI will attach to the balena engine in local network and streams the logs from there. In this case the logs are not sent to the server.

I hope these details help.

Best regards
Harald

1 Like