Unfortunately, the logging output from our application is as reduced as we can make it at the moment while allowing us to see that the condition we’re looking for has happened and hopefully figure out what went wrong just before that. This is particularly true since the journal is full of lots of network change/device prints from the balena engine and supervisor (discussed here) that take up space and may or may not be relevant to the issue we’re trying to track down. This issue is pretty difficult to detect, and when it issue happens in the wild to our customers, they haven’t noticed some time before reporting it, and by then the information is usually gone.
Since the journal is stored on a fixed-size partition, it makes sense that it can’t be changed trivially. It would be really great if there was a user control or something that could redirect the journal location to instead be stored on the larger persistent storage partition where the Docker shared volume is located. We have tons of available space and it would be really nice to take advantage of that, at least for debugging efforts.
Just a thought.