Persistent Logging Isn't working

Product support
1

I have enabled persistent logging in the balenaCloud interface,

image
image.png1758Ɨ162 18.7 KB

with an acceptable supervisor version:
image
but when I log into my device my journalctl output only shows logs since the last time I rebooted the device, which was today:

-- Logs begin at Mon 2019-10-14 21:09:40 UTC, end at Mon 2019-10-14 21:47:23 UTC. --

I turned on persistent logging to find out why my devices keep getting disconnected from their modem, so as you can imagine I need to have logs that survive a reboot.

Furthermore, cat /etc/systemd/journald.conf shows that Storage is still on auto. According to this post, it should be set to ā€˜persistentā€™:

[Journal]
Storage=auto
#Compress=yes
#Seal=yes
2

hi @cnr that is very strange. when you enabled persistent logging, did you see the supervisor lots indicate the config change are reboot the device to apply the change? Also could you tell me the OS version you are trying this on, then I could give it a try my side.

It also might be that the dropping modem issue affected the setting of the config, but that would be strange.

4

Iā€™m on the CTI Orbitty TX2 (BETA)

I just checked two other devices as well, but they donā€™t have a Storage=persistent line either (they have the Storage=auto instead), despite enabling persistent logging for them as well. So, maybe itā€™s not supposed to change there and supervisor somehow tells systemd to log events elsewhere?

That or my deviceā€™s configs arenā€™t being updated?

My devices rebooted, yes. I remember because two of them went down and I had to drive up and manually restart them.

Iā€™ve also rebooted the device several times since then.

5

Sounds like something failed to commit, so I would disable/re-enable the feature and check the config again. If itā€™s still not working then let us know.

7

Ok, last time I did this two of my devices went down, so Iā€™m going to wait until I can get an update out that will reboot them in case they loose connection before I try this.

8

OK, please let us know what happens when you get a chance to try it

1 Like
10

Ok, I disabled and reenabled and the output of cat /etc/systemd/journald.conf is still:

#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See journald.conf(5) for details.

[Journal]
Storage=auto
#Compress=yes

I am seeing the change in the logs:

18.10.19 15:43:00 (+0000) Applying configuration change {"SUPERVISOR_PERSISTENT_LOGGING":"true"}
18.10.19 15:43:00 (+0000) Applied configuration change {"SUPERVISOR_PERSISTENT_LOGGING":"true"}

Are you certain that the Host /etc/systemd/journald.conf file should be changed? Maybe itā€™s not supposed to modify the actual text file? It is a read-only system. Just checking.

Retrying that setting also caused one of my devices to drop again meaning it now needs to be manually rebooted, I would prefer not to have to do any more testing that way if possible.

12

Hi,

Can you please check if persistentLogging in config.json file in the boot partition of the device is set to True?
Also the logs will be accessible at /var/log/journal in the hostOS, can you please check if there is any logs in that directory?

14

Also the logs will be accessible at /var/log/journal in the hostOS, can you please check if there is any logs in that directory?

Yes, there are logs in that directory.

root@my_device:~$ ls /var/log/journal
8e047396788042bab1902eebf7753f5b

root@my_device:~$ ls /var/log/journal/8e047396788042bab1902eebf7753f5b/ | wc -l
8

persistentLogging in config.json file in the boot partition of the device is set to True?

It is not.

root@my_device:~$ cat /resin-boot/config.json
{
  "deviceType": "orbitty-tx2",
  "localMode": true,
  "persistentLogging": false
}
16

Hi,

Regarding the persistentLogging, can you please check /mnt/boot/config.json instead?
And journalctl wonā€™t show the old logs by default, as I mentioned above, you can access them via /var/log/journal or run journalctl --file <path to old journal file>

18

ā€œpersistentLoggingā€:true}

Yeah, and if I journalctl --file <old file> it works. Thanks, I didnā€™t realize that I would have to manually select the old logs, for some reason I thought journalctl pulled those up automatically. This is user error and is solved.

However, Iā€™m only seeing up to 8 1Mb files. I seem to remember that thatā€™s the limit. Is there a way to increase that limit? Iā€™d be happy giving 1 Gb to my logging, since I have a problem that seems to be really intermittant and hard to debug and I have tons of (non-SD) disk space that Iā€™d like to use. Is this the right Github issue to track: Move location of persistent logs to data partition and increase size to 32Mb Ā· Issue #1646 Ā· balena-os/meta-balena Ā· GitHub ?

Thank you for your amazing support :smile:

19

@cnr Glad you got it figured out. Unfortunately its not currently possible to expand the persistent logging beyond the 8M, its something that we will hopefully enable in balena OS 3.0, however there is a way to get the journald logs streamed into your container where you can write/stream it to where ever you need it. You can see the docs on that API here: https://github.com/balena-io/balena-supervisor/blob/master/docs/API.md#journald-logs , just note that its only available in supervisor version v10.2.0 and above, so might not be possible in your case without upgrading.

1 Like
21

Gotcha, thank you!

24

Hi @cnr just following up here to let you know that as of balenaOS 2.45 and above the persistent logging has been increased to 32MB. You should see those versions of the OS rolling out over the next few weeks.

1 Like