Advice wanted: Log rotation without systemctl?

Product support balenaOS
1

I’m wondering if using logrotate is the best approach for rotating the logs written by our various programs in a container. All logs are written to a persistent shared volume.

We have in one container:

  • a C++ program that writes to a log file,
  • a bash script that is our dockerfile CMD and it writes to a log file,
  • a health check as bash script that also writes to a log file

In another container:

  • a tomcat host java servlet that writes logs using log4j2
  • a bash script that is our dockerfile CMD and it writes to a log file,
  • a C++ program that writes to a log file

In another container:

  • an nginx proxy that writes to access and error logs

Other than the log4j2 logging, which has the ability to rotate logs, all of the logs are being rotated using logrotate, which is if I understand it correctly requires systemctl. Which may require INITSYSTEM=0.

Is there a better way to rotate logs than using logrotate in each container? I realize that having a single container rotate all of the logs would require a single list of logs to be configured, although we could use a file pattern to rotate everything the same way.

4

Hi @jason10,

While logrotate is often controlled via systemd timer, it is not strictly necessary. You can call logrotate from any daemonized program if you like. It is quite common to run such housekeeping tasks from cron if systemd is not available on device.

Personally, I would recommend logging from all containers to a single root (ie mapping each container’s storage in a central place, like /data/logs/{container1,container2,container3} and then rotating those files from another container altogether, using a logrotate configuration similar to the following (ymmv). You should be able to run this all from a simple crontab entry or daemon.

/data/logs/*/* {
    monthly
    rotate 2
    olddir /data/logs/old
    missingok
}

Please do let us know what you decide to do!

6

I’m going to go with writing all of the logs to a shared volume and have one of the containers that already requires systemctl specify the log rotation. Thanks

8

Now that I’ve moved to newer balenaOS base images, I have found that systemctl is deprecated. I’ve tried to use cron to rotate the logs but my log rotations are not being performed.

Is there any additional examples of using cron and logrotate?

This older blog post used systemctl: https://www.balena.io/blog/how-to-create-a-custom-logging-system-for-longer-log-retention/

9

Hi,
Are you getting any error back?
Can you share your dockerfile and your logrotate configuration?
Please re-check that the container running the logrotate should have enough permissions to access the location of the logs in the shared volume.

Let me point you to our cron example:

Have you tried running logrotate manually using SSH, to config that the configuration is correct and it indeed works as expected?
If the repeating job seems to be the source of the trouble you could as easily have a separate container that runs logrotate then sleep in a loop.

Let me also point you to our docks about using systemd in newer balenalib base images, in case you prefer a workaround for now:
https://www.balena.io/docs/reference/base-images/base-images/#installing-your-own-initsystem

Kind regards,
Thodoris

12

Hi Thodoris,

No errors, and when I run the logrotate manually it works. Permissions are good for /etc/logrotate.etc, /etc/cron.hourly/logrotate, /etc/cron.daily/logrotate and everything else I have checked.

I looked at the “balena-io-playground/cron-example” but it is using crontab rather than /etc/cron.hourly/logrotate. If my understanding is correct, scripts in cron.hourly are executed using run-parts. When I test if my logrotate script in cron.hourly meets the requirements of run-parts the answer appears to be yes:

$ sudo run-parts --report --test /etc/cron.hourly
/etc/cron.hourly/logrotate

So, cron installed, logrotate installed, run-parts installed, shared-volume accessible, scripts executable, manually running logrotate works, manually running /etc/cron.hourly/logrotate works, manually running run-parts works.

Github repository: https://github.com/EIODiagnostics/Balena-logrotate-example
Balena device: https://dashboard.balena-cloud.com/devices/2e456d5d2326890c3a90c4a8d58b7295/summary

support access granted.

13

Hello,

This container is only running your script.
cron is not running here.

I see you are using a balenalib image with ENV INITSYSTEM on.
This can not work: https://www.balena.io/docs/reference/base-images/base-images/#major-changes
If you need an initsystem please follow these instructions: https://www.balena.io/docs/reference/base-images/base-images/#installing-your-own-initsystem

Otherwise you’ll need to start cron manually from your script.

15

I don’t see cron being started in the balena cron example. Where is that happening?

The initsystem is leftover from the resin base image. Sometimes migration to a new base image isn’t clean. I’ll look into it. Unfortunately I’m not an expert on what an “initsystem” does. So much to learn.

16

The cron example is quite old and as you can see, it is not using a balenalib/xxxx base image (it uses resin/%%RESIN_MACHINE_NAME%%-debian)
These old images had an option to run with an init system (systemd or openrc): ENV INITSYSTEM on.
If you have systemd running and you install cron, systemd will start cron when the container is started.

This feature was removed in balenalib base images so you need to either:

18

I’ll try these instructions https://www-cyberciti-biz.cdn.ampproject.org/v/s/www.cyberciti.biz/faq/howto-linux-unix-start-restart-cron/amp/?amp_js_v=a2&amp_gsa=1&usqp=mq331AQCKAE%3D#aoh=15732319727617&referrer=https%3A%2F%2Fwww.google.com&amp_tf=From%20%1%24s&ampshare=https%3A%2F%2Fwww.cyberciti.biz%2Ffaq%2Fhowto-linux-unix-start-restart-cron%2F

19

This won’t work, you don’t have systemd installed so service or systemctl won’t be available.
If you need an init system, please follow these instructions: https://www.balena.io/docs/reference/base-images/base-images/#installing-your-own-initsystem

21

Okay, so this is "working:

in my Dockerfile.template, try rotating files every minute (so I don’t have to wait too long):

echo "*/1 * * * * /etc/cron.daily/logrotate" | crontab

in my runCommand.bash

/etc/init.d/cron start

and then rm /data/log/logrotate/status and wait a minute…and boom, files are rotated.

TODO: adjust interval of logrotate script being run, remove /etc/cron.hourly/logrotate

22

Hi there,

Just a small suggestion, if you place a file in /etc/cron.d/ with the contents * * * * * {{path to script}} you shouldn’t need to pipe that into crontab, as cron instead will just execute that every minute.

24

Thanks @xginn8. Actually I found that as long as I start cron, the default crontab will run the cron.hourly and cron.daily scripts:

17 *    * * *   root    cd / && run-parts --report /etc/cron.hourly
25 6    * * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6    * * 7   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6    1 * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )