Splitting Logs Every Hour with logrotate and Compressing with zstd
Server
Lastmod: 2023-11-30
Published: 2021-03-29

I will leave a note about when I wanted to split nginx logs every hour while compressing them with zstd using logrotate.

Overview

Simply putting the settings in /etc/logrotate.conf or /etc/logrotate.d/ will basically execute it with the daily CRON of /etc/cron.daily/logrotate.

This time, since I want to rotate logs every hour, I prepared a separate logrotate setting and made it run using the timer function of systemd.

Enabling zstd

# apt install zstd

logrotate

I will create /etc/logrotate-nginx.conf.
This time it’s zstd, but you can change it to bzip2 or other settings similarly.

# cat << _EOF_ > /etc/logrotate-nginx.conf
compresscmd /usr/bin/zstd # Specify the path to zstd
compressoptions -12       # Options passed to zstd. Personally, -12 felt good
compressext .zst          # Extension to append

/var/log/nginx/*.log {
        missingok
        rotate 720 # 24 hours * 30 days
        dateext
        dateformat -%Y%m%d%H
        compress
        notifempty
        create 0640 www-data adm
        sharedscripts
        prerotate
                if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
                        run-parts /etc/logrotate.d/httpd-prerotate; \
                fi \
        endscript
        postrotate
                invoke-rc.d nginx rotate >/dev/null 2>&1
        endscript
}
_EOF_

systemd

  • Define the service
# cat << _EOF_ > /etc/systemd/system/logrotate-nginx.service
[Unit]
Description=logrotate-nginx
After=network.target
[Service]
Type=simple
ExecStart=/usr/sbin/logrotate -f /etc/logrotate-nginx.conf
PrivateTmp=true
[Install]
WantedBy=multi-user.target
_EOF_
  • Define the timer
# cat << _EOF_ > /etc/systemd/system/logrotate-nginx.timer
[Unit]
Description=logrotate-nginx
[Timer]
OnCalendar=*:00:00
[Install]
WantedBy=timers.target
_EOF_
  • Reload systemd settings
# systemctl daemon-reload
  • Enable the timer
# systemctl enable logrotate-nginx.timer
  • Start the timer
# systemctl start logrotate-nginx.timer
  • Check the status of the timer
# systemctl status logrotate-nginx.timer
● logrotate-nginx.timer - logrotate-nginx
     Loaded: loaded (/etc/systemd/system/logrotate-nginx.timer; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2021-03-29 20:00:00 JST; 5h 58min ago
    Trigger: n/a
   Triggers: ● logrotate-nginx.service

Mar 29 19:00:00 log systemd[1]: Started logrotate-nginx.

You can know the next scheduled execution time and the past execution times from the log when it is active 🎉

Note added in November 2023

If you divide logs every hour, it’s not much of a concern, but if you divide logs every minute, there will be deviations due to the timer’s precision, and times other than the expected ones may be included.

Therefore, if you want it to run exactly at 0 seconds, you should set it as follows.

[Unit]
Description=logrotate-nginx
[Timer]
OnCalendar=*:*:00
AccuracySec=1s
[Install]
WantedBy=timers.target

By specifying AccuracySec, you can execute it with a precision of 1 second.