Since I’m using Nginx to proxy my AdGuardHome web interface and DNS over HTTPS, the access log files increased rapidly in size. So I’m going to limit these logs with a tool called Logrotate.
Introduction
Log rotation is a process of removing/storing old logs while not affecting the latest logs. Almost every single application generates meaningful logs, and if not properly handled, logs can eat up all the space on your disk. Logrotate helps to manage logs, compress them, remove unnecessary/old logs or even email them after a certain time period.
Installation
Most linux distributions come with logrotate by default. If that’s not the case on your device, just install it with following commands (e.g. Debian/Ubuntu):
sudo apt install logrotate
Logrotate has a configuration file in which we can mention all the files we want to rotate. It needs a time period unit i.e., daily, weekly, monthly etc., and rotate count i.e., 3, 4, 5 for each rotation. Log files are rotated count times before being removed. If you set the rotate count to 0, it means old version of logs will be removed rather than rotated. Otherwise, if your logs are being saved in file named myapp.log then after a rotation a new file will be created with name myapp.1.log, and so on.
Configuration file
Logrotate is configured using the main configuration file, or the logrotate configuration folder. the configuration file usually holds the global configuration and the pointer to the configuration folder. In common Linux distributions such as Ubuntu and Debian, the logrotate configuration file can be found:
and the configuration folder can be found:
The configuration folder is usually where you would add new log file configuration which are unique for a particular log file, or set of log files. For example:
/var/lib/docker/containers/*/*.log {
rotate 5
copytruncate
missingok
notifempty
compress
maxsize 200M
daily
}
- It represents the path where logrotate will monitor the log files with
.log file extension. - Store logs till 5 rotations, which means it will store maximum 5 files of old logs when rotation hits, and will discard oldest log when there are already 5 files.
- Truncate the original log file to zero size in place after creating a copy, instead of moving the old log file and optionally creating a new one. It can be used when some program cannot be told to close its log file and thus might continue writing (appending) to the previous log file forever. Note that there is a very small time slice between copying the file and truncating it, so some logging data might be lost.
missingok indicates if the log file is missing, do not generate an error, and move on the next file.- Then do not rotate the log if it is empty.
- Old version of logs are compressed.
- Rotate the log file if it exceeds 200Mb, regardless of the rotation time unit.
- Rotation process should happen daily.
Below will explain some commonly used configurations in detail.
Create a new empty template
To create a new logrotate configuration file, you need to create a new file in /etc/logrotate.d/. You will then need to add a reference to the log files you wish to rotate. This can be directly to a single file or use pattern matching to match a group of log files. The below example matches all log files in /var/log/myapp/ which have a .log extension:
You will need to add further commands to this template before it becomes useful.
Rotate based on log file size
Use the size keyword to rotate the log file when it exceeded a given file size. The below example rotates a file when it reaches 10 KB:
/var/log/myapp/*.log {
size 10k
}
Rotate based on time
You can rotate logs using the monthly, weekly, or daily keyword to create a new log based on duration. The keywords explain them selves, and they can be used in conjunction with the size keyword to rotate on which ever criteria is met first.
/var/log/myapp/*.log {
size 10k
weekly
}
Limit how many log files are kept after rotation by number
The rotate keyword allows us to specify how many old, rotated, log files are kept before logrotate deletes them. The rotate keyword requires an integer to specify the number of old log files.
/var/log/myapp/*.log {
size 10k
weekly
rotate 8
}
Limit how many files are kept after rotation by date
You can specify how long to keep rotated files using the maxage keyword. Any rotated log files which are older then the maxage will be deleted. The below example will keep rotated log files 49 days.
/var/log/myapp/*.log {
size 10k
weekly
maxage 49
}
Compress rotated log files
Log files which have been rotated can be compressed to save disk space. Gzip is used by default.
/var/log/myapp/*.log {
size 10k
weekly
rotate 8
compress
}
You can change the default gzip compression to another format by specifying the compresscmd command with a different executable to use. For example, change it to bzip2:
/var/log/myapp/*.log {
size 10k
weekly
rotate 8
compress
compresscmd /bin/bzip2
}
Ignore missing log files
If a log file does not exist when logrotate is running then an error will be thrown. You can use the keyword missingok to avoid this scenario.
/var/log/myapp/*.log {
size 10k
weekly
rotate 8
missingok
}
Continue writing to the same file after rotation
Usually when a log file is rotated, the log file is moved to a new location. Some applications may throw an error, and others may continue to write to the relocated file. The copytruncate keyword copies all the log in the file to a new file and then truncates the original file. This keeps the original log file in place and also allows rotation to continue.
/var/log/myapp/*.log {
size 10k
weekly
rotate 8
copytruncate
}
For complete details about logrotate configuration, please refer to man logrotate.
Clean up all log files
If you intend to delete all the log file in /var/log/, you can use the following script:
# echo empty content to log files
logs=`find /var/log -type f`
for item in $logs
do
> $item
done
# delete logrotate files
find /var/log -name "*.[0-9]*" -type f -delete
