Master Linux Log Management: rsyslog, systemd‑journal, and Logrotate Explained

Linux Log Management Introduction

Table of Contents

rsyslog log management

Common log files

Log levels

Log types

Configure storage paths for different types/levels

Configure the local machine as a log server

Configure the local machine as a log receiver

Secure rsyslog configuration

Manually send syslog logs

systemd-journal log management

Journal configuration file

Journal log analysis

Persist journal logs

Log file rotation

Linux currently provides two main logging services: the traditional rsyslog and the newer systemd‑journal . rsyslog records all collected logs into separate files under /var/log, while systemd‑journal stores logs in a structured binary file and can capture kernel messages, early‑boot logs, daemon output, and even rsyslog messages.

rsyslog Log Management

Common Log Files

System log files

/var/log/wtmp – login and shutdown events

/var/log/boot.log – boot process messages

/var/log/messages – general system and service logs

/var/log/secure – security‑related logs

/var/log/lastlog – successful login timestamps

/var/log/btmp – failed login attempts

Application log files

/var/log/xfer.log – FTP logs

/var/log/httpd/access_log – HTTP access logs

/var/log/httpd/error_log – HTTP error logs

/var/log/yum.log – yum package manager logs

Log Levels

emerg – system is unusable (kernel panic)

alert – immediate action required (e.g., database corruption)

crit – critical conditions (e.g., disk failure)

err – error conditions (service start/stop failures)

warning – warning messages (misconfiguration)

notice – normal but significant condition

info – informational messages

debug – debugging information

Log Types

auth – authentication logs

authpriv – privileged authentication logs

mail – mail system logs

cron – scheduled task logs

kern – kernel logs

news – system update logs

user – user‑level logs

Configuring Storage Paths for Different Types/Levels

rsyslog configuration resides in /etc/rsyslog.conf. Example entries:

vim /etc/rsyslog.conf
*.info;mail.none;authpriv.none;cron.none    /var/log/messages
authpriv.*    /var/log/secure
mail.*    /var/log/maillog
cron.*    /var/log/cron
uucp,news.crit    /var/log/spooler
local7.*    /var/log/boot.log
.emerg    :omusrmsg:*

Configure the Local Machine as a Log Server

To forward logs to a remote server, edit /etc/rsyslog.recive:

vim /etc/rsyslog.recive
*.info;mail.none;authpriv.none;cron.none    @<remote‑log‑server>

Restart the service with systemctl restart rsyslog.recive and test by sending logs from a client (e.g., install vsftpd on the client and monitor tail -f /var/log/messages on the server).

Configure the Local Machine as a Log Receiver

Enable UDP/TCP input modules and open firewall ports:

vim /etc/rsyslog.conf
module(load="imudp")
input(type="imudp" port="514")
module(load="imtcp")
input(type="imtcp" port="514")
systemctl restart rsyslog.recive
firewall-cmd --add-port=514/tcp --permanent
firewall-cmd --add-port=514/udp --permanent
firewall-cmd --reload
lsof -i:514

Secure rsyslog Configuration

Set immutable append‑only attribute on log files to prevent deletion:

chattr +a /var/log/messages   # lock file for append‑only
lsattr /var/log/messages      # verify attributes

Manually Send Syslog Logs

Use the logger command. Example:

logger -p local7.notice "Log entry created on host"
# -p specifies facility.level (default is user.notice)

Common facilities include user, uucp, local0‑7, auth, authpriv, cron, daemon, kern, lpr, mail, news, syslog.

systemd‑journal Log Management

Journal logs are stored in a binary file, defaulting to /run/log/journal (volatile). To make storage persistent, use /var/log/journal.

Journal Configuration File

The file is /etc/systemd/journald.conf. Key options:

Storage=auto          # auto selects volatile or persistent
SystemMaxUse=        # total disk space limit
SystemMaxFileSize=   # per‑file size limit
SystemMaxFiles=100    # max number of journal files
RuntimeMaxUse=       # memory usage limit
MaxFileSec=           # rotation interval
MaxRetentionSec=     # maximum retention time (e.g., 1month)
ForwardToSyslog=no
ForwardToKMsg=no
ForwardToConsole=no
ForwardToWall=yes
MaxLevelStore=debug
MaxLevelSyslog=debug
MaxLevelKMsg=notice
MaxLevelConsole=info
MaxLevelWall=emerg

Journal Log Analysis

Use journalctl to query logs:

journalctl -n 10                     # latest 10 entries
journalctl --since "2023-04-20 19:40"   # logs after a specific time
journalctl -u httpd                  # logs for a specific service
journalctl -f                        # follow logs in real time
journalctl --disk-usage              # show journal size
journalctl --vacuum-size=1G          # limit total size
journalctl --vacuum-time=1h           # limit retention time
journalctl -p err                    # filter by priority
journalctl -o json                   # output format

Persisting Journal Logs

Two methods to enable persistent storage:

# Method 1
vim /etc/systemd/journald.conf
Storage=persistent
systemctl restart systemd-journald.service

# Method 2
vim /etc/systemd/journald.conf
Storage=auto
mkdir /var/log/journal
chgrp systemd-journal /var/log/journal
chmod 2775 /var/log/journal
systemctl restart systemd-journald.service

Log File Rotation

Use logrotate to rotate logs, preventing excessive disk usage.

logrotate Workflow

Identify log files to rotate.

Check if they exceed size limits.

Compress oversized logs.

Optionally delete old archives.

Reset the original log file and continue logging.

logrotate Configuration

Main configuration file: /etc/logrotate.conf. Example settings:

weekly               # rotate weekly
rotate 4              # keep 4 weeks of archives
create                # create new empty log files after rotation
dateext               # append date to rotated files
compress              # compress old logs
include /etc/logrotate.d   # include additional config files