BLOG POSTS

MangoHost Blog / How to View and Configure Linux Logs on Ubuntu, Debian, and CentOS

How to View and Configure Linux Logs on Ubuntu, Debian, and CentOS

Linux logging is the backbone of system monitoring and troubleshooting, yet many developers and sysadmins barely scratch the surface of what’s possible with log management across different distributions. Whether you’re running Ubuntu, Debian, or CentOS, understanding how to effectively view, configure, and manipulate logs can mean the difference between quickly resolving issues and spending hours hunting down problems. This guide covers everything from basic log viewing commands to advanced configuration techniques, distribution-specific quirks, and performance optimization strategies that’ll make you more effective at managing Linux systems.

Understanding Linux Logging Architecture

Modern Linux distributions use systemd’s journald alongside traditional syslog implementations, creating a dual logging system that can initially confuse newcomers. Here’s how it actually works:

systemd-journald: Collects logs from kernel, system services, and applications, storing them in binary format
rsyslog/syslog-ng: Traditional text-based logging daemon that often receives forwarded messages from journald
Log files: Usually stored in /var/log/ with specific naming conventions per service

The key difference between distributions lies in their default configurations and which logging daemon takes precedence. Ubuntu and Debian typically favor rsyslog with journald forwarding, while CentOS (now Rocky/AlmaLinux) leans more heavily on journald in recent versions.

Distribution	Default Logger	Journal Persistence	Key Config Files
Ubuntu 20.04+	rsyslog + journald	Volatile (RAM only)	/etc/rsyslog.conf, /etc/systemd/journald.conf
Debian 11+	rsyslog + journald	Volatile (RAM only)	/etc/rsyslog.conf, /etc/systemd/journald.conf
CentOS 8/Rocky 9	journald + rsyslog	Persistent (disk)	/etc/systemd/journald.conf, /etc/rsyslog.conf

Essential Log Viewing Commands

Let’s start with the bread-and-butter commands you’ll use daily. The journalctl command is your primary tool for systemd-based logging:

# View all logs (paginated)
journalctl

# Follow logs in real-time (like tail -f)
journalctl -f

# Show logs from last boot
journalctl -b

# Filter by service
journalctl -u nginx.service

# Show logs from specific time range
journalctl --since "2024-01-01 10:00:00" --until "2024-01-01 11:00:00"

# Display logs with specific priority (0=emergency, 7=debug)
journalctl -p err

# Show kernel messages only
journalctl -k

# Output in JSON format for parsing
journalctl -o json-pretty

For traditional log files, these commands remain essential:

# Real-time log monitoring
tail -f /var/log/syslog
tail -f /var/log/messages  # CentOS/RHEL

# Search through logs
grep "error" /var/log/syslog
grep -i "failed" /var/log/auth.log

# Show last 100 lines with line numbers
tail -n 100 /var/log/syslog | cat -n

# Monitor multiple log files simultaneously
multitail /var/log/syslog /var/log/auth.log

Distribution-Specific Configuration

Ubuntu and Debian Configuration

Ubuntu and Debian handle logging through rsyslog by default, with journald forwarding messages. Here’s how to configure both systems:

# Install rsyslog if not present
sudo apt update
sudo apt install rsyslog

# Main rsyslog configuration
sudo nano /etc/rsyslog.conf

# Enable journal forwarding (usually enabled by default)
# Add this line if missing:
# $ModLoad imjournal
# $IMJournalStateFile imjournal.state

Key configuration files and their purposes:

/etc/rsyslog.conf: Main rsyslog configuration
/etc/rsyslog.d/: Additional configuration files (processed alphabetically)
/etc/systemd/journald.conf: systemd journal configuration
/etc/logrotate.d/: Log rotation settings

To enable persistent journald logging on Ubuntu/Debian:

# Create journal directory
sudo mkdir -p /var/log/journal

# Set proper permissions
sudo systemd-tmpfiles --create --prefix /var/log/journal

# Edit journald config
sudo nano /etc/systemd/journald.conf

# Uncomment and set:
# Storage=persistent
# SystemMaxUse=500M

# Restart journald
sudo systemctl restart systemd-journald

CentOS/RHEL Configuration

CentOS and RHEL systems prioritize journald with persistent storage by default:

# Check journal storage status
sudo journalctl --disk-usage

# Configure journal limits
sudo nano /etc/systemd/journald.conf

# Recommended settings for production:
# Storage=persistent
# SystemMaxUse=1G
# SystemKeepFree=500M
# RuntimeMaxUse=100M
# MaxRetentionSec=1month

For rsyslog configuration on CentOS:

# Install rsyslog (usually pre-installed)
sudo dnf install rsyslog

# Main configuration
sudo nano /etc/rsyslog.conf

# Enable and start rsyslog
sudo systemctl enable rsyslog
sudo systemctl start rsyslog

Advanced Log Configuration Examples

Here are some practical configuration examples for common scenarios:

Custom Application Logging

Create a custom rsyslog configuration for your application:

# Create custom config file
sudo nano /etc/rsyslog.d/50-myapp.conf

# Add these rules:
# Log myapp messages to separate file
:programname, isequal, "myapp" /var/log/myapp.log
# Don't log myapp messages to syslog
:programname, isequal, "myapp" stop

# Restart rsyslog
sudo systemctl restart rsyslog

Remote Logging Setup

Configure centralized logging (sender side):

# Edit rsyslog config
sudo nano /etc/rsyslog.conf

# Add remote logging (UDP)
*.* @log-server.example.com:514

# Or use TCP for reliability
*.* @@log-server.example.com:514

# Restart rsyslog
sudo systemctl restart rsyslog

For the receiving server:

# Edit rsyslog config
sudo nano /etc/rsyslog.conf

# Uncomment these lines to enable UDP reception:
# module(load="imudp")
# input(type="imudp" port="514")

# Or for TCP:
# module(load="imtcp")
# input(type="imtcp" port="514")

# Restart and configure firewall
sudo systemctl restart rsyslog
sudo firewall-cmd --permanent --add-port=514/udp
sudo firewall-cmd --reload

Log Rotation and Management

Proper log rotation prevents disk space issues and maintains system performance. Here’s how to configure logrotate effectively:

# Check current logrotate configuration
sudo nano /etc/logrotate.conf

# Create custom rotation for your app
sudo nano /etc/logrotate.d/myapp

# Example configuration:
/var/log/myapp.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
    postrotate
        systemctl reload myapp
    endscript
}

# Test logrotate configuration
sudo logrotate -d /etc/logrotate.d/myapp

# Force rotation (for testing)
sudo logrotate -f /etc/logrotate.d/myapp

For journald, configure size limits in /etc/systemd/journald.conf:

[Journal]
# Limit journal size to 500MB
SystemMaxUse=500M

# Keep at least 1GB free on filesystem
SystemKeepFree=1G

# Limit individual journal files to 50MB
SystemMaxFileSize=50M

# Keep logs for 1 month maximum
MaxRetentionSec=1month

# Don't forward to syslog if you want journald-only
ForwardToSyslog=no

Performance Optimization and Best Practices

Logging can significantly impact system performance if not configured properly. Here are optimization strategies:

I/O Optimization

Reduce disk I/O by configuring appropriate buffering:

# In /etc/rsyslog.conf, add:
# Buffer size for file outputs (default 64k)
$OMFileFlushInterval 10
$OMFileIOBufferSize 128k

# Reduce sync frequency (trades durability for performance)
$OMFileFlushOnTXEnd off

Journal Performance Tuning

# In /etc/systemd/journald.conf:
[Journal]
# Compress logs to save space and I/O
Compress=yes

# Split journal files by user (reduces lock contention)
SplitMode=uid

# Sync to disk less frequently (potential data loss on crash)
SyncIntervalSec=60s

Log Level Optimization

Configure appropriate log levels to reduce noise:

# In /etc/rsyslog.conf, filter out debug messages in production:
# Log everything except debug to syslog
*.info;mail.none;authpriv.none;cron.none /var/log/messages

# For systemd services, set log level in service files:
sudo systemctl edit myapp.service

# Add:
[Service]
Environment="LOG_LEVEL=info"
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp

Troubleshooting Common Issues

Here are the most frequent problems you’ll encounter and their solutions:

Journal Storage Issues

# Check journal disk usage
journalctl --disk-usage

# Clean old journal entries
sudo journalctl --vacuum-time=30d
sudo journalctl --vacuum-size=500M

# If journal is corrupted
sudo systemctl stop systemd-journald
sudo rm -rf /var/log/journal/*
sudo systemctl start systemd-journald

Missing Logs

When logs aren’t appearing where expected:

# Check if journald is forwarding to rsyslog
sudo systemctl status systemd-journald
sudo systemctl status rsyslog

# Verify rsyslog is receiving journal messages
sudo rsyslogd -N1  # Test configuration

# Check for SELinux issues (CentOS/RHEL)
sudo sealert -a /var/log/audit/audit.log

# Verify file permissions
ls -la /var/log/
sudo chmod 640 /var/log/syslog
sudo chown syslog:adm /var/log/syslog

High Memory Usage

If journald is consuming too much memory:

# Check current journal memory usage
sudo systemctl status systemd-journald

# Limit runtime journal size
sudo nano /etc/systemd/journald.conf

# Add:
[Journal]
RuntimeMaxUse=50M
RuntimeKeepFree=100M
RuntimeMaxFileSize=10M

# Restart journald
sudo systemctl restart systemd-journald

Real-World Use Cases and Examples

Web Server Log Analysis

Monitoring nginx or Apache logs across different distributions:

# Ubuntu/Debian - nginx logs
sudo journalctl -u nginx -f --since "1 hour ago"
tail -f /var/log/nginx/error.log

# Filter for specific error codes
grep "404" /var/log/nginx/access.log | tail -20

# CentOS - httpd logs
sudo journalctl -u httpd -p err --since today
tail -f /var/log/httpd/error_log

Database Server Monitoring

# MySQL/MariaDB logs across distributions
# Ubuntu/Debian
sudo journalctl -u mysql -f
tail -f /var/log/mysql/error.log

# CentOS
sudo journalctl -u mariadb -f
tail -f /var/log/mariadb/mariadb.log

# PostgreSQL
sudo journalctl -u postgresql -f --since "10 minutes ago"

System Security Monitoring

Track authentication and security events:

# Monitor SSH login attempts
sudo journalctl -u ssh -f | grep "Failed"

# Ubuntu/Debian auth logs
tail -f /var/log/auth.log | grep -i "failed\|invalid"

# CentOS secure logs
tail -f /var/log/secure | grep -i "failed\|invalid"

# Show all failed sudo attempts
sudo journalctl --since today | grep "sudo.*FAILED"

Integration with Monitoring Tools

Modern logging integrates well with monitoring and alerting systems. Here are some practical examples:

Fluentd/Fluent Bit Integration

# Install fluent-bit
curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

# Configure to read from journal
sudo nano /etc/fluent-bit/fluent-bit.conf

[INPUT]
    Name systemd
    Tag host.*
    Systemd_Filter _SYSTEMD_UNIT=nginx.service

[OUTPUT]
    Name forward
    Match *
    Host log-aggregator.example.com
    Port 24224

Prometheus Node Exporter Log Metrics

# Install node_exporter with textfile collector
wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz

# Create log metrics script
sudo nano /usr/local/bin/log_metrics.sh

#!/bin/bash
echo "# HELP log_errors_total Total number of error log entries" > /var/lib/node_exporter/textfile_collector/logs.prom
echo "# TYPE log_errors_total counter" >> /var/lib/node_exporter/textfile_collector/logs.prom
journalctl --since "1 minute ago" -p err --output=json | wc -l | awk '{print "log_errors_total " $1}' >> /var/lib/node_exporter/textfile_collector/logs.prom

# Add to crontab
* * * * * /usr/local/bin/log_metrics.sh

For production environments, consider deploying your logging infrastructure on reliable hosting. VPS solutions provide excellent performance for centralized log servers, while dedicated servers offer the resources needed for high-volume log processing and retention.

Understanding Linux logging across Ubuntu, Debian, and CentOS distributions gives you the foundation to build robust monitoring and debugging workflows. The key is starting with basic log viewing commands, then gradually implementing more sophisticated configurations as your infrastructure grows. Remember that effective logging is about finding the right balance between detail and performance – capture what you need to troubleshoot issues without overwhelming your systems or storage capacity.

For additional information, check the official documentation: systemd journalctl manual and rsyslog documentation.

This article incorporates information and material from various online sources. We acknowledge and appreciate the work of all original authors, publishers, and websites. While every effort has been made to appropriately credit the source material, any unintentional oversight or omission does not constitute a copyright infringement. All trademarks, logos, and images mentioned are the property of their respective owners. If you believe that any content used in this article infringes upon your copyright, please contact us immediately for review and prompt action.

This article is intended for informational and educational purposes only and does not infringe on the rights of the copyright owners. If any copyrighted material has been used without proper credit or in violation of copyright laws, it is unintentional and we will rectify it promptly upon notification. Please note that the republishing, redistribution, or reproduction of part or all of the contents in any form is prohibited without express written permission from the author and website owner. For permissions or further inquiries, please contact us.