BLOG POSTS

MangoHost Blog / How to Configure Logging and Log Rotation in Nginx on an Ubuntu VPS

How to Configure Logging and Log Rotation in Nginx on an Ubuntu VPS

Nginx logging is one of those unglamorous but absolutely crucial aspects of web server management that’ll save your bacon when things go sideways. If you’re running an Ubuntu VPS, knowing how to properly configure access logs, error logs, and set up automatic log rotation is essential for maintaining a healthy server, debugging issues, and keeping your disk space from getting eaten alive by massive log files. In this guide, we’ll walk through the complete process of setting up comprehensive logging in Nginx, configuring logrotate to manage file sizes, customizing log formats for different use cases, and troubleshooting the most common gotchas that trip up both newcomers and seasoned admins.

Understanding Nginx Logging Architecture

Nginx operates with two main types of logs: access logs that record every request hitting your server, and error logs that capture server errors, warnings, and debugging information. By default on Ubuntu, these are stored in /var/log/nginx/, but the real power comes from understanding how to customize them for your specific needs.

The logging system works at multiple levels. You can configure logging globally in the main context, per server block, or even per location. This hierarchical approach means you can have different logging strategies for different sites or even different sections of the same site.

# Default Ubuntu Nginx logging locations
/var/log/nginx/access.log    # All access requests
/var/log/nginx/error.log     # Server errors and warnings

Log formats in Nginx are highly customizable using variables. The default format captures basic information, but you can extend it to include response times, upstream server info, user agents, or custom headers that matter for your application.

Step-by-Step Logging Configuration

Let’s start with the basics and build up to more advanced configurations. First, check your current Nginx configuration to see what’s already in place:

sudo nginx -t
sudo cat /etc/nginx/nginx.conf | grep -A 5 -B 5 log

Here’s a comprehensive logging setup that covers most real-world scenarios:

# Add this to /etc/nginx/nginx.conf in the http block
http {
    # Define custom log formats
    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
                    '$status $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" "$http_x_forwarded_for"';
    
    log_format detailed '$remote_addr - $remote_user [$time_local] "$request" '
                        '$status $body_bytes_sent "$http_referer" '
                        '"$http_user_agent" "$http_x_forwarded_for" '
                        'rt=$request_time uct="$upstream_connect_time" '
                        'uht="$upstream_header_time" urt="$upstream_response_time"';
    
    log_format json escape=json '{'
                    '"timestamp":"$time_local",'
                    '"remote_addr":"$remote_addr",'
                    '"remote_user":"$remote_user",'
                    '"request":"$request",'
                    '"status":"$status",'
                    '"body_bytes_sent":"$body_bytes_sent",'
                    '"http_referer":"$http_referer",'
                    '"http_user_agent":"$http_user_agent",'
                    '"request_time":"$request_time"'
                    '}';

    # Global access log
    access_log /var/log/nginx/access.log main;
    
    # Error log with info level
    error_log /var/log/nginx/error.log info;
}

For individual server blocks, you can override or supplement the global logging:

server {
    listen 80;
    server_name example.com;
    
    # Site-specific access log with detailed format
    access_log /var/log/nginx/example.com-access.log detailed;
    
    # Separate error log for this site
    error_log /var/log/nginx/example.com-error.log warn;
    
    # Log API requests separately
    location /api/ {
        access_log /var/log/nginx/api-access.log json;
        proxy_pass http://backend;
    }
    
    # Don't log static assets (optional)
    location ~* \.(jpg|jpeg|png|gif|ico|css|js)$ {
        access_log off;
        expires 1y;
    }
}

After making changes, always test the configuration and reload:

sudo nginx -t
sudo systemctl reload nginx

Implementing Log Rotation with Logrotate

Ubuntu comes with logrotate pre-installed, and there’s usually a default configuration for Nginx. However, the default setup might not fit your needs, especially if you’re dealing with high-traffic sites or have specific retention requirements.

First, check the existing logrotate configuration:

cat /etc/logrotate.d/nginx

Here’s a more robust logrotate configuration that handles various scenarios:

# Create or edit /etc/logrotate.d/nginx
/var/log/nginx/*.log {
    daily
    missingok
    rotate 52
    compress
    delaycompress
    notifempty
    create 0644 www-data adm
    sharedscripts
    prerotate
        if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
            run-parts /etc/logrotate.d/httpd-prerotate; \
        fi \
    endscript
    postrotate
        invoke-rc.d nginx rotate >/dev/null 2>&1
    endscript
}

# High-traffic sites might need hourly rotation
/var/log/nginx/high-traffic-*.log {
    hourly
    missingok
    rotate 168
    compress
    delaycompress
    notifempty
    create 0644 www-data adm
    sharedscripts
    postrotate
        invoke-rc.d nginx rotate >/dev/null 2>&1
    endscript
}

You can test logrotate manually to ensure it’s working correctly:

# Force logrotate to run (for testing)
sudo logrotate -f /etc/logrotate.d/nginx

# Debug logrotate issues
sudo logrotate -d /etc/logrotate.d/nginx

Real-World Use Cases and Examples

Different applications require different logging strategies. Here are some practical configurations I’ve used in production environments:

E-commerce Site with Performance Monitoring:

log_format ecommerce '$remote_addr - $remote_user [$time_local] "$request" '
                     '$status $body_bytes_sent "$http_referer" '
                     '"$http_user_agent" "$http_x_forwarded_for" '
                     'rt=$request_time uct="$upstream_connect_time" '
                     'uht="$upstream_header_time" urt="$upstream_response_time" '
                     'cart_id="$cookie_cart_id" user_id="$cookie_user_id"';

server {
    server_name shop.example.com;
    access_log /var/log/nginx/shop-access.log ecommerce buffer=32k flush=5s;
    
    # Log slow requests separately
    location / {
        access_log /var/log/nginx/shop-slow.log ecommerce if=$slow_request;
        set $slow_request 0;
        if ($request_time ~ "^[1-9]") {
            set $slow_request 1;
        }
        proxy_pass http://app_backend;
    }
}

API Server with JSON Logging for ELK Stack:

log_format api_json escape=json '{'
    '"@timestamp":"$time_iso8601",'
    '"remote_addr":"$remote_addr",'
    '"request_method":"$request_method",'
    '"request_uri":"$request_uri",'
    '"status":"$status",'
    '"body_bytes_sent":"$body_bytes_sent",'
    '"request_time":"$request_time",'
    '"upstream_response_time":"$upstream_response_time",'
    '"api_key":"$http_x_api_key",'
    '"user_agent":"$http_user_agent"'
'}';

server {
    server_name api.example.com;
    access_log /var/log/nginx/api-json.log api_json;
    
    location /v1/ {
        # Rate limit logging
        access_log /var/log/nginx/api-rate-limit.log api_json if=$rate_limited;
        limit_req zone=api burst=10 nodelay;
        proxy_pass http://api_backend;
    }
}

Comparison of Logging Approaches

Approach	Pros	Cons	Best For
Default Text Logs	Human readable, small size, universal compatibility	Harder to parse programmatically, limited structure	Small sites, debugging, simple monitoring
JSON Logging	Easy parsing, structured data, great for analytics	Larger file sizes, less human readable	APIs, microservices, ELK/EFK stacks
Custom Formats	Tailored to specific needs, optimal information density	Requires custom parsing tools, maintenance overhead	Specialized applications, performance monitoring
Syslog Integration	Centralized logging, better for distributed systems	Additional complexity, potential single point of failure	Multi-server setups, compliance requirements

Performance Considerations and Best Practices

Logging can impact performance, especially on high-traffic sites. Here are some optimization strategies:

Use buffering: The buffer=32k flush=5s directive reduces disk I/O by batching log writes
Selective logging: Use conditional logging with if= to log only what matters
Separate disks: Store logs on a different disk than your web content when possible
Log compression: Enable gzip compression in logrotate to save disk space
Monitor disk space: Set up alerts for log directory disk usage

Here’s a performance-optimized logging configuration:

http {
    # Use buffering for high-traffic sites
    access_log /var/log/nginx/access.log main buffer=64k flush=5s;
    
    # Log errors but not debug info in production
    error_log /var/log/nginx/error.log warn;
    
    server {
        # Conditional logging for different response codes
        map $status $error_log {
            ~^[23] 0;
            default 1;
        }
        
        access_log /var/log/nginx/errors-only.log main if=$error_log;
        
        # Don't log health checks
        location /health {
            access_log off;
            return 200 "OK";
        }
    }
}

Troubleshooting Common Issues

Here are the most frequent problems you’ll encounter and how to solve them:

Permission Issues:

# Fix log file permissions
sudo chown www-data:adm /var/log/nginx/*.log
sudo chmod 644 /var/log/nginx/*.log

# Ensure nginx can create new log files
sudo chown www-data:adm /var/log/nginx/

Logs Not Rotating:

# Check logrotate status
cat /var/lib/logrotate/status | grep nginx

# Manually test rotation
sudo logrotate -d /etc/logrotate.d/nginx

# Check for errors in logrotate
sudo cat /var/log/syslog | grep logrotate

High Disk Usage:

# Find large log files
sudo find /var/log/nginx/ -type f -size +100M -exec ls -lh {} \;

# Emergency log cleanup (use with caution)
sudo truncate -s 0 /var/log/nginx/access.log
sudo systemctl reload nginx

Missing Log Entries:

Check if logging is disabled in specific locations, verify file permissions, and ensure the log format doesn’t have syntax errors that would cause Nginx to fail silently.

For advanced monitoring and log analysis, consider integrating with tools like Filebeat for shipping logs to Elasticsearch, or Fluentd for real-time log processing. The official Nginx logging documentation provides comprehensive details on all available variables and configuration options.

Remember that good logging is about finding the right balance between having enough information to troubleshoot issues and not overwhelming your storage or analysis systems. Start with the basics, monitor what you actually use, and adjust your configuration based on real-world usage patterns.

This article incorporates information and material from various online sources. We acknowledge and appreciate the work of all original authors, publishers, and websites. While every effort has been made to appropriately credit the source material, any unintentional oversight or omission does not constitute a copyright infringement. All trademarks, logos, and images mentioned are the property of their respective owners. If you believe that any content used in this article infringes upon your copyright, please contact us immediately for review and prompt action.

This article is intended for informational and educational purposes only and does not infringe on the rights of the copyright owners. If any copyrighted material has been used without proper credit or in violation of copyright laws, it is unintentional and we will rectify it promptly upon notification. Please note that the republishing, redistribution, or reproduction of part or all of the contents in any form is prohibited without express written permission from the author and website owner. For permissions or further inquiries, please contact us.