HAProxy is a fantastic load balancer that can handle massive traffic loads, but without proper logging, you’re flying blind when things go sideways. While HAProxy has built-in logging capabilities, getting them to work seamlessly with rsyslog on CentOS 8 requires some specific configuration magic that isn’t always obvious from the docs. In this post, we’ll walk through the complete setup process, troubleshoot common gotchas, and get you proper visibility into your load balancer’s performance and behavior.

How HAProxy Logging Works with Rsyslog

HAProxy doesn’t write log files directly to disk like many other services. Instead, it sends log messages to a syslog daemon via UDP sockets, which then handles the actual file writing and log rotation. This design keeps HAProxy lean and fast since it doesn’t need to deal with disk I/O operations.

The communication happens through a UDP socket that rsyslog listens on. HAProxy sends structured log messages containing connection details, response times, status codes, and other critical metrics. Rsyslog then processes these messages according to your configuration rules and writes them to designated log files.

Here’s what a typical HAProxy log entry looks like:

Nov 15 14:23:45 loadbalancer haproxy[1234]: 192.168.1.100:52341 [15/Nov/2023:14:23:44.567] frontend backend/server1 0/0/1/23/24 200 1024 - - ---- 45/23/12/8/0 0/0 "GET /api/users HTTP/1.1"

Step-by-Step Configuration Guide

Configure Rsyslog for HAProxy

First, we need to configure rsyslog to listen for HAProxy messages and route them properly. Create a new rsyslog configuration file specifically for HAProxy:

sudo vim /etc/rsyslog.d/49-haproxy.conf

Add the following configuration:

# Enable UDP syslog reception on port 514
$ModLoad imudp
$UDPServerRun 514
$UDPServerAddress 127.0.0.1

# Create separate log files for HAProxy
local0.*    /var/log/haproxy.log
&stop

The &stop directive prevents HAProxy messages from appearing in other log files like /var/log/messages. If you want to keep them in the general log as well, remove this line.

Configure SELinux (Critical Step)

CentOS 8 ships with SELinux enabled by default, and it will block rsyslog from binding to UDP port 514. You need to allow this explicitly:

sudo setsebool -P rsyslog_can_network 1

You can verify the current status with:

getsebool rsyslog_can_network

Configure HAProxy Logging

Now modify your HAProxy configuration file (typically /etc/haproxy/haproxy.cfg) to enable logging:

global
    log 127.0.0.1:514 local0
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

defaults
    mode http
    log global
    option httplog
    option dontlognull
    option log-health-checks
    timeout connect 5000
    timeout client 50000
    timeout server 50000

frontend web_frontend
    bind *:80
    default_backend web_servers

backend web_servers
    balance roundrobin
    server web1 192.168.1.10:80 check
    server web2 192.168.1.11:80 check

Key logging directives explained:

• log 127.0.0.1:514 local0 – Sends logs to rsyslog on localhost port 514 using facility local0
• log global – Inherits the global log configuration
• option httplog – Enables detailed HTTP logging format
• option dontlognull – Prevents logging of connections with no data transfer
• option log-health-checks – Logs health check results

Restart Services and Test

Restart both services to apply the configuration:

sudo systemctl restart rsyslog
sudo systemctl restart haproxy

Verify that rsyslog is listening on the UDP port:

sudo netstat -ulnp | grep :514

You should see output like:

udp        0      0 127.0.0.1:514           0.0.0.0:*                           1234/rsyslogd

Test the logging by generating some traffic through HAProxy, then check the log file:

sudo tail -f /var/log/haproxy.log

Common Issues and Troubleshooting

No Log File Created

If /var/log/haproxy.log doesn’t exist, the most common causes are:

• SELinux blocking rsyslog – Make sure you ran the setsebool command
• Firewall blocking UDP 514 – Check with sudo firewall-cmd --list-ports
• Rsyslog not listening – Verify with netstat command above

Enable rsyslog debug mode to see what’s happening:

sudo rsyslogd -dn

Empty Log File

If the file exists but remains empty:

• Check HAProxy is actually sending logs: sudo tcpdump -i lo port 514
• Verify HAProxy configuration syntax: sudo haproxy -c -f /etc/haproxy/haproxy.cfg
• Check facility mismatch between HAProxy and rsyslog configs

Permission Issues

Sometimes rsyslog can’t write to the log file due to permission issues:

sudo chown syslog:adm /var/log/haproxy.log
sudo chmod 640 /var/log/haproxy.log

Advanced Configuration Options

Custom Log Format

You can customize the log format using the capture directive and custom log formats:

defaults
    mode http
    log global
    option httplog
    capture request header Host len 32
    capture request header User-Agent len 64
    capture response header Content-Type len 32

Separate Log Files by Service

For complex setups, you might want different log files for different services. Modify the rsyslog configuration:

# Use different facilities for different services
local0.*    /var/log/haproxy-web.log
local1.*    /var/log/haproxy-api.log
local2.*    /var/log/haproxy-admin.log
&stop

Then configure different HAProxy sections to use different facilities:

frontend web_frontend
    bind *:80
    log 127.0.0.1:514 local0
    default_backend web_servers

frontend api_frontend
    bind *:8080
    log 127.0.0.1:514 local1
    default_backend api_servers

Performance Considerations and Best Practices

Log Rotation

HAProxy can generate substantial log volumes under high load. Set up proper log rotation:

sudo vim /etc/logrotate.d/haproxy

/var/log/haproxy.log {
    daily
    rotate 52
    missingok
    notifempty
    compress
    delaycompress
    postrotate
        /bin/kill -HUP `cat /var/run/rsyslogd.pid 2> /dev/null` 2> /dev/null || true
    endrotate
}

Performance Impact Comparison

Logging Level	CPU Impact	Disk I/O	Log Volume/Hour	Use Case
No logging	0%	None	0 MB	High-performance prod (not recommended)
Error only	<0.1%	Very low	<10 MB	Minimal monitoring
Standard HTTP log	0.2-0.5%	Moderate	100-500 MB	Production recommended
Full debug	1-2%	High	1-5 GB	Troubleshooting only

Security Considerations

• Bind to localhost only – Never expose the syslog UDP port externally
• Log file permissions – Restrict access to log files containing potentially sensitive data
• Log retention – Don’t keep logs longer than required for compliance
• Sensitive data filtering – Be careful not to log authentication tokens or personal data

Real-World Use Cases and Examples

E-commerce Platform Monitoring

For an e-commerce site handling thousands of requests per minute, you’d want detailed logging to track:

defaults
    mode http
    log global
    option httplog
    option log-health-checks
    capture request header X-Forwarded-For len 64
    capture request header Referer len 128
    capture response header Set-Cookie len 128

API Gateway Logging

For API services, focus on response times and error tracking:

backend api_servers
    mode http
    balance roundrobin
    option httpchk GET /health
    http-check expect status 200
    server api1 10.0.1.10:8080 check inter 30s
    server api2 10.0.1.11:8080 check inter 30s
    
    # Log slow responses
    timeout server 5000
    log 127.0.0.1:514 local1 info

Log Analysis with Standard Tools

Once logging is working, you can analyze the data with standard Unix tools:

# Top 10 IP addresses by request count
awk '{print $6}' /var/log/haproxy.log | sort | uniq -c | sort -nr | head -10

# Average response time by hour
awk '{print $9, $11}' /var/log/haproxy.log | grep -v "^-" | 
awk '{sum+=$2; count++} END {print "Average response time:", sum/count "ms"}'

# Error rate analysis
grep " 5[0-9][0-9] " /var/log/haproxy.log | wc -l

Integration with Monitoring Systems

HAProxy logs integrate well with various monitoring solutions:

• ELK Stack – Use Filebeat to ship logs to Elasticsearch
• Prometheus – Use HAProxy Exporter alongside log analysis
• Grafana – Create dashboards from log metrics
• Nagios/Zabbix – Alert on error patterns in logs

For more detailed information on HAProxy logging options, check the official HAProxy documentation and the rsyslog configuration guide.

With proper logging configured, you’ll have the visibility needed to troubleshoot issues, optimize performance, and maintain reliable service delivery. The initial setup investment pays dividends when you need to diagnose problems quickly or demonstrate SLA compliance.

---

---

This article incorporates information and material from various online sources. We acknowledge and appreciate the work of all original authors, publishers, and websites. While every effort has been made to appropriately credit the source material, any unintentional oversight or omission does not constitute a copyright infringement. All trademarks, logos, and images mentioned are the property of their respective owners. If you believe that any content used in this article infringes upon your copyright, please contact us immediately for review and prompt action.

This article is intended for informational and educational purposes only and does not infringe on the rights of the copyright owners. If any copyrighted material has been used without proper credit or in violation of copyright laws, it is unintentional and we will rectify it promptly upon notification. Please note that the republishing, redistribution, or reproduction of part or all of the contents in any form is prohibited without express written permission from the author and website owner. For permissions or further inquiries, please contact us.