BLOG POSTS

MangoHost Blog / How to Create a High Availability Setup with Heartbeat and Reserved IPs on Ubuntu 24

How to Create a High Availability Setup with Heartbeat and Reserved IPs on Ubuntu 24

High availability (HA) setups are crucial for modern web applications where downtime translates directly to lost revenue and user trust. Heartbeat, combined with reserved/floating IPs, provides an elegant solution for creating failover mechanisms that automatically redirect traffic when primary servers fail. This guide walks you through implementing a robust HA configuration on Ubuntu 24, covering everything from initial setup to troubleshooting common issues that’ll inevitably pop up during your deployment.

How High Availability with Heartbeat Works

Heartbeat operates on a simple but effective principle: two or more servers continuously monitor each other’s health through network communications. When the primary server becomes unresponsive, the secondary server automatically takes over by claiming the reserved IP address and continuing service delivery.

The architecture consists of several key components:

Heartbeat daemon – Monitors cluster member health and manages failover
Reserved/Floating IP – A virtual IP that can move between servers
Resource scripts – Handle service startup, shutdown, and IP migration
Split-brain protection – Prevents both nodes from claiming resources simultaneously

Unlike more complex solutions like Pacemaker or Keepalived, Heartbeat focuses specifically on simple two-node clusters, making it ideal for small to medium-scale deployments where you need reliability without excessive complexity.

Prerequisites and Environment Setup

Before diving into configuration, ensure you have two Ubuntu 24 servers with root access and a reserved IP from your hosting provider. For this tutorial, we’ll use:

Primary server: 192.168.1.10 (node1)
Secondary server: 192.168.1.11 (node2)
Reserved IP: 192.168.1.100

Both servers need consistent hostnames and network connectivity. Update your systems first:

sudo apt update && sudo apt upgrade -y
sudo hostnamectl set-hostname node1  # or node2 for secondary

Configure the hosts file on both servers:

sudo nano /etc/hosts

Add these entries:

192.168.1.10    node1
192.168.1.11    node2

Installing and Configuring Heartbeat

Ubuntu 24 requires installing Heartbeat from the universe repository:

sudo apt install heartbeat -y

Heartbeat configuration involves three main files in /etc/ha.d/:

Main Configuration (ha.cf)

Create the primary configuration file:

sudo nano /etc/ha.d/ha.cf

Add the following configuration:

logfile /var/log/ha-log
logfacility local0
keepalive 2
deadtime 30
warntime 10
initdead 120
udpport 694
ucast eth0 192.168.1.10
ucast eth0 192.168.1.11
auto_failback on
node node1
node node2
use_logd yes

Authentication Keys (authkeys)

Security is critical for cluster communications:

sudo nano /etc/ha.d/authkeys

auth 3
3 md5 your-super-secret-key-here-change-this

Set proper permissions:

sudo chmod 600 /etc/ha.d/authkeys

Resource Configuration (haresources)

Define which resources Heartbeat manages:

sudo nano /etc/ha.d/haresources

node1 IPaddr::192.168.1.100/24/eth0 nginx

This configuration tells Heartbeat that node1 is the preferred primary, managing the reserved IP 192.168.1.100 and the nginx service.

Setting Up Reserved IP Management

Most cloud providers offer reserved/floating IPs through their APIs. Create a script to handle IP migration:

sudo nano /etc/ha.d/resource.d/FloatingIP

#!/bin/bash
# Floating IP management script for cloud providers

FLOATING_IP="192.168.1.100"
INTERFACE="eth0"

case "$1" in
    start)
        # Add the floating IP to this server
        ip addr add $FLOATING_IP/24 dev $INTERFACE
        # Send gratuitous ARP to update network switches
        arping -c 3 -I $INTERFACE $FLOATING_IP
        exit 0
        ;;
    stop)
        # Remove the floating IP from this server
        ip addr del $FLOATING_IP/24 dev $INTERFACE 2>/dev/null
        exit 0
        ;;
    status)
        # Check if IP is assigned to this interface
        if ip addr show $INTERFACE | grep -q $FLOATING_IP; then
            exit 0
        else
            exit 1
        fi
        ;;
    *)
        echo "Usage: $0 {start|stop|status}"
        exit 1
        ;;
esac

Make the script executable:

sudo chmod +x /etc/ha.d/resource.d/FloatingIP

Update the haresources file to use your custom script:

node1 FloatingIP nginx

Configuring Application Services

Install and configure your web server (nginx in this example) on both nodes:

sudo apt install nginx -y
sudo systemctl enable nginx

Create a simple health check page:

sudo nano /var/www/html/health.html

<html>
<head><title>Health Check</title></head>
<body>
<h1>Server: node1</h1>  <!-- Change to node2 on secondary -->
<p>Status: Active</p>
<p>Timestamp: <script>document.write(new Date());</script></p>
</body>
</html>

Starting and Testing the Cluster

Copy all configuration files to the secondary node, ensuring identical content except for node-specific references. Start Heartbeat on both servers:

sudo systemctl enable heartbeat
sudo systemctl start heartbeat

Monitor the logs to verify successful cluster formation:

sudo tail -f /var/log/ha-log

You should see messages indicating successful resource acquisition on the primary node:

info: Running /etc/ha.d/rc.d/FloatingIP start
info: Running /etc/ha.d/rc.d/nginx start

Test the floating IP assignment:

curl http://192.168.1.100/health.html
ping -c 3 192.168.1.100

Failover Testing and Validation

Testing failover scenarios is crucial for validating your HA setup. Here are several methods:

Graceful Failover Test

Stop Heartbeat on the primary node:

sudo systemctl stop heartbeat

Monitor the secondary node’s logs. You should see resource acquisition messages within 30 seconds (your configured deadtime).

Network Partition Test

Simulate network issues using iptables:

# On primary node, block heartbeat traffic
sudo iptables -A INPUT -p udp --dport 694 -j DROP
sudo iptables -A OUTPUT -p udp --dport 694 -j DROP

Restore connectivity after testing:

sudo iptables -F

Service Failure Test

Kill the nginx process and observe if Heartbeat detects the failure:

sudo pkill nginx

Performance Considerations and Optimization

Heartbeat’s performance directly impacts your failover time. Here’s a comparison of different timing configurations:

Configuration	Keepalive (s)	Deadtime (s)	Failover Time	Network Load	Use Case
Conservative	5	60	60-90s	Low	Non-critical applications
Balanced	2	30	30-45s	Medium	Most web applications
Aggressive	1	10	10-15s	High	Critical services

Monitor resource usage during normal operations:

# Check heartbeat process resource usage
ps aux | grep heartbeat
netstat -tulpn | grep 694

Common Issues and Troubleshooting

Split-Brain Scenarios

Split-brain occurs when both nodes think they’re primary. Symptoms include both servers claiming the floating IP. Prevention strategies:

Use multiple communication paths (serial, network)
Implement STONITH (Shoot The Other Node In The Head)
Add quorum mechanisms for larger clusters

Diagnostic command for split-brain detection:

sudo cl_status rscstatus

Network Configuration Issues

Common networking problems and solutions:

# Test heartbeat connectivity
sudo tcpdump -i eth0 udp port 694

# Verify multicast is working (if using mcast)
sudo ip maddr show

# Check firewall rules
sudo ufw status
sudo iptables -L

Log Analysis

Key log entries to monitor:

# Monitor heartbeat status
sudo grep -i "error\|warn\|fail" /var/log/ha-log

# Check resource script execution
sudo grep "Running.*resource" /var/log/ha-log

# Identify timing issues
sudo grep "deadtime\|keepalive" /var/log/ha-log

Alternatives and Comparisons

While Heartbeat works well for simple setups, consider these alternatives based on your requirements:

Solution	Complexity	Max Nodes	Active Development	Best For
Heartbeat	Low	2	Maintenance mode	Simple 2-node clusters
Keepalived	Medium	Unlimited	Active	Load balancer HA
Pacemaker	High	32+	Active	Complex multi-service clusters
Corosync	High	32+	Active	Enterprise clusters

Best Practices and Security Considerations

Implement these practices for production deployments:

Use dedicated network interfaces for heartbeat communications when possible
Implement monitoring beyond basic heartbeat checks (application-level health checks)
Regular testing of failover scenarios during maintenance windows
Document procedures for manual intervention during split-brain scenarios
Backup configurations and test restoration procedures

Security hardening steps:

# Secure heartbeat communications with iptables
sudo iptables -A INPUT -s 192.168.1.10 -p udp --dport 694 -j ACCEPT
sudo iptables -A INPUT -s 192.168.1.11 -p udp --dport 694 -j ACCEPT
sudo iptables -A INPUT -p udp --dport 694 -j DROP

# Use strong authentication keys
openssl rand -hex 32

Real-World Use Cases and Applications

Heartbeat excels in several scenarios:

Web Application Failover

E-commerce sites using VPS hosting benefit from automatic failover during server maintenance or unexpected failures. The setup ensures minimal downtime during peak shopping periods.

Database High Availability

Configure MySQL or PostgreSQL with master-slave replication, using Heartbeat to manage the writer endpoint. When the primary database fails, applications automatically connect to the promoted secondary.

# Example MySQL resource configuration
node1 FloatingIP mysql-master-switch

API Gateway Redundancy

For microservices architectures, implement API gateway redundancy using Heartbeat with nginx or HAProxy. This ensures continuous service availability even during gateway server failures.

Integration with Cloud Platforms

Modern deployments often require integration with cloud provider APIs for reserved IP management. Here’s an enhanced script for DigitalOcean floating IPs:

#!/bin/bash
# DigitalOcean Floating IP management

API_TOKEN="your-api-token"
FLOATING_IP="192.168.1.100"
DROPLET_ID=$(curl -s http://169.254.169.254/metadata/v1/id)

assign_floating_ip() {
    curl -X POST \
        -H "Content-Type: application/json" \
        -H "Authorization: Bearer $API_TOKEN" \
        -d "{\"type\":\"assign\",\"resource\":\"$DROPLET_ID\"}" \
        "https://api.digitalocean.com/v2/floating_ips/$FLOATING_IP/actions"
}

case "$1" in
    start)
        assign_floating_ip
        # Wait for API propagation
        sleep 10
        ip addr add $FLOATING_IP/32 dev eth0
        ;;
    stop)
        ip addr del $FLOATING_IP/32 dev eth0 2>/dev/null
        ;;
esac

Similar approaches work with AWS Elastic IPs, Google Cloud, and other providers through their respective APIs.

Monitoring and Alerting

Implement comprehensive monitoring to detect issues before they cause outages:

# Simple monitoring script
#!/bin/bash
FLOATING_IP="192.168.1.100"
SERVICE_URL="http://$FLOATING_IP/health.html"

if ! curl -f -s $SERVICE_URL > /dev/null; then
    echo "ALERT: Service unreachable at $SERVICE_URL"
    # Send alert via email, Slack, etc.
fi

# Check cluster status
if ! sudo cl_status rscstatus | grep -q "all"; then
    echo "ALERT: Heartbeat cluster issue detected"
fi

For production environments, integrate with monitoring solutions like Nagios, Zabbix, or modern alternatives like Prometheus with Grafana.

This setup provides a solid foundation for high availability on Ubuntu 24. The combination of Heartbeat and reserved IPs offers reliable failover capabilities suitable for small to medium-scale applications. For larger deployments or more complex requirements, consider migrating to Pacemaker or implementing load balancer-based solutions with dedicated servers. Remember that high availability is an ongoing process requiring regular testing, monitoring, and maintenance to ensure optimal performance when you need it most.

For additional technical documentation, refer to the Linux-HA project documentation and the Ubuntu Server clustering guide.

This article incorporates information and material from various online sources. We acknowledge and appreciate the work of all original authors, publishers, and websites. While every effort has been made to appropriately credit the source material, any unintentional oversight or omission does not constitute a copyright infringement. All trademarks, logos, and images mentioned are the property of their respective owners. If you believe that any content used in this article infringes upon your copyright, please contact us immediately for review and prompt action.

This article is intended for informational and educational purposes only and does not infringe on the rights of the copyright owners. If any copyrighted material has been used without proper credit or in violation of copyright laws, it is unintentional and we will rectify it promptly upon notification. Please note that the republishing, redistribution, or reproduction of part or all of the contents in any form is prohibited without express written permission from the author and website owner. For permissions or further inquiries, please contact us.