The ELK Stack (Elasticsearch, Logstash, and Kibana) is a powerful open-source solution for searching, analyzing, and visualizing data in real-time, making it essential for log management, security analytics, and business intelligence. Installing ELK on Ubuntu 24 gives you a complete data pipeline that can handle massive amounts of structured and unstructured data, from server logs to application metrics. This guide will walk you through the complete installation process, configuration tweaks, and troubleshooting common issues you’ll likely encounter.

How the ELK Stack Works

The ELK Stack operates as a three-tier architecture where each component serves a specific purpose. Elasticsearch acts as the distributed search and analytics engine, storing and indexing your data across multiple nodes for fast queries. Logstash functions as the data processing pipeline, ingesting data from various sources, transforming it, and shipping it to Elasticsearch. Kibana provides the visualization layer, offering dashboards, graphs, and search interfaces for your data.

Data flows through the stack like this: raw logs enter Logstash, get parsed and enriched, then stored in Elasticsearch indices. Kibana queries Elasticsearch to create visualizations and dashboards. This architecture scales horizontally, meaning you can add more nodes as your data volume grows.

Prerequisites and System Requirements

Before diving into installation, your Ubuntu 24 system needs adequate resources. For a development setup, 4GB RAM and 2 CPU cores work fine, but production environments should start with 8GB RAM minimum. Elasticsearch is particularly memory-hungry, so don’t skimp on RAM if you’re processing large datasets.

# Update system packages
sudo apt update && sudo apt upgrade -y

# Install Java 11 (required for ELK components)
sudo apt install openjdk-11-jdk -y

# Verify Java installation
java -version

# Install curl and wget for downloads
sudo apt install curl wget gnupg2 -y

You’ll also need to configure system limits since Elasticsearch requires higher file descriptor limits:

# Edit limits configuration
sudo nano /etc/security/limits.conf

# Add these lines at the end
elasticsearch soft nofile 65536
elasticsearch hard nofile 65536
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited

Installing Elasticsearch

Elasticsearch installation starts with adding the official Elastic repository. This ensures you get the latest stable version with proper security updates.

# Import Elasticsearch GPG key
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg

# Add Elasticsearch repository
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list

# Update package index
sudo apt update

# Install Elasticsearch
sudo apt install elasticsearch -y

After installation, you need to configure Elasticsearch for your environment. The main configuration file is located at `/etc/elasticsearch/elasticsearch.yml`:

# Edit Elasticsearch configuration
sudo nano /etc/elasticsearch/elasticsearch.yml

# Key settings to modify:
cluster.name: my-elk-cluster
node.name: node-1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: localhost
http.port: 9200
discovery.type: single-node

# For production, also set:
bootstrap.memory_lock: true

Configure JVM heap size based on your available RAM. Never allocate more than 50% of your system’s RAM to Elasticsearch:

# Edit JVM options
sudo nano /etc/elasticsearch/jvm.options

# For 8GB system, set:
-Xms4g
-Xmx4g

# For 16GB system, set:
-Xms8g
-Xmx8g

Start and enable Elasticsearch:

# Start Elasticsearch service
sudo systemctl start elasticsearch

# Enable auto-start on boot
sudo systemctl enable elasticsearch

# Check status
sudo systemctl status elasticsearch

# Test Elasticsearch
curl -X GET "localhost:9200/"

Installing Logstash

Logstash installation uses the same Elastic repository you already added:

# Install Logstash
sudo apt install logstash -y

# Start and enable Logstash
sudo systemctl start logstash
sudo systemctl enable logstash

Logstash configuration requires creating pipeline files that define how to process your data. Here’s a basic configuration for processing syslog data:

# Create a basic Logstash configuration
sudo nano /etc/logstash/conf.d/01-syslog.conf

input {
  file {
    path => "/var/log/syslog"
    start_position => "beginning"
    type => "syslog"
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{IPORHOST:host} %{DATA:program}(?:\[%{POSINT:pid}\])?: %{GREEDYDATA:message}" }
    }
    date {
      match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "logstash-syslog-%{+YYYY.MM.dd}"
  }
  stdout { codec => rubydebug }
}

Test your Logstash configuration before starting the service:

# Test Logstash configuration
sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t

# Restart Logstash with new configuration
sudo systemctl restart logstash

# Check Logstash logs for errors
sudo journalctl -u logstash -f

Installing Kibana

Kibana installation completes your ELK Stack setup:

# Install Kibana
sudo apt install kibana -y

# Configure Kibana
sudo nano /etc/kibana/kibana.yml

# Key configuration settings:
server.port: 5601
server.host: "localhost"
elasticsearch.hosts: ["http://localhost:9200"]
logging.dest: /var/log/kibana/kibana.log

Create the log directory and set proper permissions:

# Create Kibana log directory
sudo mkdir /var/log/kibana
sudo chown kibana:kibana /var/log/kibana

# Start and enable Kibana
sudo systemctl start kibana
sudo systemctl enable kibana

Access Kibana through your web browser at `http://localhost:5601`. The first startup takes a few minutes as Kibana initializes its indices in Elasticsearch.

Performance Optimization and Configuration

ELK Stack performance depends heavily on proper configuration. Here’s a comparison of different configuration approaches:

Configuration	Memory Usage	Query Performance	Suitable For
Single Node	2-4GB	Good for <1M docs	Development, small deployments
Multi-Node Cluster	8GB+ per node	Excellent	Production environments
Hot-Warm Architecture	Variable	Optimized for retention	Large-scale logging

For better performance, configure index templates and lifecycle policies:

# Create an index template for better performance
curl -X PUT "localhost:9200/_index_template/logstash-template" -H 'Content-Type: application/json' -d'
{
  "index_patterns": ["logstash-*"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 0,
      "refresh_interval": "30s"
    },
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "message": {
          "type": "text",
          "analyzer": "standard"
        }
      }
    }
  }
}
'

Real-World Use Cases and Examples

The ELK Stack excels in several scenarios. For web application monitoring, you can ingest Apache/Nginx access logs to track user behavior, identify performance bottlenecks, and monitor error rates. Security teams use ELK for log aggregation from firewalls, intrusion detection systems, and authentication servers to spot suspicious activities.

Here’s a practical example for monitoring web server logs:

# Logstash configuration for Apache access logs
input {
  file {
    path => "/var/log/apache2/access.log"
    start_position => "end"
    type => "apache-access"
  }
}

filter {
  if [type] == "apache-access" {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    
    date {
      match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
    
    mutate {
      convert => { "response" => "integer" }
      convert => { "bytes" => "integer" }
    }
    
    if [clientip] {
      geoip {
        source => "clientip"
        target => "geoip"
      }
    }
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "apache-access-%{+YYYY.MM.dd}"
  }
}

Common Issues and Troubleshooting

Several issues frequently pop up during ELK Stack installations. Memory-related problems are the most common, especially when Elasticsearch fails to start due to insufficient heap space or memory lock issues.

# Check Elasticsearch cluster health
curl -X GET "localhost:9200/_cluster/health?pretty"

# View cluster nodes
curl -X GET "localhost:9200/_cat/nodes?v"

# Check indices
curl -X GET "localhost:9200/_cat/indices?v"

# Monitor Elasticsearch logs
sudo tail -f /var/log/elasticsearch/my-elk-cluster.log

If Elasticsearch won’t start, check these common issues:

• Java version compatibility – ELK 8.x requires Java 11 or later
• Memory lock configuration – ensure bootstrap.memory_lock is properly set
• File permissions – Elasticsearch user needs proper access to data directories
• Port conflicts – verify ports 9200 and 9300 aren’t in use by other services

For Logstash troubleshooting, pipeline validation is crucial:

# Debug Logstash configuration
sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash --config.test_and_exit

# Run Logstash in debug mode
sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash --log.level debug

Security Best Practices

Production ELK deployments require security configurations beyond the basic setup. Enable authentication and SSL/TLS encryption for all communications:

# Generate certificates for secure communication
sudo /usr/share/elasticsearch/bin/elasticsearch-certutil ca
sudo /usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12

# Configure SSL in elasticsearch.yml
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

Set up user authentication and role-based access control:

# Setup built-in users passwords
sudo /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive

# Create custom users and roles
curl -X POST "localhost:9200/_security/role/log_reader" -H 'Content-Type: application/json' -d'
{
  "cluster": ["monitor"],
  "indices": [
    {
      "names": ["logstash-*"],
      "privileges": ["read"]
    }
  ]
}
'

Scaling and Performance Monitoring

As your data volume grows, monitoring ELK Stack performance becomes critical. Set up monitoring indices to track cluster health, query performance, and resource utilization:

# Enable monitoring
curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "xpack.monitoring.collection.enabled": true
  }
}
'

For high-volume environments, consider implementing index lifecycle management (ILM) to automatically rotate and delete old indices:

# Create ILM policy for log retention
curl -X PUT "localhost:9200/_ilm/policy/logstash-policy" -H 'Content-Type: application/json' -d'
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "10GB",
            "max_age": "7d"
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}
'

The ELK Stack on Ubuntu 24 provides a robust foundation for data analytics and log management. With proper configuration and monitoring, it handles everything from small application logs to enterprise-scale data processing. Remember that successful ELK deployments require ongoing tuning and maintenance, especially as your data patterns and volume change over time.

For production deployments requiring high availability and dedicated resources, consider using robust infrastructure solutions like VPS services or dedicated servers that provide the consistent performance ELK Stack demands.

Additional resources for deepening your ELK Stack knowledge include the official Elastic documentation and the Elastic GitHub repositories for community configurations and plugins.

---

---

This article incorporates information and material from various online sources. We acknowledge and appreciate the work of all original authors, publishers, and websites. While every effort has been made to appropriately credit the source material, any unintentional oversight or omission does not constitute a copyright infringement. All trademarks, logos, and images mentioned are the property of their respective owners. If you believe that any content used in this article infringes upon your copyright, please contact us immediately for review and prompt action.

This article is intended for informational and educational purposes only and does not infringe on the rights of the copyright owners. If any copyrighted material has been used without proper credit or in violation of copyright laws, it is unintentional and we will rectify it promptly upon notification. Please note that the republishing, redistribution, or reproduction of part or all of the contents in any form is prohibited without express written permission from the author and website owner. For permissions or further inquiries, please contact us.