Setup Prometheus + Grafana + Loki Stack for Metrics, Logs, and Alerts

Table of Contents

• What This Article Is About & Why You Should Care
• A Dramatic Hook: The Midnight Pager Nightmare
• The Real Problem: Observability, Not Just Monitoring
• How Does Prometheus + Grafana + Loki Stack Work?
• Fast & Easy Setup: Step-by-Step Guide
• Tree of Use Cases & Benefits
• Mini Glossary: Real-Talk Explanations
• Comic Comparison Table: Why This Stack Rocks (& When It Doesn’t)
• Beginner Mistakes, Myths & Similar Solutions
• 🧭 “Use This If…” Decision Flowchart
• Automation, Scripting & Fun Facts
• Fictionalized Admin Story: The Case of the Disappearing Disk Space
• Final Take: Should You Set Up This Stack?

What This Article Is About & Why You Should Care

Let’s face it—if you run servers, websites, apps, or even a modest Raspberry Pi fleet, you’ve probably heard of Prometheus, Grafana, and Loki. This article is your no-nonsense, quick-start, real-world guide to setting up the full stack for metrics, logs, and alerts—the holy trinity of “observability.”

Why does this matter? Because running blind is the worst way to run anything. Whether you’re a solo coder, a startup CTO, or the resident “server person,” you need to know what’s happening on your systems—preferably before your users or boss do. This stack is how you get there.

Who is this for? Anyone who wants to see, understand, and react to what their servers are actually doing, with as little pain as possible. If you want to run your own infrastructure—cloud, VPS, Docker, or bare metal—read on.

A Dramatic Hook: The Midnight Pager Nightmare

It’s 2:42 AM. Your phone buzzes. PagerDuty (or Slack, or Telegram, or… you get the idea) is lighting up. Users are complaining. Your site’s slower than a snail on a treadmill. But why? Is it a code bug? Disk full? A wild process eating all the RAM? You have no idea.

You ssh in, poke around, tail some logs, stare at top, and, in desperation, restart everything. Maybe it helps. Maybe not. The problem is, you’re flying blind.

There’s a better way. And it starts with a proper observability stack.

The Real Problem: Observability, Not Just Monitoring

It’s not enough to just “monitor” CPU and RAM anymore. Modern systems have layers of services, containers, microservices, and mysterious cloud stuff. Problems hide in logs, spikes, and odd patterns. You need metrics, logs, and alerts—all in one place, with context.

The Prometheus + Grafana + Loki stack is the free, open-source solution that does it all—without vendor lock-in, with real community support, and at any scale you like.

How Does Prometheus + Grafana + Loki Stack Work?

Let’s break down the stack into its main roles (imagine them as a team of quirky superheroes):

• Prometheus: The data collector and time-series database. It scrapes metrics from your apps, servers, and exporters (think: CPU, RAM, app-specific stuff).
• Loki: The log aggregator. Unlike old log systems, Loki is “Prometheus for logs”—cheap, fast, and designed for label-based search (no grep gymnastics needed).
• Grafana: The dashboard king. It visualizes both metrics and logs. Also handles alerting, so you get pinged when stuff breaks (or is about to).

How do they talk?

• Prometheus scrapes metrics from endpoints (exporters or apps with built-in Prometheus support).
• Loki receives logs via promtail (tailing files or systemd/journal).
• Grafana connects to both, letting you create dashboards that mix metrics and logs, trigger alerts, and more.

Diagrams for the visual types:

         +-----------+          +-----------+
         |           |          |           |
         | Prometheus|<-------->| Exporters |
         |           |          | (node,app)|
         +-----------+          +-----------+
               |
               v
         +-----------+
         |           |
         |  Grafana  |<---+
         |           |    |
         +-----------+    |
                          |
         +-----------+    |
         |           |    |
         |   Loki    |<---+
         |           |
         +-----------+

In short: metrics in one place, logs in another, dashboards and alerts everywhere.

Fast & Easy Setup: Step-by-Step Guide

Here’s the “get it running in an hour” recipe. You’ll need a server (cloud, VPS, or dedicated—order from MangoHost VPS or Dedicated Server if you don’t have one!). Ubuntu/Debian is easiest, but these work on most Linux distros.

Step 1: Install Prometheus

1. Download and extract:
   
wget https://github.com/prometheus/prometheus/releases/latest/download/prometheus-*.tar.gz
tar xvf prometheus-*.tar.gz

2. Create a user & move binaries:
   
useradd --no-create-home --shell /bin/false prometheus
mv prometheus-*/prometheus /usr/local/bin/
mv prometheus-*/promtool /usr/local/bin/

3. Setup config and service (sample /etc/prometheus/prometheus.yml):
   
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']

4. Start Prometheus:
   
systemctl daemon-reload
systemctl enable prometheus
systemctl start prometheus


Step 2: Install Node Exporter (for server metrics)

1. 
wget https://github.com/prometheus/node_exporter/releases/latest/download/node_exporter-*.tar.gz
tar xvf node_exporter-*.tar.gz
mv node_exporter-*/node_exporter /usr/local/bin/
systemctl enable --now node_exporter

2. Add localhost:9100 to Prometheus config under scrape_configs.

Step 3: Install Loki & Promtail (for logs)

1. Download Loki & Promtail:
   
wget https://github.com/grafana/loki/releases/latest/download/loki-linux-amd64.zip
unzip loki-linux-amd64.zip
mv loki-linux-amd64 /usr/local/bin/loki



wget https://github.com/grafana/loki/releases/latest/download/promtail-linux-amd64.zip
unzip promtail-linux-amd64.zip
mv promtail-linux-amd64 /usr/local/bin/promtail

2. Sample Loki config (loki-config.yaml):
   
auth_enabled: false
server:
http_listen_port: 3100
ingester:
lifecycler:
address: 127.0.0.1
ring:
kvstore:
store: inmemory
final_sleep: 0s
chunk_idle_period: 5m
chunk_retain_period: 30s
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
storage_config:
boltdb_shipper:
active_index_directory: /tmp/loki/index
cache_location: /tmp/loki/cache
shared_store: filesystem
filesystem:
directory: /tmp/loki/chunks

3. Sample Promtail config (promtail-config.yaml):
   
server:
http_listen_port: 9080
positions:
filename: /tmp/positions.yaml
clients:
- url: http://localhost:3100/loki/api/v1/push
scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
job: varlogs
__path__: /var/log/*.log

4. Start Loki and Promtail:
   
nohup loki -config.file=loki-config.yaml &
nohup promtail -config.file=promtail-config.yaml &


Step 4: Install Grafana

1. 
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update
sudo apt install grafana
sudo systemctl enable --now grafana-server

2. Open http://your-server-ip:3000 (user: admin, pass: admin). Add Prometheus (default http://localhost:9090) and Loki (http://localhost:3100) as data sources.

Step 5: Create Dashboards & Alerts

• Import community dashboards (search within Grafana for “Node Exporter Full” and “Loki Log Explorer”).
• Set up alerts (in Grafana 8+, alerts live in the Alerting section).

Boom, you’re done! You now have real-time metrics, log search, and alerts—free, open source, and running on your own box.

Tree of Use Cases & Benefits

• DevOps & SRE: Full-stack visibility, incident response, performance analysis, root-cause detection.
• Developers: Debugging, profiling, seeing what their code actually does in prod.
• Sysadmins: Uptime, capacity planning, “Why is disk filling up?” automatic alerts.
• Startups: All the above, but on a shoestring budget (no SaaS bills!).
• Home-labbers/hobbyists: For fun, learning, and bragging rights.

Benefits:

• Open source, free, and battle-tested
• Easy to extend (add exporters, apps, etc.)
• Works anywhere: cloud, on-prem, containers, VMs
• Massive ecosystem (plugins, exporters, dashboards)

Mini Glossary: Real-Talk Explanations

• Metrics: Numbers over time (CPU, RAM, “users online now”, etc.)
• Logs: Text output from apps, servers, etc. (aka “what just happened”)
• Exporter: A small program exporting metrics for Prometheus (e.g., node_exporter for server stats)
• Alert: Triggered when your defined rule is breached (like “disk over 90% full”)
• Promtail: The Loki-sidekick that tails your logs and ships them to Loki

Comic Comparison Table: Why This Stack Rocks (& When It Doesn’t)

Beginner Mistakes, Myths & Similar Solutions

• Mistake 1: Only install Prometheus—no logs means you’re still blind to half of your problems.
• Mistake 2: Forget to set up alerts. (Don’t wait for users to tell you stuff is broken!)
• Myth: “Prometheus is only for Kubernetes.” Nope! Works everywhere—from single servers to huge clouds.
• Myth: “Loki is a clone of ELK.” Not true. Loki is log-indexing “the Prometheus way” (label-based, not full-text).
• Similar solutions: ELK/ELastic/Opensearch (logs only, much heavier), InfluxDB/Telegraf (metrics), Datadog/NewRelic (proprietary, $$$).

🧭 “Use This If…” Decision Flowchart

Want to see metrics?  ---> Yes
                            |
                Need log search? ---> Yes
                            |
           Want to host yourself? ---> Yes
                            |
             Need open source? ---> Yes
                            |
         ---> Use Prometheus + Grafana + Loki Stack!

Otherwise...
|
+---> Only metrics: Prometheus + Grafana
|
+---> Only logs: Loki (light), or ELK/Opensearch (heavy)
|
+---> No time, but $$$: Datadog, NewRelic, etc.

If you want to own your own data, customize, and avoid recurring SaaS bills—this is the stack for you!

Automation, Scripting & Fun Facts

• Automate all the things: Use Ansible, Terraform, or simple Bash scripts to deploy and update your stack. (There are many community playbooks/modules!)
• Fun fact: Prometheus scrapes metrics via HTTP endpoints, so you can add anything—even your coffee machine, if it speaks HTTP!
• Unconventional use: Grafana can visualize any time-series data, including weather, trading stats, home automation sensors, and more.

Example: Bash one-liner to monitor disk usage and send custom metrics to Prometheus via Pushgateway:


#!/bin/bash
USAGE=$(df / | awk 'END{print $5}' | tr -d '%')
echo "disk_usage_percent $USAGE" | curl --data-binary @- http://localhost:9091/metrics/job/disk


(Pro tip: Set up a cron job and visualize this in Grafana!)

Fictionalized Admin Story: The Case of the Disappearing Disk Space

Once upon a midnight dreary, a junior admin named Sam noticed their web app was sluggish. Logs? Nowhere to be found—someone had rotated them into oblivion. But, with Prometheus humming, they noticed a spike in disk usage. Loki showed exactly which container log was growing out of control.

Sam set up an alert: "Warn me if disk usage > 85%." Next time, Grafana pinged them before users complained. Moral of the story: metrics + logs + alerts = superpowers, even for sleepy admins.

Final Take: Should You Set Up This Stack?

If you care about your uptime, user happiness, or just want to sleep through the night, setting up the Prometheus + Grafana + Loki stack is a no-brainer. It’s open source, flexible, and works on anything—from your home lab to your cloud empire.

Where to run it? Practically anywhere. For best results, grab a solid VPS or dedicated server—check out MangoHost VPS or dedicated server for hassle-free hosting.

Why use it?

• You own your data—no vendor lock-in
• No surprise SaaS bills
• Full context: metrics, logs, and alerts in one place
• Battle-tested and loved by the world’s best ops teams

Ready to see what your servers are really doing? Set up the stack, sleep better, and join the ranks of those who know before it breaks.

Further Reading / Official Docs (non-commercial):

• Prometheus Documentation
• Grafana Documentation
• Loki Documentation

Got questions, stories, or weird use cases? Drop them in the comments, share your dashboards, or automate something wild!