Why Should You Care About Packet Tracing with pwru?

If you’ve ever run a web server, managed a Docker swarm, or juggled a few VPSes, you know that networking issues are like gremlins: they pop up at the worst times, and they’re devilishly hard to pin down. Maybe your app’s traffic is mysteriously dropping, or some packets are vanishing into the void between your containers and the outside world. You’ve tried tcpdump, wireshark, and even strace, but sometimes you need to see what’s happening *inside* the Linux kernel, not just on the wire. That’s where pwru comes in—a shiny, modern tool that leverages eBPF magic to trace packets as they journey through the kernel’s networking stack.

Let’s break down why this is a game-changer, how it works, and how you can get it running on your own cloud, VPS, or dedicated server (if you need one, check out VPS or dedicated).

The Problem: Black Boxes and Blind Spots in Networking

Here’s the deal: traditional packet sniffers like tcpdump only see what’s on the wire. But what if the packet never makes it out of the kernel? Or gets dropped by a firewall rule, mangled by NAT, or rerouted by some obscure kernel module? Debugging these issues is like trying to fix a car engine while blindfolded.

Enter eBPF—a technology that lets you run tiny programs inside the Linux kernel, safely and efficiently. And pwru (Packet Watch and Routing Utility) is a tool that harnesses eBPF to trace packets as they move through the kernel’s networking stack, step by step.

Why is this important for people running cloud, VPS, Docker, or dedicated servers?

• Pinpoint where packets are dropped, mangled, or delayed
• Debug complex firewall (iptables/nftables) and routing issues
• Understand how containers and overlays (like Kubernetes, Docker) handle networking
• Optimize performance by identifying bottlenecks

How Does pwru Work? (And Why Is It So Cool?)

Let’s geek out for a second. pwru uses eBPF to attach probes to dozens of key functions inside the Linux networking stack. When a packet enters the system (say, from a network interface), pwru can log every function the packet passes through—like ip_rcv(), ip_forward(), nf_hook(), and more. You get a live, step-by-step trace of the packet’s journey, including metadata like interface, protocol, and verdict (accepted, dropped, etc).

Algorithms & Structure:

• eBPF programs are attached to kernel tracepoints and kprobes
• Each time a packet hits a traced function, pwru logs the event
• Filters can be applied (by IP, port, protocol, etc) to focus on the packets you care about
• Output is streamed in real time, with details about each step

This is way deeper than what tcpdump or wireshark can do. You’re not just seeing packets on the wire—you’re seeing their fate inside the kernel.

How to Set Up pwru Quickly (on Cloud, VPS, Docker, or Bare Metal)

Let’s get practical. Here’s how to get pwru running on your Linux box. (You’ll need a recent kernel, 5.10+ is best, and root access.)

1. Install Prerequisites

• Linux kernel 5.10 or newer (for best results)
• Root access (sudo)
• git, clang, llvm, make, libbpf-dev (for building from source)

2. Get pwru

The official repo is here: https://github.com/cilium/pwru


git clone https://github.com/cilium/pwru.git
cd pwru
make


Or, if you’re on a bleeding-edge distro, you might find a package.

3. Run pwru

Basic usage (trace all packets):


sudo ./pwru


Filter by source IP and port:


sudo ./pwru --src-ip 10.0.0.5 --src-port 8080


Trace only TCP packets:


sudo ./pwru --proto tcp


You’ll see output like:

[0.000000] ip_rcv() dev=eth0 proto=0x0800
[0.000123] nf_hook_slow() hook=1
[0.000234] ip_forward()
[0.000345] ip_output()

4. Docker and Containers

If you’re running Docker, you can install pwru on the host. To trace packets from a specific container, find its veth interface:


docker exec -it mycontainer sh -c 'ip addr'


Then on the host:


sudo ./pwru --ifname vethXXXX


5. Kubernetes and Cloud

Same deal: install pwru on the node (VM, VPS, or bare metal). Use kubectl get pod -o wide to find pod IPs, then filter with --src-ip or --dst-ip.

Examples, Cases: The Good, The Bad, and The Ugly

Let’s look at some real-world scenarios.

Case	Old Tools	pwru	Advice
Packets dropped by firewall	tcpdump shows nothing; iptables logs are cryptic	pwru shows packet hitting nf_hook() and being dropped	Use --proto and --src-ip to zoom in; check which hook drops it
Container networking issues	tcpdump on host sees nothing; hard to trace veth pairs	pwru can trace packets as they enter/leave container veth	Find veth name, use --ifname filter
Strange NAT behavior	tcpdump sees rewritten packets, not the original	pwru shows packet before and after NAT hooks	Trace both pre- and post-NAT hooks for full picture

Beginner Mistakes and Myths

• Myth: “pwru replaces tcpdump.”
  Reality: They’re complementary. Use tcpdump for wire-level, pwru for kernel-level.
• Mistake: Not filtering!
  Advice: Always use filters (--src-ip, --proto, etc) or you’ll drown in output.
• Myth: “eBPF tools are dangerous.”
  Reality: eBPF is sandboxed and safe for tracing (just don’t run random code as root!)
• Mistake: Running on old kernels.
  Advice: Upgrade to 5.10+ for best compatibility.

Similar Solutions and How pwru Compares

Tool	Level	Pros	Cons
tcpdump/wireshark	Wire	Easy, detailed packet view	Can’t see dropped packets or kernel path
strace	User-space syscalls	See app-level networking	No kernel path, no packet details
trace-cmd/perf	Kernel tracing	Powerful, generic	Complex, not packet-focused
pwru	Kernel networking stack	Packet journey, drop points, eBPF-powered	Requires recent kernel, some learning curve

Interesting Facts and Non-Standard Usage

• pwru is built by the Cilium team, who are eBPF wizards (they also make Tetragon for security tracing)
• You can use pwru to trace not just drops, but performance bottlenecks—see which functions take the most time for your packets
• pwru can help debug weird Kubernetes CNI plugin issues, especially with overlays and multi-homed pods
• Combine pwru with automation scripts (bash, Python) to auto-diagnose network issues in CI/CD pipelines

Automation and Scripting: New Opportunities

Because pwru is CLI-based and outputs structured logs, you can:

• Integrate with monitoring tools (Prometheus, Grafana) for live packet tracing dashboards
• Write scripts to auto-capture traces when certain errors are detected
• Use in incident response playbooks to quickly localize networking faults

Example: Auto-trace packets to a failing service:


#!/bin/bash
SERVICE_IP="10.0.0.42"
sudo ./pwru --dst-ip $SERVICE_IP --proto tcp --output trace.log


Conclusion: Why You Should Try pwru Today

If you’re running anything more complex than a static HTML site, you’ll eventually hit a networking mystery that traditional tools can’t solve. pwru gives you X-ray vision into the Linux kernel’s networking stack, showing you exactly where packets go, get dropped, or get mangled. It’s a must-have for anyone running cloud, VPS, Docker, or bare metal servers—especially if you’re dealing with firewalls, NAT, or container networking.

My recommendation:

• Try pwru on your dev/staging environment first—get familiar with the output and filters
• Use it alongside tcpdump for a full picture (wire + kernel)
• Automate common traces for your stack (Kubernetes, Docker, etc)
• If you need a playground, grab a VPS or dedicated server and experiment

For more info, check the official repo: https://github.com/cilium/pwru

Happy tracing, and may your packets always find their way home!

---

---

This article incorporates information and material from various online sources. We acknowledge and appreciate the work of all original authors, publishers, and websites. While every effort has been made to appropriately credit the source material, any unintentional oversight or omission does not constitute a copyright infringement. All trademarks, logos, and images mentioned are the property of their respective owners. If you believe that any content used in this article infringes upon your copyright, please contact us immediately for review and prompt action.

This article is intended for informational and educational purposes only and does not infringe on the rights of the copyright owners. If any copyrighted material has been used without proper credit or in violation of copyright laws, it is unintentional and we will rectify it promptly upon notification. Please note that the republishing, redistribution, or reproduction of part or all of the contents in any form is prohibited without express written permission from the author and website owner. For permissions or further inquiries, please contact us.