Getting started

Build it, run it, break something

Building, vetting and testing work on any platform — the collectors ship non-Linux stubs precisely so the inner loop does not require Linux. Observing anything requires a real Linux kernel: netlink for interface, neighbour and route state, and eBPF with BTF for connection outcomes.

01
Build
One target does formatting, vet, the tests and both binaries. CGO stays off, so the result is a static binary with no runtime dependencies — the SQLite driver is pure Go.
$make all
```
 
```
02
Record
netrewindd is the recorder. Reading netlink notifications and attaching eBPF programs needs CAP_NET_ADMIN, which in a lab means root.
Point --rules at the rule directory and correlation runs live, as events arrive; pass an empty value and the recorder only records.
$sudo ./build/netrewindd --db ./var/events.db --rules ./rules
```
 
```
03
Read it back
netrewind is the query tool. It never writes to the store, so it is safe to run against a recorder that is still going.
$./build/netrewind timeline --last 15m
```
 
```

Or watch it work without a network to break

The synthetic lab builds a topology out of network namespaces and veth pairs, starts the recorder inside it, injects fourteen scenarios — thirteen faults and one stretch of ordinary successful traffic — and then checks the record for every one of them.

Nothing in it touches the real network: the namespaces, the routes and even the nftables ruleset are per-namespace, so a fault cannot escape the lab.

The run fails if anything injected left no trace, and fails separately if correlation did not reach the right conclusion. Having the evidence and drawing the conclusion are different achievements.

$sudo make lab

lab-up and lab-down bring the topology up and down on their own, and inject.sh run <scenario> injects a single fault into a running lab.

One trap worth knowing

Entering a network namespace gets a fresh mount namespace with /sys remounted, so tracefs disappears and no eBPF tracepoint can be attached. It has to be mounted inside the same ip netns exec that runs the recorder — which is why the lab script looks the way it does.

The recorder

netrewindd

Flag	Default	Meaning
--db	/var/lib/netrewind/events.db	Path to the event store
--rules	rules	Directory of correlation rules; empty disables correlation entirely
--retention	168h	How much history to keep
--gap-threshold	30s	An absence longer than this is recorded as a hole in the record
--observer-id	hostname	Identity of this recorder, stamped on every event it produces
--metrics-addr	off	Serve Prometheus metrics here, e.g. 127.0.0.1:9464
--log-level	info	debug, info, warn or error

Reading it back

Six commands, and --db is common to all of them. The one you want during an incident is what-happened; the one you want afterwards is incidents.

netrewind events

List recorded events in time order

The flat, unopinionated view. Reach for it when you already know what you are looking for and want it unedited — or when you want JSON to pipe somewhere else.

$ netrewind events --last 10m
$ netrewind events --last 1h --family link
$ netrewind events --last 24h --host 192.168.20.10 -o json

Flag	Default	Meaning
--last	1h	Look back this far from now
--since / --until	now	An explicit RFC3339 window; --since overrides --last
--kind	all	Restrict to these event kinds, e.g. link.down
--family	all	Restrict to these families, e.g. link,l2
--host	all	Restrict to one entity, by address or name
-o, --output	table	table or json
--newest-first	off	Reverse the order

netrewind timeline

Read recorded history as a narrative

The same record, rendered as sentences rather than fields. Severity is a glyph so it survives a monochrome terminal and a screenshot, which is where these end up.

$ netrewind timeline --last 30m
$ netrewind timeline --last 2h --min-severity warn

Flag	Default	Meaning
--last	30m	How far back to read
--family	all	Restrict to these families
--min-severity	info	info, notice, warn or error

netrewind what-happened

Reconstruct what happened around a moment

The question that can only be asked afterwards. Give it a machine and a time; it resolves that machine through the identity table first, so it finds the events recorded while it held a different address — and does not drag in the events of whichever machine holds that address now.

$ netrewind what-happened --host 192.168.20.10 --at 15:00
$ netrewind what-happened --at 2026-08-23T10:43:00Z --window 5m
$ netrewind what-happened --host 192.168.20.10 --at -2h

Flag	Default	Meaning
--host	everything	The machine to follow, by address or hardware address
--at	now	The moment to look around: RFC3339, HH:MM, -2h, or now
--window	5m	How far either side of that moment to look

netrewind incidents

List what correlation concluded, with the chain behind each conclusion

Conclusions rather than observations. Every one carries the chain it rests on, the relation claimed at each step, and advice on what to look at next.

$ netrewind incidents --last 24h
$ netrewind incidents --last 1h --rule change-broke-a-path

Flag	Default	Meaning
--last	24h	How far back to look
--rule	all	Only incidents from this rule
--min-severity	all	info, notice, warn or error
-o, --output	text	text or json

netrewind serve

Read the record in a browser

The same record the CLI shows, rendered as pages: incidents, a timeline, and a per-host view. Server-rendered with no JavaScript, and read-only - the store is evidence, and the thing that displays evidence has no business modifying it. It binds to loopback because there is no authentication, and a recorder should not be reachable from the network it is watching.

$ netrewind serve
$ netrewind serve --db /var/lib/netrewind/events.db

Flag	Default	Meaning
--addr	127.0.0.1:8464	Where to listen; loopback unless you deliberately change it
--observer-id	hostname	Which recorder this view is labelled as

netrewind rules

Load and check the correlation rules

Validates the library exactly the way the recorder does, so a broken rule is found before it is deployed rather than by never matching anything.

$ netrewind rules --dir rules

Flag	Default	Meaning
--dir	rules	Directory of rule files

$netrewind rules --dir rules

19 rules loaded from rules
 
  address-changed-hands      warn    75%  1 clauses (1 required)  window 1m0s
                             An address moved to a different machine
 
  change-broke-a-path        error   88%  3 clauses (1 required)  window 5m0s
                             A path that was working stopped working
 
  collector-not-watching     error  100%  1 clauses (1 required)  window 1m0s
                             The recorder is running but one of its senses is not
 
  contested-address          error   85%  2 clauses (1 required)  window 2m0s
                             Two machines are claiming the same address
 
  default-route-lost         error   95%  2 clauses (1 required)  window 2m0s
                             The default route disappeared
 
  default-route-moved        error   85%  2 clauses (1 required)  window 2m0s
                             The default route was re-pointed
 
  gateway-hijack             error   90%  3 clauses (1 required)  window 3m0s
                             The default gateway is being answered by a different machine
 
  gateway-returned-different error   85%  2 clauses (1 required)  window 10m0s
                             The gateway came back, but as a different machine
 
  host-moved-ports           warn    70%  1 clauses (1 required)  window 1m0s
                             A machine appeared behind a different interface
 
  link-degrading             error   90%  2 clauses (1 required)  window 10m0s
                             A link is failing rather than having failed
 
  link-down-isolated-hosts   warn    80%  2 clauses (2 required)  window 1m30s
                             A link went down and took hosts with it
 
  path-mtu-blackhole         error   90%  2 clauses (1 required)  window 10m0s
                             A path is silently dropping large packets
 
  port-flapping              warn    88%  2 clauses (1 required)  window 5m0s
                             A port is flapping
 
  reachability-lost          error   90%  3 clauses (1 required)  window 5m0s
                             A measured path stopped answering
 
  recorder-was-blind         warn   100%  1 clauses (1 required)  window 1m0s
                             The record has a hole in it
 
  resolver-hijacked          warn    80%  3 clauses (1 required)  window 5m0s
                             A machine's resolver changed under it
 
  rogue-dhcp-server          error   92%  3 clauses (1 required)  window 5m0s
                             A second DHCP server is answering on this segment
 
  route-flapping             error   85%  2 clauses (1 required)  window 5m0s
                             A route is flapping
 
  service-unreachable        warn    80%  2 clauses (1 required)  window 2m0s
                             A service stopped answering

Metrics

The recorder exports how much it did not see

Every observability tool exports counts of what it saw. Those are the easy numbers, and they are reassuring in exactly the wrong way: a recorder that has stopped watching reports a beautifully quiet network.

The two series that matter here are the ones counting the holes. netrewind_recorder_blind_seconds_total and netrewind_dropped_events_total let an operator alert on the record being untrustworthy for a period, rather than discovering it during the incident review — when it is too late to do anything about it.

Every series is declared at zero on startup, so an alert on an increase works from the first scrape rather than waiting for the first bad thing to happen.

$sudo ./build/netrewindd --db ./var/events.db --metrics-addr 127.0.0.1:9464

Served at /metrics, in the Prometheus text format. Empty disables it.

Series	Type	What it counts
netrewind_recorder_blind_seconds_total	counter	Seconds the recorder was not watching. Any increase means the record has a hole in it.
netrewind_dropped_events_total	counter	Events lost because they arrived faster than they could be read, labelled by source.
netrewind_clock_steps_total	counter	Wall-clock jumps observed. Each one reorders the timeline for anyone reading it later.
netrewind_events_total	counter	Events recorded, by kind and severity. Folded events count their occurrences, not their rows — a metric that counted rows would under-report exactly when things are worst.
netrewind_incidents_total	counter	Incidents concluded, by rule and severity.
netrewind_collector_up	gauge	1 while a collector is running, 0 once it has stopped.
netrewind_stored_events	gauge	Events currently held in the store.
netrewind_build_info	gauge	Version and observer id of the running recorder.

Writing a rule

A rule describes one recognisable shape of failure

Rules are files, not code. The library is the part of this project that cannot be cloned — it accumulates from real incidents — and anyone who has spent a night debugging a network can contribute to it without touching Go.

Below is rules/change-broke-a-path.yaml in full: the most interesting rule in the shipped library, because it is the one that asks the question backwards.

recorded output

id: change-broke-a-path
title: A path that was working stopped working
severity: error
confidence: 88
window: 300s
root_cause: change
advice: >-
  The change named here is the nearest one in time, not a proven cause. Confirm
  it against your change record before acting: if it was intended, the breakage
  is a side effect nobody planned; if it was not, something changed the network
  without going through you.
match:
  - as: change
    kinds:
      - l2.arp_binding_changed
      - l2.duplicate_ip
      - l3.route_changed
      - l3.default_route_changed
      - l3.route_removed
      - link.down
      - policy.rule_changed
      - change.config_applied
    optional: true
    why: >-
      This is the last thing that changed on the path before it broke.
  - as: breakage
    kinds: [flow.first_failure_for_pair]
    relation: causes
    why: >-
      Two machines that had been connecting successfully can no longer complete
      a handshake. Whatever else is true, something between them changed.
  - as: spread
    kinds: [flow.first_failure_for_pair, flow.handshake_fail, l2.neighbor_failed]
    optional: true
    relation: correlates
    why: Other connections started failing in the same window.

rules/change-broke-a-path.yaml

match[0] — as: change, optional: true: This clause sits before the anchor, so it is searched backwards in time, nearest first. It is what makes the rule ask what changed just before this broke rather than what followed. It is optional, so with no preceding change in the window the rule still fires with a one-link chain: the symptom stands on its own rather than being attached to a guess.
match[1] — as: breakage, required: The anchor: the only required clause, and the event whose arrival makes the engine evaluate the rule at all. flow.first_failure_for_pair is the strongest signal in the system, which is why the rule is built around its arrival.
relation: causes: A claim about mechanism, and the strongest thing a clause can say about its predecessor. Reach for it only when you could defend it to a sceptical colleague. Every clause after the first must state a relation; the validator rejects one that does not.
relation: correlates: The spread clause claims nothing about mechanism. Other connections failed in the same window; that is all that is being asserted, and the CLI prints it as “and at the same time” rather than an arrow.
root_cause: change: Which clause the rule blames, named by its as. The validator refuses a rule that blames a clause it does not have.
window: 300s: How far apart the first and last clause may be. A rule with no window is refused: without one it would eventually join two unrelated events.
advice: What to look at next, never what to change. The recorder observes and does not touch the network, and it does not tell anyone else to either. “Restart the interface” would destroy the evidence on a flapping port. A test enforces that every shipped rule has advice.

Clause fields

as: Names the clause so root_cause can refer to it
kinds: The event kinds that satisfy this clause; any one of them
where: Narrows by attrs.<name>, min_severity or subject_kind. Attributes compare as text, because a collector may write a bool and the store returns json.Number
min_count: Requires several matches. One port going down is an event; the same port going down five times in five minutes is a fault
optional: Lets the rule fire without this clause. Before the anchor it also makes the clause a backward search
relation: causes, correlates or precedes. Required on every clause after the first
why: The sentence this step contributes to the account. It is what the operator reads instead of the field names

What the validator refuses

The engine will not load a rule that cannot work, and the same check is available from the command line so a rule fails at your desk rather than by silently matching nothing forever.

A rule with no window, or a confidence outside 1–100.
A clause that matches no kinds.
A clause after the first that does not say how it relates to the one before it.
A rule that blames a clause it does not have.
A rule in which every clause is optional — it would match everything, so it is not a rule.

That last check caught a real mistake in one of the rules shipped here, which would otherwise have sat in the library matching nothing forever.

$ netrewind rules --dir rules
$ go test ./internal/correlate/...
$ sudo lab/inject.sh all

correlate_on keeps unrelated failures apart

It names fields every matched event must agree on: subject, subject.id, attrs.<name>, or observer. Without it, two failures happening at the same moment on different machines get woven into one incident that never happened. With correlate_on: [subject], a port flapping on one switch cannot be joined to a neighbour failing on another.

Everything on this page is in the repository.

The schema contractdocs/schema.md
Rule authoringdocs/rules.md
Operating itdocs/runbook.md

Source