One record, one clock, and a chain that names every link

Administrative state — someone shut the port — is distinguished from operational state — the carrier dropped — through attrs.cause. Reporting both as “interface down” is what makes most tools useless mid-incident: the two have completely different causes and completely different fixes. link.flap counts only carrier drops, never administrative ones: a run of outages is a different finding from one outage, but an engineer shutting a port repeatedly is somebody working, and counting that as flapping blames the cable for the engineer. link.error_rate_high is not a state change at all — it comes from the interface error counters crossing a proportion of total packets.

l2.arp_binding_changed opens more incidents than anything else at this layer: a binding moving under a running network is exactly what a hijack, a failover and a misconfigured static entry all look like from here. When the address it moved under is the default gateway it is raised to an error rather than a warning — the same observation, a much larger consequence. l2.lldp_neighbor_changed and l2.vlan_seen are in the schema; the LLDP collector is not written, so switch-side topology is the largest thing this recorder still cannot see.

A change is reported only when the route that actually wins for a destination changes, so a standby path being installed is not mistaken for traffic moving. The last two come from reading ICMP rather than from netlink: l3.mtu_blackhole is the one worth knowing about, because a path that silently drops large packets while small ones sail through is the fault most often misdiagnosed as an application problem.

flow.first_failure_for_pair is the strongest single signal in the system. An unanswered connection on its own is ordinary — closed ports are closed. An unanswered connection between two machines that were talking a moment ago means something changed. Individual connections are not events: a busy segment opens thousands a second, so everything ordinary is aggregated into a flow.rollup every ten seconds and only anomalies earn a row. A second tracepoint separates a connection that was refused outright from one that simply stopped answering, which are different faults with the same symptom. That is also why there is deliberately no flow.open or flow.close: a row per connection would tell an operator nothing a counter could not. flow.retransmit_spike is declared and not yet produced.

Metadata only, never payloads: a cBPF filter attached to the socket admits IPv4 ICMP and UDP on three ports, and nothing else is ever copied to userspace. It is written as instructions in the repository rather than compiled from a tcpdump string at runtime, because a filter that decides what a process is ever handed is a security boundary and should be reviewable as one. dhcp.server_seen — a second DHCP server answering on a segment — is one of the events most likely to open an incident, and it is the fault that is hardest to find by hand because the machines that took the wrong lease are not the machine that is misbehaving.

The ruleset is read every five seconds and compared by digest, so a change to what the firewall permits lands on the same timeline as its consequences — which is what lets a filtering change appear as the opening link of a causal chain. Each rule is qualified by the table and chain that contains it, so the evidence reads table inet filter / chain input :: tcp dport 9300 drop: comparing bare rule text would call a rule moved between chains no change at all, and that is exactly the sort of edit that breaks a network quietly. What is deliberately not here is the consequence of a filtering change. A pair that stopped connecting is observable whatever did the blocking — a rule on this box, an ACL on a switch, a firewall three hops away — so that belongs to flow.first_failure_for_pair; an event here could only ever see the first of the three, which would be the narrowest case dressed as the general one. policy.drop_burst is declared and needs counters that are not read yet.

The only collector that puts a packet on the network. Everything else here is passive, which leaves a hole: loss is an absence, and nothing announces an absence. A handful of small probes a minute go to targets given to it or discovered from the routing table, and metric.anomaly is emitted when the answers stop coming back or start taking much longer than they used to — measured against that target’s own baseline rather than a fixed threshold, because a link that always ran at 40 ms is not in trouble at 40 ms. Interface counters and conntrack table size are specified as series and are not collected yet.

The only family that is not observed, and the only one with nothing feeding it. It would come from NetIntent, or from diffing device configuration over SSH or SNMP. It is the family that lets the timeline answer what were we doing when it broke, and the change-broke-a-path rule already matches on it — so the day something feeds it, that rule gets better without being edited.

Never remove these. system.gap is measured against a heartbeat written every ten seconds and states exactly how long the recorder was blind. system.drop states how many events had to be thrown away when they arrived faster than they could be read. All five are produced today, and two of the nineteen shipped rules exist only to surface them: one for a hole in the record, one for a single collector that died while the rest kept running — which is the more dangerous of the two, because the timeline goes on looking healthy.