claude-plugin-fleet-integration/references/failure-catalog.md

The Integration Failure Catalog

Distilled from the 2026-07-21 k3s-over-ZeroTier connectivity research: 27 incidents across 12 repos, reduced to the failure classes that produced them. Full evidence (file:line citations, live verification) lives on the oleks/cluster wiki → K3s-over-ZeroTier-Connectivity page. These classes are platform-independent: a different overlay, cluster, or cloud would have failed the same way. Every playbook in this plugin exists to close one or more of them.

F1 — Undeclared state

State that exists live but in no repo. The fleet's worst instance: emmett's k3s underlay depended on a ZeroTier network join performed by hand (zerotier-cli join), declared nowhere in Nix — a rebuild would silently destroy the node's pod network (oleks/cluster#260). Same class: the ZT controller's member/IP database ("on-disk, NOT reconstructable from git"), an imperative chown that "will drift on nixos-rebuild" (Zot incident).

Detector: for every layer, produce declared and live lists and diff them. Anything live-but-undeclared is a filing-grade defect even when it works.

F2 — Half-finished migrations

The old thing is never removed; both live indefinitely. Fleet instance: the ZT Central → self-hosted-controller cutover left every host joined to BOTH networks with the SAME IP on two interfaces — a state the migration's own runbook says "does NOT work" (oleks/cluster#260). The status header said "member cutover pending" for a week+ while everything sat on the ambiguity.

Rule: removal of the old state is a tracked issue created at migration START, not an afterthought. A migration without its leave-step issue is not planned, it is started.

F3 — Hardcoded environment assumptions

A value correct on the host where it was written, exported to hosts where it is not. Instances: flannel-iface=zt7yynb4r2 hardcoded into an image for cloud nodes that have no such interface (oleks/builder-arbitrage#349); bim/howard scripts pinning MTU 1450 "matching armer" when armer is 1300/1220; an Oracle NSG admitting flannel only from armer's VCN IP while the cluster peers over the overlay.

Rule: every cross-host constant lives in exactly one module (fleet-modules/cluster-topology.nix pattern) and consumers derive from it. A comment asserting equality with a remote value is a bug waiting to be false.

F4 — Encapsulation/MTU budget ignored

Every tunnel layer spends MTU and the path may blackhole PMTUD. Fleet instance: ZT default 2800 on an OCI path of ~1335 with ICMP frag-needed dropped — "encapsulated packets exceed the physical path MTU and are silently dropped". Current stack budget: 1500 → 1300 (ZT) → 1220 (flannel WG).

Rule: when adding any encapsulation, write the full budget arithmetic down at the declaration site, and verify with a max-size no-fragment probe, not a default ping.

F5 — Half-implemented lifecycle

The create half ships; the destroy half is deferred and never lands. Fleet instance: per-node ZT identities minted and pre-authorized before boot (#430) but never revoked on node death — "the /24 leaks one address per ci node ever provisioned and allocation eventually fails — a slow-fuse outage" (oleks/builder-arbitrage#427).

Rule: create and destroy land in the same change, or the create does not merge. "Slow-fuse" is the tell: if exhaustion arrives long after the cause, the missing destroy-half will not be found during the incident.

F6 — Control plane on its own data path

The thing that admits members is reachable only through the network it admits them to. Fleet instance: the ZT controller is reached over the ZT overlay — "can be slow to re-establish, so an unbounded client could wedge a provision". Bootstrap and recovery both inherit the fragility of the thing being managed.

Rule: control/admission endpoints get an underlay path (public IP, direct route) or, better, no runtime role at all (pre-provisioned credentials).

F7 — Watchdog amplification and babysitter sprawl

Health-checkers that restart things can become the outage: "the monitor was itself the restart amplifier during the 2026-05-30 incident (fleet#87)". And they accumulate: the overlay stack grew EIGHT distinct babysitter mechanisms, two restarting k3s on a timer — each one evidence of an unfixed failure underneath.

Rule: see the add-watchdog playbook. Every watchdog needs a cooldown, a dependency-fallback guard, and a review answer to "why can't the failure it defends against be removed instead?"

F8 — Multi-layer firewalls changed separately

Two admission layers (cloud NSG + host firewall) that must agree, changed in separate steps. The failure is silent: "Oracle's external stateful firewall drops the media leg before it ever reaches the host — silently, with no log on armer."

Rule: both layers change in one reviewed commit/change-set, and the verification probe runs from OUTSIDE the outermost layer.

F9 — Stale documentation as active misinformation

Docs and comments that assert state which has since changed — worse than no docs, because they redirect the next reader confidently. Instances: a manifest header still claiming "PREREQUISITE — NOT YET SATISFIED: cert-manager is NOT installed" after it was; the same file shipping a REPLACE_ME Secret that would overwrite the live token if applied (oleks/armer#164); a gate comment claiming "iface must match --flannel-iface above" while watching a different interface (oleks/armer#166).

Rule: status assertions in docs carry a date; any change touching a resource greps for comments/docs referencing it; placeholder values must be un-appliable (absent resource, not fake value).

F10 — Guard drift

A guard/gate/precondition that watches a different resource than the one it protects. Fleet instance: armer's k3s ExecStartPre gate — built to stop k3s FATAL-exiting into a ZT flap — watches the OLD network's interface while flannel rides the NEW one (oleks/armer#166). The guard passes while the guarded thing is down, and after cutover would block startup forever.

Rule: guard and guarded derive their target from the same variable. A guard added during an incident gets re-verified after ANY change to the resource it watches.

F11 — Read-modify-write without exclusion

Concurrent allocation against a store with no compare-and-swap: "two concurrent provisions cannot read the same free address and both claim it" only because a local mutex serializes them — a second operator instance would race. Same class as Terraform state before the pg-backend decision (oleks/servers#8): git push detects the collision only after both applies mutated the real world.

Rule: allocation needs real mutual exclusion (advisory lock, CAS, single-writer) — detection-after-the-fact is not exclusion. If a partial view is possible, fail the allocation rather than risk double-assignment.

F12 — External dependency never smoke-tested from the runtime context

An endpoint the new work depends on is assumed working because the credential is right and the host answers — without one real authenticated request from the address the code will actually call from. Fleet instance (2026-07-21): a full release cycle (merge → fleet config → release → deploy) shipped against the self-hosted ZeroTier controller API, whose settings.allowManagementFrom defaults to localhost only. The token was byte-identical and correctly injected; 127.0.0.1 returned 200 while the ZT IP every provisioning pod calls from returned 401, so every ci provision failed (oleks/armer#171). The failure was predicted in a code comment and still never probed. F6 makes this class routine on this fleet: the control plane sits on the network it admits members to, so the operator's own shell is never the caller.

Rule: "reachable" and "authorized from where it will actually run" are different claims. Before building on an external endpoint, issue ONE real authenticated request — right source address, right credential fetched the way the workload fetches it, the actual verb depended on (authorization is per-route) — and paste the status code. A comment or a prior session's assertion is not a smoke test (F9); dated output is.

F13 — Preflight treated as a judgment call

The declared-vs-live audit exists and is skipped, because the state "looks fine" and running it feels like overhead. Fleet instance (2026-07-21): two independent sessions ran major ZeroTier work without invoking integration-preflight; both then failed inside its target class — hosts dual-homed on old and new networks with the same IP double-assigned and the new join declared nowhere in Nix (oleks/cluster#260, F1+F2), and the unprobed controller ACL above (oleks/armer#171, F12). Roughly two hours of diagnose-and-mitigate each, against a thirty-second check. The retro: "the session verified its outputs rigorously but its inputs lazily — every major stall traces to an unchecked precondition, not an unchecked result."

Rule: preflight is step 0 of every multi-step rollout and a non-skippable checklist gate, named as such by every rollout-shaped playbook (onboard-host, integrate-service, finish-migration). Its completion is evidence — the layer table and the dependency table, both filled in — not an opinion that the state was fine.