Initial commit: fleet-integration plugin

Six integration playbooks (integration-preflight, onboard-host,
integrate-service, ephemeral-lifecycle, add-watchdog, finish-migration)
plus the F1-F11 failure catalog, distilled from the 2026-07-21
k3s-over-ZeroTier connectivity research (oleks/cluster wiki:
K3s-over-ZeroTier-Connectivity). Failure classes over current facts;
live sources of truth over restated values.

.claude-plugin/plugin.json

@@ -0,0 +1,24 @@
+{
+	"name": "fleet-integration",
+	"version": "1.0.0",
+	"description": "Integration playbooks for the oleks fleet, distilled from the 2026-07-21 k3s-over-ZeroTier connectivity research (27 incidents, 9 recurring patterns — catalogued on the oleks/cluster wiki page K3s-over-ZeroTier-Connectivity). Core finding: the failures were integration-shaped, not platform-shaped — undeclared state, half-finished migrations, hardcoded environment assumptions, half-implemented lifecycles, watchdog amplification. Six skills cover the levels of the system: integration-preflight (declared-vs-live drift audit before building on top), onboard-host (join a host to the fleet network + k3s), integrate-service (service end-to-end: chart, Flux, ingress, DNS, secrets, DB, TLS, deploy tracking), ephemeral-lifecycle (scale-to-zero credential/member lifecycle rules), add-watchdog (health-check design that doesn't amplify failures), finish-migration (cutover discipline: done means old state REMOVED and declared==live). Playbooks encode failure classes, not current facts — they direct verification against live sources of truth (fleet-modules/cluster-topology.nix, the wiki) rather than restating values that rot.",
+	"author": {
+		"name": "oleks",
+		"email": "plugins@oleks.space"
+	},
+	"repository": "https://claude-plugins.oleks.space/plugins/fleet-integration.git",
+	"license": "MIT",
+	"keywords": [
+		"integration",
+		"playbook",
+		"fleet",
+		"onboarding",
+		"migration",
+		"lifecycle",
+		"watchdog",
+		"runbook",
+		"k3s",
+		"zerotier",
+		"self-hosted"
+	]
+}

.gitignore

@@ -0,0 +1 @@
+.claude/

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Oleks Kuksenko
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,46 @@
+# fleet-integration
+
+Integration playbooks for the oleks fleet, distilled from the 2026-07-21
+k3s-over-ZeroTier connectivity research (27 incidents across 12 repos; full
+evidence on the **oleks/cluster wiki → K3s-over-ZeroTier-Connectivity** page).
+
+**Core finding:** the fleet's failures were *integration-shaped, not
+platform-shaped*. A different overlay, cluster, or cloud would have failed the
+same way — through undeclared state, half-finished migrations, hardcoded
+environment assumptions, half-implemented lifecycles, and watchdogs that
+amplify. These playbooks encode the failure classes (durable) rather than
+current fleet facts (which rot — stale docs were themselves failure class F9).
+Where a current value matters, the playbooks point at the live source of truth
+(`fleet-modules/cluster-topology.nix`, the wiki) instead of restating it.
+
+## Skills
+
+| Skill | When | Closes |
+|---|---|---|
+| `integration-preflight` | Before integrating anything — diff declared vs live per layer | F1 F2 F3 F9 |
+| `onboard-host` | New host into the fleet network and/or k3s | F1 F3 F4 F8 F10 |
+| `integrate-service` | New workload end to end: chart, Flux, ingress, DNS, secrets, DB, TLS, tracking | F3 F8 F9 F11 |
+| `ephemeral-lifecycle` | Scale-to-zero nodes: credentials, admission, allocation, reaping | F5 F6 F11 |
+| `add-watchdog` | Any health-check/auto-restart mechanism, new or under review | F7 F10 |
+| `finish-migration` | Cutovers — planning them, and auditing half-done ones | F1 F2 F9 F10 |
+
+## The failure catalog
+
+`references/failure-catalog.md` defines F1–F11 with the real incidents behind
+each (issue refs in full `owner/repo#N` form). Read it once; the skills cite
+it by class.
+
+- **F1** Undeclared state · **F2** Half-finished migrations · **F3** Hardcoded
+  environment assumptions · **F4** MTU/encapsulation budget ignored · **F5**
+  Half-implemented lifecycle · **F6** Control plane on its own data path ·
+  **F7** Watchdog amplification · **F8** Multi-layer firewalls changed
+  separately · **F9** Stale documentation as active misinformation · **F10**
+  Guard drift · **F11** Read-modify-write without exclusion
+
+## Provenance
+
+Built from: the tracker sweep (oleks/armer, oleks/cluster, oleks/fleet,
+oleks/builder-arbitrage, oleks/matrix-moq and others), a file:line code sweep
+of `~/projects/servers` and `builder-arbitrage`, live fleet verification, and
+session memory. Defects discovered during the research were filed as
+oleks/armer#166 and oleks/cluster#260 before this plugin cited them.

references/failure-catalog.md

@@ -0,0 +1,140 @@
+# The Integration Failure Catalog
+
+Distilled from the 2026-07-21 k3s-over-ZeroTier connectivity research: 27
+incidents across 12 repos, reduced to the failure *classes* that produced them.
+Full evidence (file:line citations, live verification) lives on the
+**oleks/cluster wiki → K3s-over-ZeroTier-Connectivity** page. These classes are
+platform-independent: a different overlay, cluster, or cloud would have failed
+the same way. Every playbook in this plugin exists to close one or more of
+them.
+
+## F1 — Undeclared state
+
+State that exists live but in no repo. The fleet's worst instance: emmett's
+k3s underlay depended on a ZeroTier network join performed by hand
+(`zerotier-cli join`), declared nowhere in Nix — a rebuild would silently
+destroy the node's pod network (oleks/cluster#260). Same class: the ZT
+controller's member/IP database ("on-disk, NOT reconstructable from git"), an
+imperative `chown` that "will drift on nixos-rebuild" (Zot incident).
+
+**Detector:** for every layer, produce declared and live lists and diff them.
+Anything live-but-undeclared is a filing-grade defect even when it works.
+
+## F2 — Half-finished migrations
+
+The old thing is never removed; both live indefinitely. Fleet instance: the ZT
+Central → self-hosted-controller cutover left every host joined to BOTH
+networks with the SAME IP on two interfaces — a state the migration's own
+runbook says "does NOT work" (oleks/cluster#260). The status header said
+"member cutover pending" for a week+ while everything sat on the ambiguity.
+
+**Rule:** removal of the old state is a tracked issue created at migration
+START, not an afterthought. A migration without its leave-step issue is not
+planned, it is started.
+
+## F3 — Hardcoded environment assumptions
+
+A value correct on the host where it was written, exported to hosts where it
+is not. Instances: `flannel-iface=zt7yynb4r2` hardcoded into an image for
+cloud nodes that have no such interface (oleks/builder-arbitrage#349);
+bim/howard scripts pinning MTU 1450 "matching armer" when armer is 1300/1220;
+an Oracle NSG admitting flannel only from armer's VCN IP while the cluster
+peers over the overlay.
+
+**Rule:** every cross-host constant lives in exactly one module
+(`fleet-modules/cluster-topology.nix` pattern) and consumers derive from it.
+A comment asserting equality with a remote value is a bug waiting to be false.
+
+## F4 — Encapsulation/MTU budget ignored
+
+Every tunnel layer spends MTU and the path may blackhole PMTUD. Fleet
+instance: ZT default 2800 on an OCI path of ~1335 with ICMP frag-needed
+dropped — "encapsulated packets exceed the physical path MTU and are silently
+dropped". Current stack budget: 1500 → 1300 (ZT) → 1220 (flannel WG).
+
+**Rule:** when adding any encapsulation, write the full budget arithmetic down
+at the declaration site, and verify with a max-size no-fragment probe, not a
+default ping.
+
+## F5 — Half-implemented lifecycle
+
+The create half ships; the destroy half is deferred and never lands. Fleet
+instance: per-node ZT identities minted and pre-authorized before boot (#430)
+but never revoked on node death — "the /24 leaks one address per ci node ever
+provisioned and allocation eventually fails — a slow-fuse outage"
+(oleks/builder-arbitrage#427).
+
+**Rule:** create and destroy land in the same change, or the create does not
+merge. "Slow-fuse" is the tell: if exhaustion arrives long after the cause,
+the missing destroy-half will not be found during the incident.
+
+## F6 — Control plane on its own data path
+
+The thing that admits members is reachable only through the network it admits
+them to. Fleet instance: the ZT controller is reached over the ZT overlay —
+"can be slow to re-establish, so an unbounded client could wedge a provision".
+Bootstrap and recovery both inherit the fragility of the thing being managed.
+
+**Rule:** control/admission endpoints get an underlay path (public IP, direct
+route) or, better, no runtime role at all (pre-provisioned credentials).
+
+## F7 — Watchdog amplification and babysitter sprawl
+
+Health-checkers that restart things can become the outage: "the monitor was
+itself the restart amplifier during the 2026-05-30 incident (fleet#87)". And
+they accumulate: the overlay stack grew EIGHT distinct babysitter mechanisms,
+two restarting k3s on a timer — each one evidence of an unfixed failure
+underneath.
+
+**Rule:** see the add-watchdog playbook. Every watchdog needs a cooldown, a
+dependency-fallback guard, and a review answer to "why can't the failure it
+defends against be removed instead?"
+
+## F8 — Multi-layer firewalls changed separately
+
+Two admission layers (cloud NSG + host firewall) that must agree, changed in
+separate steps. The failure is silent: "Oracle's external stateful firewall
+drops the media leg before it ever reaches the host — silently, with no log on
+armer."
+
+**Rule:** both layers change in one reviewed commit/change-set, and the
+verification probe runs from OUTSIDE the outermost layer.
+
+## F9 — Stale documentation as active misinformation
+
+Docs and comments that assert state which has since changed — worse than no
+docs, because they redirect the next reader confidently. Instances: a manifest
+header still claiming "PREREQUISITE — NOT YET SATISFIED: cert-manager is NOT
+installed" after it was; the same file shipping a `REPLACE_ME` Secret that
+would overwrite the live token if applied (oleks/armer#164); a gate comment
+claiming "iface must match --flannel-iface above" while watching a different
+interface (oleks/armer#166).
+
+**Rule:** status assertions in docs carry a date; any change touching a
+resource greps for comments/docs referencing it; placeholder values must be
+un-appliable (absent resource, not fake value).
+
+## F10 — Guard drift
+
+A guard/gate/precondition that watches a different resource than the one it
+protects. Fleet instance: armer's k3s ExecStartPre gate — built to stop k3s
+FATAL-exiting into a ZT flap — watches the OLD network's interface while
+flannel rides the NEW one (oleks/armer#166). The guard passes while the
+guarded thing is down, and after cutover would block startup forever.
+
+**Rule:** guard and guarded derive their target from the same variable. A
+guard added during an incident gets re-verified after ANY change to the
+resource it watches.
+
+## F11 — Read-modify-write without exclusion
+
+Concurrent allocation against a store with no compare-and-swap: "two
+concurrent provisions cannot read the same free address and both claim it"
+only because a local mutex serializes them — a second operator instance would
+race. Same class as Terraform state before the pg-backend decision
+(oleks/servers#8): git push detects the collision only after both applies
+mutated the real world.
+
+**Rule:** allocation needs real mutual exclusion (advisory lock, CAS,
+single-writer) — detection-after-the-fact is not exclusion. If a partial view
+is possible, fail the allocation rather than risk double-assignment.

skills/add-watchdog/SKILL.md

@@ -0,0 +1,73 @@
+---
+name: add-watchdog
+description: This skill should be used when the user asks to "add a health check", "add a watchdog", "auto-restart a service when it breaks", "add a babysitter timer", "monitor and restart X", or when reviewing an existing health-check module. It encodes the design rules that keep a watchdog from becoming the outage — learned from a monitor that was itself the restart amplifier.
+---
+
+# Add a Watchdog — without becoming the outage
+
+The fleet's overlay stack accumulated EIGHT distinct babysitter mechanisms,
+two of which restart k3s itself on a timer. One of them made an incident
+worse: "the monitor was itself the restart amplifier during the 2026-05-30
+incident (fleet#87)" (F7, `references/failure-catalog.md`). A watchdog is a
+confession that something underneath is unfixed — sometimes a necessary one,
+but it must be designed so the confession cannot escalate the crime.
+
+## Before writing it — the exit question
+
+Answer in the module header: **why can't the failure this defends against be
+removed instead?** Sometimes it can: half the fleet's babysitters exist to
+survive an overlay flap that a simpler underlay would not have. If the honest
+answer is "it can, later", link the tracking issue for that removal — the
+watchdog is then explicitly temporary. A babysitter without this paragraph is
+sprawl (F7).
+
+## Design rules
+
+1. **Cooldown/lockout after every action.** The in-tree pattern: 10-minute
+   lockout after any restart — "prevents restart storms". A watchdog that can
+   fire back-to-back is an amplifier by construction.
+
+2. **Dependency-fallback guard.** Check the failure is attributable to the
+   watched thing before acting: the ZT self-health unit "skips ZT restart if
+   internet is also unreachable". Restarting a healthy service because its
+   dependency is down converts one outage into two.
+
+3. **Dampen anything that restarts what it monitors.** If the action is
+   `systemctl restart` of the monitored service, require N consecutive
+   failures over M minutes, not one probe miss — a transient stall escalated
+   into a hard restart loop is exactly fleet#87.
+
+4. **Watched values derive from the config they guard — same variable, same
+   module** (F10). A gate hardcoding an interface name drifted from the
+   flannel config it protected and ended up guarding the WRONG interface
+   (oleks/armer#166): passing while the real underlay was down, and set to
+   block startup entirely after the migration. Never write the watched
+   iface/IP/MTU as a literal if the guarded config reads it from a module.
+
+5. **Every firing is observable.** Log with a stable tag, and alert on
+   REPEATED firings — a watchdog firing hourly is masking a real defect, and
+   silence about that masking is how eight babysitters accumulate. Ideally
+   export a counter metric.
+
+6. **Actions are proportionate and bounded.** Escalation ladder (log → probe
+   again → restart leaf → restart service), never jumping to the top; and
+   never an action that can sever the operator's own access path (a network
+   watchdog must not be able to take down the SSH path it shares).
+
+7. **Re-verify after ANY change to the watched resource.** Watchdogs are
+   written during incidents and never revisited; a migration that renames the
+   watched resource silently voids them (the fleet had a unit ordered after a
+   service name that didn't exist — ordering "silently ignored on all
+   consumers", oleks/fleet-modules#5). Add the watchdog to the migration
+   checklist of whatever it watches.
+
+## Review checklist
+
+- [ ] header answers the exit question (or links the removal issue)
+- [ ] cooldown present; no back-to-back firing possible
+- [ ] dependency-fallback guard present
+- [ ] restart-of-monitored requires consecutive failures
+- [ ] watched values derived, not literal
+- [ ] firings logged with stable tag; repeated-firing alert or metric
+- [ ] action cannot sever operator access
+- [ ] listed in the watched resource's migration checklist

skills/ephemeral-lifecycle/SKILL.md

@@ -0,0 +1,71 @@
+---
+name: ephemeral-lifecycle
+description: This skill should be used when the user asks to "design an ephemeral node lifecycle", "add scale-to-zero workers", "provision ephemeral builders", "review the builder join path", "why do ephemeral nodes leak", "credential lifecycle for spot nodes". It encodes the rules for components that are created and destroyed continuously — where half-implemented lifecycles become slow-fuse outages.
+---
+
+# Ephemeral Lifecycle — scale-to-zero without slow fuses
+
+Ephemeral nodes (spot builders, ci workers) fail differently from static
+hosts: their integration bugs are LIFECYCLE bugs, and the worst ones detonate
+long after their cause. The catalogued example: per-node network identities
+minted at provision but never revoked at death — "the /24 leaks one address
+per ci node ever provisioned and allocation eventually fails — a slow-fuse
+outage rather than an obvious one" (oleks/builder-arbitrage#427).
+
+## The rules
+
+1. **Create and destroy land in the same change.** A PR that mints
+   credentials/members/allocations but defers revocation "to a follow-up"
+   does not merge (F5). The follow-up historically never lands, and the leak
+   is invisible until the pool exhausts — at which point EVERY provision
+   fails at once and the cause is months old.
+
+2. **Credentials are minted BEFORE boot by the provisioner** — never
+   generated on first boot. First-boot identity means nothing exists to
+   pre-authorize, so the node joins unauthorized, gets no address, and the
+   admission controller becomes a runtime dependency of every boot. The
+   working pattern in-tree: `Grant() = Mint + Authorize` before the instance
+   exists, with the identity injected via provisioning metadata.
+
+3. **No control plane on its own data path** (F6). If admission requires
+   calling a controller, that call must not traverse the network the node is
+   being admitted to. Give the controller an underlay path — or remove the
+   runtime call entirely via pre-provisioned credentials.
+
+4. **Every provision-path network call is bounded.** Timeouts on the client,
+   a burned-timeout alarm on the operator. An unbounded call into a
+   slow-to-re-establish path wedges the whole provision loop.
+
+5. **Allocation needs real exclusion** (F11). Read-modify-write against a
+   store with no CAS is a race the moment a second operator replica exists.
+   If the allocator can have a partial view, FAIL the allocation — a
+   double-assigned address "silently breaks BOTH nodes".
+
+6. **Fail-closed needs an alarm.** A node that boots, is refused admission,
+   and burns a bootstrap timeout is correct but expensive — the refusal must
+   surface as a distinct, attributable error, not a generic timeout
+   (oleks/cluster#250's cost).
+
+7. **Reap is verified, not assumed.** Best-effort revoke ("revoke failed;
+   address leaked until swept") requires an actual sweeper: a periodic
+   reconcile of controller members vs live instances, with a metric. If the
+   sweep doesn't exist, the best-effort revoke is rule 1 violated with extra
+   steps.
+
+8. **Prefer no membership at all.** Public-IP ephemeral nodes may not need
+   the overlay: kernel-WireGuard flannel peering over the public underlay
+   (`--node-external-ip` + cloud firewall for exactly the cluster ports)
+   deletes rules 2–7 for that node class. Precedent: the runner/nix-builder
+   classes are overlay-free by explicit design; only ci-class carries the
+   membership machinery. Every lifecycle rule you can delete beats every
+   lifecycle rule you can implement.
+
+## Review checklist for an ephemeral-node PR
+
+- [ ] destroy-half present and exercised in the same change
+- [ ] credentials pre-minted, injected via metadata, nothing generated at boot
+- [ ] no admission call over the admitted network
+- [ ] all provision-path calls bounded, refusals distinctly attributable
+- [ ] allocation exclusive or fail-closed; partial view → abort
+- [ ] sweeper reconciles membership vs live instances, with a metric
+- [ ] the "does this class need membership at all?" question answered in a comment

skills/finish-migration/SKILL.md

@@ -0,0 +1,75 @@
+---
+name: finish-migration
+description: This skill should be used when the user asks to "plan a migration", "cut over to a new system", "finish the migration", "retire the old network/service/backend", "why are both systems still running", "audit a half-done cutover". It encodes cutover discipline — a migration is done only when the old state is REMOVED and declared config matches live — learned from a controller migration that left every host dual-homed on the same subnet.
+---
+
+# Finish a Migration — done means the old thing is GONE
+
+The fleet's defining example (F2, `references/failure-catalog.md`): the
+ZeroTier Central → self-hosted-controller migration deployed the new
+controller, cut members over — and never left the old network. Result,
+discovered months of incidents later: every host joined to BOTH networks with
+the SAME IP on two interfaces, a state the migration's own runbook calls out
+as not working; packet routing decided by route ordering; every diagnosis
+carrying the unknown "which network did this packet use?"; and hosts whose
+new-network membership existed only as undeclared hand-run state
+(oleks/cluster#260). The gate protecting k3s startup still watched the OLD
+interface (oleks/armer#166).
+
+## Plan-time rules (before any cutover work starts)
+
+1. **The leave-step is a tracked issue created at migration START.** "Remove
+   old X everywhere" gets filed with the design, in full `owner/repo#N` form,
+   blocking migration-done. A migration without its leave-step issue is not
+   planned — it is merely started, and F2 is its default outcome.
+
+2. **Inventory every reference to the old resource NOW.** Grep the fleet for
+   the old name/id/interface across: configs, start gates and watchdogs
+   (F10), firewall rules, comments and docs (F9), monitoring, and hardcoded
+   values in images. Each hit becomes a checklist line. The references you
+   don't inventory are the ones that bite after cutover — the gate watching
+   `ztpp6dw7uf` was exactly an un-inventoried reference.
+
+3. **Forbid coexistence ambiguity explicitly.** If old and new can claim the
+   same identity (same subnet, same name, same port), the plan must serialize
+   them — the runbook's own words: "Same-subnet dual-homing does NOT work."
+   If coexistence is genuinely needed for the transition, it must be
+   asymmetric (different subnets/names) and time-boxed with a date.
+
+## Execution rules
+
+1. **Every step lands declaratively.** A cutover action run by hand on a host
+   (`zerotier-cli join`, a console toggle) is F1 the moment it succeeds:
+   config and reality diverge, and a rebuild reverts the migration silently.
+   If a probe/emergency action must be imperative, its declarative
+   equivalent lands the same day and the runbook records both.
+
+2. **The status header is updated at EVERY step, with a date.** "Status
+   (2026-07-14): member cutover pending" that still reads that way a week
+   later is active misinformation for the next reader (F9). Stale-dated
+   status is itself a finding.
+
+3. **Guards, gates and watchdogs cut over WITH their resource, in the same
+   change** — not before (they'd block the old state) and not after (they
+   guard nothing during the gap, or block the new state at the next restart).
+
+## Done-criteria (all four, verified live)
+
+- [ ] Old state REMOVED everywhere — not disabled, not "will clean up later":
+      left, deleted, de-seeded. Verified by live enumeration on every host.
+- [ ] Declared == live on every touched host (run `integration-preflight`
+      as the exit audit).
+- [ ] The reference inventory from rule 2 is fully checked off — including
+      comments and docs.
+- [ ] The leave-step issue closes with the verification evidence quoted, and
+      the runbook's status header reads "complete" with today's date.
+
+## Auditing an existing half-done migration
+
+Symptoms: two of something where the design says one; a status header frozen
+at a past date; hand-run state a rebuild would lose; guards referencing names
+that no longer match. File what you find (full `owner/repo#N`), then drive it
+through the done-criteria above — resolution order matters: declare the live
+state first (make Nix match reality), fix the guards, THEN remove the old
+state. Removing the old state first, while hosts still depend on it
+undeclared, converts drift into an outage.

skills/integrate-service/SKILL.md

@@ -0,0 +1,82 @@
+---
+name: integrate-service
+description: This skill should be used when the user asks to "integrate a new service", "deploy a new app to the cluster", "wire a service end to end", "add a service to the fleet", "stand up a new workload", "new helm chart into flux". It is the end-to-end playbook — chart, GitOps, ingress, DNS, secrets, database, TLS, deploy tracking — with the failure classes each step must avoid.
+---
+
+# Integrate a Service — end to end
+
+The order below is a dependency chain surfaced by real failures: the QUIC
+ingress work (oleks/matrix-moq#42) spent days on certificates and firewall
+plumbing for a relay that was never deployed — §4 before §5. Deploy the
+workload's skeleton first; everything else attaches to it.
+
+## 1. Chart + GitOps
+
+- Self-contained chart in the app's own repo; the fleet repo carries only a
+  thin pointer (HelmRelease + GitRepository). Flux reconciles from `main` —
+  never deploy a chart by hand (`kubectl apply` of app resources creates
+  state matching no reviewed commit).
+- Node placement: justify any `nodeSelector` in a comment with the REAL
+  reason for THIS workload. The MoQ relay inherited a "sub-ms link to armer
+  Postgres" pin from Synapse while having no database at all — and that
+  inherited pin forced an entire cross-host DNAT/SNAT apparatus (F3 applied
+  to scheduling). Match image architecture to the pinned node (arm64 for
+  kotkan).
+- Submodules: Flux does not fetch them by default — anything the deploy needs
+  must live in the chart path, not a submodule.
+
+## 2. Ingress
+
+- The cluster edge is Caddy (caddy-ingress-watcher), not an Ingress
+  controller: one host → one Service, Ingress paths are DISCARDED. Path
+  routing goes in the `caddy.oleks.space/extra-config` annotation.
+- Anything the edge must NOT terminate (WebTransport/QUIC, raw TLS to the
+  pod) is an L4 concern: NodePort + host-level forward. Then F8 applies —
+  cloud rule and host rule in one change — and if the packet path crosses
+  hosts, write down the return path explicitly (asymmetric return is
+  silently dropped; the SNAT that fixes it costs client source-IP, which
+  must then be documented as NOT preserved — oleks/matrix-moq#70).
+
+## 3. DNS + TLS
+
+- DNS through the cloudflare-dns Terranix config only — never the console.
+  Decide proxied (orange) vs DNS-only (grey) deliberately: anything
+  end-to-end-encrypted or UDP needs grey; note the proxied wildcard means an
+  unlisted subdomain resolves orange by default.
+- Certificates via cert-manager against the existing `cloudflare-dns01`
+  ClusterIssuer — reference it, never duplicate it. NEVER commit a
+  placeholder Secret value: a `REPLACE_ME` next to a live secret of the same
+  name is one `kubectl apply` away from destroying the working credential
+  (F9, oleks/armer#164). Document out-of-band secret creation; keep the
+  resource itself out of appliable manifests.
+
+## 4. Secrets + database
+
+- Secrets via sops in the owning host repo; passwords applied by the
+  idempotent post-start pattern (`postgresql-set-passwords`), never in the
+  Nix store.
+- Databases via the declarative `serviceDbs` path (own DB + owning role)
+  where collation allows. pg_hba gets the NARROWEST stanza that works — a
+  dedicated `database role network` line, deliberately excluding networks
+  that have no business connecting (the `terraform_state` stanza admits the
+  operator range and deliberately not the pod CIDR — copy that discipline,
+  with a comment saying why).
+
+## 5. State and allocation
+
+- If the service allocates from any shared pool (IPs, ports, names): real
+  mutual exclusion or fail-closed (F11). Detection-after-the-fact (git
+  non-fast-forward, "the second apply failed") is not exclusion — the damage
+  is already live by then.
+
+## 6. Deploy tracking + verification
+
+- NixOS-side changes: `nixos-deploy-pending` on the issue the moment source
+  is pushed; comment the host's rolling deploy issue; committed ≠ live.
+- Close issues only on LIVE verification: the resource exists on the
+  cluster, the endpoint answers from outside the outermost firewall layer,
+  and the check is quoted in the closing comment. A green `helm template` or
+  a successful reconcile message is not verification.
+- Grep for docs/comments referencing what this integration changed and
+  update them in the same change (F9) — a stale "NOT YET SATISFIED" header
+  redirects the next reader away from working infrastructure.

skills/integration-preflight/SKILL.md

@@ -0,0 +1,56 @@
+---
+name: integration-preflight
+description: This skill should be used before integrating anything new into the fleet, or when the user asks to "run an integration preflight", "check declared vs live", "audit config drift", "is the fleet state clean", "verify state before building on top", "what out-of-band state exists". It diffs declared configuration against live reality so new work is not built on undeclared or half-migrated state.
+---
+
+# Integration Preflight — declared vs live
+
+Run this BEFORE integrating a new host, service, or lifecycle into the fleet.
+Its purpose is to refuse to build on drifted foundations. The 2026-07-21
+research found the k3s underlay of a node depended on a network join that
+existed nowhere in any repo (failure class F1, `references/failure-catalog.md`)
+— and every diagnosis on the fleet carried the unknown "which network did this
+packet actually use?" because a migration was half-finished (F2).
+
+## Procedure
+
+1. **Name the layers the new work will touch.** Typical stack: cloud firewall
+   (NSG / Security List), host firewall, overlay network, cluster membership,
+   pod network, DNS, TLS, storage, database admittance, deploy pipeline.
+
+2. **For each layer, produce the DECLARED list from its single source of
+   truth.** Fleet sources: `fleet-modules/cluster-topology.nix` (network id,
+   interface, IPs, MTUs), host repos under `~/projects/servers/<host>/`
+   (joins, firewall, k3s flags), `terranix/` configs (cloud rules, DNS),
+   fleet HelmReleases, sops files. If two sources disagree about the same
+   value, stop — that disagreement is a finding by itself (F3).
+
+3. **Produce the LIVE list.** Read-only probes only: `zerotier-cli
+   listnetworks` / `peers`, `ip -d link` / `ip addr`, live firewall rulesets,
+   `kubectl get nodes -o wide`, actual DNS answers (`dig` pinned at the
+   origin), the real cloud rule set. Never fix anything during the audit.
+
+4. **Diff, and classify every mismatch:**
+   - live-but-undeclared → F1; a rebuild destroys it. Filing-grade even if
+     everything currently works.
+   - declared-but-not-live → committed-but-never-deployed, or a failed apply.
+     Check the deploy trackers (`nixos-deploy-pending` labels, rolling issues)
+     before assuming.
+   - both-present-but-different (two networks, two MTUs, two IPs) → F2/F3;
+     check for a migration doc and read its dated status header.
+
+5. **Check the documentation freshness of every runbook you relied on** (F9):
+   a status header without a date, or dated before the last incident touching
+   that area, is untrustworthy — verify its claims live before acting on them.
+
+6. **File drift before building.** Every mismatch becomes a tracker issue in
+   full `owner/repo#N` form (create first, then cite — numbers written in
+   advance land wrong). Only proceed with the integration once the new work
+   does not DEPEND on any unfixed drift; drift in unrelated layers may be
+   filed and left.
+
+## Output
+
+Produce a short table: layer | declared | live | verdict (clean / F-class +
+issue ref). The preflight is complete when every layer the new work touches
+reads "clean".

skills/onboard-host/SKILL.md

@@ -0,0 +1,76 @@
+---
+name: onboard-host
+description: This skill should be used when the user asks to "onboard a host", "add a node to the fleet", "join a machine to the network", "add a k3s node", "wire a new server into the cluster", "connect a new box to zerotier". It is the level-by-level playbook for bringing a host into the fleet network and (optionally) the k3s cluster without recreating the catalogued integration failures.
+---
+
+# Onboard a Host — network level, then cluster level
+
+Bring a host into the fleet in two strictly ordered levels. Do not start
+level 2 until level 1 verifies. Run `integration-preflight` first if any
+existing drift could interact with the join.
+
+## Level 0 — decide whether the host needs the overlay at all
+
+If the host has a **public, routable IP** and only needs cluster membership
+(not the home-LAN peers), prefer the overlay-free path: kernel-WireGuard
+flannel peering over the public underlay with `--node-external-ip`, cloud
+firewall admitting exactly the cluster ports. The research's Option E: the
+ephemeral runner/nix-builder classes are already overlay-free BY DESIGN
+("must never be given overlay membership"). Every overlay member added is
+identity-lifecycle surface (F5) and controller load (F6) — pay it only for
+hosts behind NAT.
+
+## Level 1 — fleet network membership
+
+1. **Declaratively only.** Join via the fleet membership module
+   (`fleet-modules` zerotier-membership pattern), never by hand on the host.
+   A hand join works immediately and becomes F1 (undeclared state) forever —
+   this exact mistake cost a node its declared pod-network underlay
+   (oleks/cluster#260).
+2. **One network.** The host joins exactly the network id exported by
+   `cluster-topology.nix`. If the host is currently on another/older network,
+   this is a migration — invoke `finish-migration`, do not stack joins.
+   Same-subnet dual-homing is forbidden by the controller runbook.
+3. **MTU from the topology module, never defaulted.** Verify after join:
+   interface MTU equals the declared `ztMtu`, and probe the path with a
+   max-size no-fragment packet (F4 — PMTUD is blackholed on parts of the
+   underlay; a default ping proves nothing).
+4. **Controller/membership lifecycle:** static hosts get a pinned IP declared
+   in the controller's seed config AND in `cluster-topology.nix` (same value,
+   one review). Ephemeral hosts: use `ephemeral-lifecycle` instead.
+5. **Verify:** membership OK, assigned IP matches topology, peer path DIRECT
+   (or knowingly relayed), and — from another fleet host — reachability at
+   the declared IP.
+
+## Level 2 — k3s membership
+
+1. **Node addressing:** `--node-ip` (and `--node-external-ip` where relevant)
+   set explicitly from the topology module. Never let k3s guess an interface.
+2. **Pod network:** flannel backend AND iface AND MTU all derived from the
+   topology module — the backend must match the cluster's
+   (`wireguard-native`), the MTU must be `flannelMtu` (ztMtu − 80). An image
+   that defaults to vxlan against a wireguard-native cluster "can never peer"
+   (oleks/builder-arbitrage#349). Never hardcode an interface name into an
+   image — cloud hosts will not have it (F3).
+3. **Both firewall layers in ONE change** (F8): the cloud NSG/Security-List
+   rule and the host firewall rule land in the same commit/change-set. The
+   outer layer drops silently with no log on the host — a split change
+   produces an undiagnosable black hole. Verify from OUTSIDE the outer layer.
+4. **Bidirectional check before declaring success:** apiserver→kubelet:10250
+   (this is the one that 502s when membership half-works) and node→apiserver.
+   Then a pod-level check: schedule a pod on the new node and resolve CoreDNS
+   + clone from the internal git host — DNS-then-clone is where half-formed
+   overlays actually surface (pattern 5 in the catalog).
+5. **Guards:** if any start-gate or watchdog watches an interface/IP for this
+   host, it derives from the same topology variable as the config it guards
+   (F10 — a gate watching a different iface than flannel used boobytrapped a
+   whole cutover, oleks/armer#166).
+6. **Record and track:** add the host to `cluster-topology.nix`, monitoring,
+   and the deploy trackers (`nixos-deploy-pending` until activation is
+   verified live; rolling issue per host).
+
+## Done means
+
+Every value the host runs with traces to a declared source; both firewall
+layers agree; a pod on the host can resolve and clone; and a teardown of the
+host leaves no leaked membership (check the controller's member list).