oleks/claude-plugin-fleet-integration
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

preflight: make integration-preflight a mandatory step 0 with a dependency smoke test

Browse Source 
Closes-part-of oleks/cluster#271.

- integration-preflight: framed as the non-skippable step 0 of any
  multi-step rollout; new procedure step 6 requires ONE real authenticated
  request against every external endpoint FROM THE RUNTIME CONTEXT, with
  the allowManagementFrom case (oleks/armer#171) as the worked example;
  output now carries a dependency table alongside the layer table;
  description rewritten so the skill actually routes on rollout-shaped work.
- onboard-host / integrate-service / finish-migration: each names
  integration-preflight as its step 0 entry gate.
- failure-catalog: adds F12 (external dependency never smoke-tested from
  the runtime context, oleks/armer#171) and F13 (preflight treated as a
  judgment call, oleks/cluster#260 + oleks/armer#171).
- README + plugin.json 1.0.0 -> 1.0.1.
This commit is contained in:
Oleks
2026-07-21 16:12:49 +03:00
parent 4ac127a1a8
commit 8a28233417
7 changed files with 150 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude-plugin/plugin.json
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "fleet-integration",
"version": "1.0.0",
"version": "1.0.1",
"description": "Integration playbooks for the oleks fleet, distilled from the 2026-07-21 k3s-over-ZeroTier connectivity research (27 incidents, 9 recurring patterns — catalogued on the oleks/cluster wiki page K3s-over-ZeroTier-Connectivity). Core finding: the failures were integration-shaped, not platform-shaped — undeclared state, half-finished migrations, hardcoded environment assumptions, half-implemented lifecycles, watchdog amplification. Six skills cover the levels of the system: integration-preflight (declared-vs-live drift audit before building on top), onboard-host (join a host to the fleet network + k3s), integrate-service (service end-to-end: chart, Flux, ingress, DNS, secrets, DB, TLS, deploy tracking), ephemeral-lifecycle (scale-to-zero credential/member lifecycle rules), add-watchdog (health-check design that doesn't amplify failures), finish-migration (cutover discipline: done means old state REMOVED and declared==live). Playbooks encode failure classes, not current facts — they direct verification against live sources of truth (fleet-modules/cluster-topology.nix, the wiki) rather than restating values that rot.",
"author": {
"name": "oleks",
README.md
+6 -4
View File
@@ -16,8 +16,8 @@ Where a current value matters, the playbooks point at the live source of truth
## Skills
| Skill | When | Closes |
|---|---|---|
| `integration-preflight` | Before integrating anything — diff declared vs live per layer | F1 F2 F3 F9 |
| --- | --- | --- |
| `integration-preflight` | **Step 0 of every multi-step rollout (mandatory)** — diff declared vs live per layer, then smoke-test every external dependency from the runtime context | F1 F2 F3 F9 F12 F13 |
| `onboard-host` | New host into the fleet network and/or k3s | F1 F3 F4 F8 F10 |
| `integrate-service` | New workload end to end: chart, Flux, ingress, DNS, secrets, DB, TLS, tracking | F3 F8 F9 F11 |
| `ephemeral-lifecycle` | Scale-to-zero nodes: credentials, admission, allocation, reaping | F5 F6 F11 |
@@ -26,7 +26,7 @@ Where a current value matters, the playbooks point at the live source of truth
## The failure catalog
`references/failure-catalog.md` defines F1–F11 with the real incidents behind
`references/failure-catalog.md` defines F1–F13 with the real incidents behind
each (issue refs in full `owner/repo#N` form). Read it once; the skills cite
it by class.
@@ -35,7 +35,9 @@ it by class.
Half-implemented lifecycle · **F6** Control plane on its own data path ·
**F7** Watchdog amplification · **F8** Multi-layer firewalls changed
separately · **F9** Stale documentation as active misinformation · **F10**
Guard drift · **F11** Read-modify-write without exclusion
Guard drift · **F11** Read-modify-write without exclusion · **F12** External
dependency never smoke-tested from the runtime context · **F13** Preflight
treated as a judgment call
## Provenance
references/failure-catalog.md
+40
View File
@@ -138,3 +138,43 @@ mutated the real world.
**Rule:** allocation needs real mutual exclusion (advisory lock, CAS,
single-writer) — detection-after-the-fact is not exclusion. If a partial view
is possible, fail the allocation rather than risk double-assignment.
## F12 — External dependency never smoke-tested from the runtime context
An endpoint the new work depends on is assumed working because the credential
is right and the host answers — without one real authenticated request from
the address the code will actually call from. Fleet instance (2026-07-21): a
full release cycle (merge → fleet config → release → deploy) shipped against
the self-hosted ZeroTier controller API, whose `settings.allowManagementFrom`
defaults to localhost only. The token was byte-identical and correctly
injected; `127.0.0.1` returned 200 while the ZT IP every provisioning pod
calls from returned 401, so every ci provision failed (oleks/armer#171). The
failure was *predicted in a code comment* and still never probed. F6 makes
this class routine on this fleet: the control plane sits on the network it
admits members to, so the operator's own shell is never the caller.
**Rule:** "reachable" and "authorized from where it will actually run" are
different claims. Before building on an external endpoint, issue ONE real
authenticated request — right source address, right credential fetched the
way the workload fetches it, the actual verb depended on (authorization is
per-route) — and paste the status code. A comment or a prior session's
assertion is not a smoke test (F9); dated output is.
## F13 — Preflight treated as a judgment call
The declared-vs-live audit exists and is skipped, because the state "looks
fine" and running it feels like overhead. Fleet instance (2026-07-21): two
independent sessions ran major ZeroTier work without invoking
`integration-preflight`; both then failed inside its target class — hosts
dual-homed on old and new networks with the same IP double-assigned and the
new join declared nowhere in Nix (oleks/cluster#260, F1+F2), and the
unprobed controller ACL above (oleks/armer#171, F12). Roughly two hours of
diagnose-and-mitigate each, against a thirty-second check. The retro:
"the session verified its outputs rigorously but its inputs lazily — every
major stall traces to an unchecked precondition, not an unchecked result."
**Rule:** preflight is step 0 of every multi-step rollout and a non-skippable
checklist gate, named as such by every rollout-shaped playbook
(`onboard-host`, `integrate-service`, `finish-migration`). Its completion is
evidence — the layer table and the dependency table, both filled in — not an
opinion that the state was fine.
skills/finish-migration/SKILL.md
+14
View File
@@ -16,6 +16,20 @@ new-network membership existed only as undeclared hand-run state
(oleks/cluster#260). The gate protecting k3s startup still watched the OLD
interface (oleks/armer#166).
## Step 0 — entry gate: run `integration-preflight` (mandatory)
A cutover is the archetypal multi-step rollout, so `integration-preflight`
runs before plan-time rule 1 — as the ENTRY audit, not only as the exit
audit in the done-criteria below. It establishes what the old and new state
actually are (you cannot plan a removal against a live inventory you have
not taken) and smoke-tests every endpoint the cutover depends on with one
real authenticated request from the runtime context. Both 2026-07-21 ZeroTier
sessions skipped this gate (F13): one built on top of the dual-homing it
would have surfaced (oleks/cluster#260), the other shipped a full release
cycle against a controller API that answered 401 to every non-localhost
caller (oleks/armer#171, F12). Do not write the plan until both preflight
tables read clean.
## Plan-time rules (before any cutover work starts)
1. **The leave-step is a tracked issue created at migration START.** "Remove
skills/integrate-service/SKILL.md
+13
View File
@@ -10,6 +10,19 @@ ingress work (oleks/matrix-moq#42) spent days on certificates and firewall
plumbing for a relay that was never deployed — §4 before §5. Deploy the
workload's skeleton first; everything else attaches to it.
## 0. Entry gate — run `integration-preflight` (mandatory)
Integrating a service is a multi-step rollout, so step 0 is always
`integration-preflight`: declared-vs-live per layer, plus one real
authenticated request against every external endpoint this service will
depend on — registry, database, controller/admission API, webhook target,
auth issuer — issued FROM THE RUNTIME CONTEXT (the pod or runner, not your
laptop). "Reachable" and "authorized from where it will actually run" are
different claims (F12, oleks/armer#171: same token, 200 from localhost, 401
from the ZeroTier IP the pods call from — an entire release-and-deploy cycle
shipped on top of it). Treating this gate as optional is F13. Do not write
the chart until both preflight tables read clean.
## 1. Chart + GitOps
- Self-contained chart in the app's own repo; the fleet repo carries only a
skills/integration-preflight/SKILL.md
+63 -6
View File
@@ -1,11 +1,23 @@
---
name: integration-preflight
description: This skill should be used before integrating anything new into the fleet, or when the user asks to "run an integration preflight", "check declared vs live", "audit config drift", "is the fleet state clean", "verify state before building on top", "what out-of-band state exists". It diffs declared configuration against live reality so new work is not built on undeclared or half-migrated state.
description: MANDATORY step 0 of any multi-step rollout on the fleet — invoke it BEFORE the first change, not when drift is suspected. Use it whenever work will span more than one step (onboarding a host, integrating a service, a network/controller cutover, a merge→release→deploy cycle), whenever new work will depend on an external endpoint (controller/admission API, registry, database, webhook target), and when the user asks to "run an integration preflight", "check declared vs live", "audit config drift", "is the fleet state clean", "verify state before building on top", "what out-of-band state exists". It diffs declared configuration against live reality, and makes one real authenticated request against every external dependency from the runtime context, so new work is not built on undeclared, half-migrated, or unauthorized-from-where-it-runs state.
---
# Integration Preflight — declared vs live
Run this BEFORE integrating a new host, service, or lifecycle into the fleet.
**This is step 0 of every multi-step rollout, and it is not optional.** Any
work that spans more than one change — onboarding a host, integrating a
service, a cutover, a release-and-deploy cycle — starts here. It is a
checklist gate, not a judgment call: you do not get to decide the state
"looks fine" and skip it. Two sessions on 2026-07-21 skipped it and both
landed squarely in its target class (F13, `references/failure-catalog.md`),
at a cost of roughly two hours of diagnose-and-mitigate each against a
thirty-second check. The retro's finding, which is the whole reason this
gate exists:
> the session verified its outputs rigorously but its inputs lazily — every
> major stall traces to an unchecked precondition, not an unchecked result.
Its purpose is to refuse to build on drifted foundations. The 2026-07-21
research found the k3s underlay of a node depended on a network join that
existed nowhere in any repo (failure class F1, `references/failure-catalog.md`)
@@ -43,7 +55,45 @@ packet actually use?" because a migration was half-finished (F2).
a status header without a date, or dated before the last incident touching
that area, is untrustworthy — verify its claims live before acting on them.
6. **File drift before building.** Every mismatch becomes a tracker issue in
6. **Smoke-test every external dependency, from the runtime context. Not
skippable.** Enumerate every endpoint the new work will depend on —
controller/admission API, container registry, webhook target, database,
object store, DNS resolver, auth issuer — and make ONE real, authenticated
request against each **from where the code will actually run**: inside the
pod / on the CI runner / on the target host, not from your laptop and not
from the endpoint's own box. One `curl`, one `psql -c 'select 1'`, one
`skopeo inspect`. Then paste the status code or the response into the
plan.
**"Reachable" and "authorized from where it will actually run" are
different claims, and only the second one counts.** Worked example
(oleks/armer#171, F12): a self-hosted ZeroTier controller's API was
fronted by `settings.allowManagementFrom`, which defaults to localhost
only. The token was byte-identical to the one in the secret and correctly
injected. From `127.0.0.1` on the controller host it returned `200`. From
the ZeroTier IP — the address every provisioning pod actually calls from —
the same token on the same endpoint returned `401`. Nothing about the
credential was wrong; the *source address* was. A full merge → fleet
config → release → deploy cycle shipped against it and every CI provision
failed 401. The failure had even been predicted in a code comment. One
request from the right place would have caught it before any of it.
Rules that follow:
- Test from the runtime source address. A probe run on the endpoint's own
host, or through a different network path, proves nothing about the
caller (F6 makes this routine: the control plane often sits on the very
network it admits members to).
- Test with the real credential the workload will present, fetched the way
the workload fetches it — not a personal token from your shell history.
- Exercise the actual verb you depend on. A `GET /status` that answers
does not establish that `POST /controller/network/.../member/...` is
permitted; authorization is usually per-route.
- A comment, a runbook, or a prior session asserting the endpoint works is
not a smoke test (F9). Dated, pasted output is.
- An endpoint you cannot reach from the runtime context is a BLOCKER, not
a note: file it and stop, do not build the rollout on top of it.
7. **File drift before building.** Every mismatch becomes a tracker issue in
full `owner/repo#N` form (create first, then cite — numbers written in
advance land wrong). Only proceed with the integration once the new work
does not DEPEND on any unfixed drift; drift in unrelated layers may be
@@ -51,6 +101,13 @@ packet actually use?" because a migration was half-finished (F2).
## Output
Produce a short table: layer | declared | live | verdict (clean / F-class +
issue ref). The preflight is complete when every layer the new work touches
reads "clean".
Produce two short tables:
1. **Layers:** layer | declared | live | verdict (clean / F-class + issue ref).
2. **Dependencies:** endpoint | called from (pod/runner/host) | credential |
verb | result (pasted status code).
The preflight is complete when every layer the new work touches reads
"clean" AND every external dependency has a pasted, authenticated response
obtained from the runtime context. Either table left blank means the
preflight did not run.
skills/onboard-host/SKILL.md
+13 -2
View File
@@ -6,8 +6,19 @@ description: This skill should be used when the user asks to "onboard a host", "
# Onboard a Host — network level, then cluster level
Bring a host into the fleet in two strictly ordered levels. Do not start
level 2 until level 1 verifies. Run `integration-preflight` first if any
existing drift could interact with the join.
level 2 until level 1 verifies.
## Step 0 — entry gate: run `integration-preflight` (mandatory)
Onboarding is a multi-step rollout, so `integration-preflight` runs FIRST,
every time — not "if drift seems likely". It produces the declared-vs-live
diff for the layers this join touches AND the authenticated smoke test of
every endpoint the join depends on (controller API, registry, cluster
API), issued from the runtime context. Skipping this gate is failure class
F13 (`references/failure-catalog.md`); the controller API that admits the
new member is exactly the endpoint that answered 200 from localhost and 401
from the network in oleks/armer#171 (F12). Do not begin level 1 until both
preflight tables read clean.
## Level 0 — decide whether the host needs the overlay at all