integration-preflight is skipped in practice — a situation-shaped mandate with only phrase-shaped triggers and no enforcement #2

New Issue

2026-08-07T01:28:45+03:00

oleks commented

2026-08-07 01:28:45 +03:00

The gap

skills/integration-preflight/SKILL.md declares itself unambiguously mandatory:

MANDATORY step 0 of any multi-step rollout on the fleet — invoke it BEFORE the first change, not when drift is suspected. Use it whenever work will span more than one step (onboarding a host, integrating a service, a network/controller cutover, a merge→release→deploy cycle)

and the body reinforces it:

This is step 0 of every multi-step rollout, and it is not optional. … It is a checklist gate, not a judgment call: you do not get to decide the state "looks fine" and skip it.

Despite that, it keeps not firing. The body already records two skips on 2026-07-21, each costing "roughly two hours of diagnose-and-mitigate … against a thirty-second check". This issue records a third.

The third occurrence (2026-08-06/07)

A session ran three merge→release→deploy cycles on emmett — a plugin rename landed across two repos, then services.metamcp.servers.*.command updates, then a flake-input bump — and never invoked the skill. Surfaced afterwards by decision-flow:audit-flow.

It would very likely have caught a real bug. The skill's stated purpose is that it "diffs declared configuration against live reality". The session's first deploy (generation 767) succeeded and changed nothing, because metamcp's seed SQL guarded its INSERT with WHERE NOT EXISTS and its UPDATE only wrote env/error_status — so command could never reconcile on an existing row. Declared Nix config vs live DB state had silently diverged: precisely this skill's target class. That was filed and fixed separately as oleks/emmett#274, at the cost of a wasted deploy cycle and the diagnosis time to work out why a successful deploy had no effect.

Why this is NOT a trigger-phrase problem

The obvious fix is to widen the description trigger list, and I think that is wrong:

The scope prose already covers it, verbatim. "a merge→release→deploy cycle" is listed explicitly. Adding more phrasing duplicates existing coverage.
The phrases that would have matched are too generic to add. The user's actual words were "…and deploy" and "go". Adding those makes the skill fire on nearly every request and collide with cluster:ship-it-agent and cluster:nixos-agent.

The mismatch is structural: the mandate is situation-shaped ("any multi-step rollout"), but the discovery mechanism is phrase-shaped (trigger words in a description). A model that never scans the skill list for a request phrased as "fix this and deploy" will miss it every time, no matter how emphatic the prose is. Three skips across three independent sessions is the evidence that prose alone does not carry this.

Suggested direction

Enforcement rather than phrasing. Rough options, in increasing cost:

A PreToolUse hook that fires on rollout-shaped actions — nix run .#deploy, deploy-rs, nixos-rebuild switch, flux reconcile, helm upgrade — and blocks (or warns once per session) unless a preflight has been recorded for this session. Same shape as the guards already shipping in anxious (issuer-ladder-guard.py) and cluster (gitea-agent-tea-write-guard.py), so there is precedent in-fleet for exactly this pattern.
A session-scoped marker (e.g. under ${CLAUDE_PLUGIN_DATA}) written when the preflight runs, so the hook can distinguish "ran it" from "never ran it" and stay quiet for the rest of the session.
If a hard block is too aggressive for a one-line deploy of an already-reviewed change, a single warning on the first rollout action would still have closed all three recorded occurrences.

Not doing this unilaterally

Filed rather than implemented — this is a behavioural change to a plugin I don't own, and the block-vs-warn call and the exact action list are judgment calls for the owner. Happy to implement whichever shape you prefer.

Context: found via decision-flow:audit-flow; the sibling routing-miss it surfaced was fixed in anxious:wip-claim v0.33.4.

## The gap `skills/integration-preflight/SKILL.md` declares itself unambiguously mandatory: > **MANDATORY step 0 of any multi-step rollout on the fleet** — invoke it BEFORE the first change, not when drift is suspected. Use it whenever work will span more than one step (onboarding a host, integrating a service, a network/controller cutover, **a merge→release→deploy cycle**) and the body reinforces it: > **This is step 0 of every multi-step rollout, and it is not optional.** … It is a checklist gate, not a judgment call: you do not get to decide the state "looks fine" and skip it. Despite that, it keeps not firing. The body already records two skips on 2026-07-21, each costing "roughly two hours of diagnose-and-mitigate … against a thirty-second check". **This issue records a third.** ## The third occurrence (2026-08-06/07) A session ran three merge→release→deploy cycles on emmett — a plugin rename landed across two repos, then `services.metamcp.servers.*.command` updates, then a flake-input bump — and never invoked the skill. Surfaced afterwards by `decision-flow:audit-flow`. **It would very likely have caught a real bug.** The skill's stated purpose is that it "diffs declared configuration against live reality". The session's first deploy (generation 767) succeeded and changed nothing, because metamcp's seed SQL guarded its INSERT with `WHERE NOT EXISTS` and its UPDATE only wrote `env`/`error_status` — so `command` could never reconcile on an existing row. Declared Nix config vs live DB state had silently diverged: precisely this skill's target class. That was filed and fixed separately as oleks/emmett#274, at the cost of a wasted deploy cycle and the diagnosis time to work out why a successful deploy had no effect. ## Why this is NOT a trigger-phrase problem The obvious fix is to widen the `description` trigger list, and I think that is wrong: 1. **The scope prose already covers it, verbatim.** "a merge→release→deploy cycle" is listed explicitly. Adding more phrasing duplicates existing coverage. 2. **The phrases that would have matched are too generic to add.** The user's actual words were "…and deploy" and "go". Adding those makes the skill fire on nearly every request and collide with `cluster:ship-it-agent` and `cluster:nixos-agent`. The mismatch is structural: the mandate is **situation-shaped** ("any multi-step rollout"), but the discovery mechanism is **phrase-shaped** (trigger words in a description). A model that never scans the skill list for a request phrased as "fix this and deploy" will miss it every time, no matter how emphatic the prose is. Three skips across three independent sessions is the evidence that prose alone does not carry this. ## Suggested direction Enforcement rather than phrasing. Rough options, in increasing cost: 1. **A `PreToolUse` hook** that fires on rollout-shaped actions — `nix run .#deploy`, `deploy-rs`, `nixos-rebuild switch`, `flux reconcile`, `helm upgrade` — and blocks (or warns once per session) unless a preflight has been recorded for this session. Same shape as the guards already shipping in `anxious` (`issuer-ladder-guard.py`) and `cluster` (`gitea-agent-tea-write-guard.py`), so there is precedent in-fleet for exactly this pattern. 2. **A session-scoped marker** (e.g. under `${CLAUDE_PLUGIN_DATA}`) written when the preflight runs, so the hook can distinguish "ran it" from "never ran it" and stay quiet for the rest of the session. 3. If a hard block is too aggressive for a one-line deploy of an already-reviewed change, a **single warning on the first rollout action** would still have closed all three recorded occurrences. ## Not doing this unilaterally Filed rather than implemented — this is a behavioural change to a plugin I don't own, and the block-vs-warn call and the exact action list are judgment calls for the owner. Happy to implement whichever shape you prefer. Context: found via `decision-flow:audit-flow`; the sibling routing-miss it surfaced was fixed in `anxious:wip-claim` v0.33.4.

oleks added the kind/capability-gap reliability labels 2026-08-07 02:18:12 +03:00

oleks commented

2026-08-07 02:18:29 +03:00

Triage — priority raised: the description-level fix has already been tried and did not hold

Found #1 after filing this, and it changes the assessment materially.

#1 (closed 2026-07-21) was exactly the "fix the description" remedy. Per its body, it "reframed as step 0 of every multi-step rollout, non-optional", inlined the retro quote, added the dependency smoke test, and — critically —

description: frontmatter rewritten so the skill routes on rollout-shaped work rather than only on drift-shaped questions

So the prose-and-triggers approach was already applied, deliberately and well, in direct response to the first two skips. The occurrence reported here happened after that fix was in place. That is the strongest argument in this issue and it was missing from the original body: this is not "the description could be better", it is "the description was already improved for precisely this reason and a session still skipped it."

That is what moves the remaining fix from phrasing to enforcement. Three skips, one of which occurred after a targeted description rewrite, is enough evidence that a description cannot carry a situation-shaped mandate on its own.

Priority: medium-high, and I'd act on it

Recurrence: 3 known, across independent sessions, spanning ~2.5 weeks.
Cost per occurrence: ~2h each for the 2026-07-21 pair (per the skill body). This one cost a wasted deploy cycle plus the diagnosis time to work out why a successful deploy had changed nothing — the answer being oleks/emmett#274, declared-vs-live drift, this skill's exact target class.
Remedy already exhausted: the cheap fix (#1) is done and did not close it.
Effort: small — the guard-hook pattern already ships twice in-fleet (anxious/hooks/issuer-ladder-guard.py, cluster/hooks/gitea-agent-tea-write-guard.py), so this is a copy-and-adapt, not a design from scratch.

Suggested scope for a first cut

Start with warn-once-per-session, not a hard block. It would have closed all three recorded occurrences, and it avoids the obvious objection that a one-line redeploy of an already-reviewed change should not need a gate. Escalate to blocking only if a warn-level hook also gets ignored — which would itself be the useful next data point.

Labelled kind/capability-gap + reliability. (This repo had no labels at all; created those two to match the vocabulary oleks/claude-plugin-cluster already uses, rather than inventing a parallel taxonomy.)

## Triage — priority raised: the description-level fix has already been tried and did not hold Found #1 after filing this, and it changes the assessment materially. **#1 (closed 2026-07-21) was exactly the "fix the description" remedy.** Per its body, it "reframed as **step 0 of every multi-step rollout, non-optional**", inlined the retro quote, added the dependency smoke test, and — critically — > `description:` frontmatter rewritten so the skill routes on rollout-shaped work rather than only on drift-shaped questions So the prose-and-triggers approach was already applied, deliberately and well, in direct response to the first two skips. **The occurrence reported here happened after that fix was in place.** That is the strongest argument in this issue and it was missing from the original body: this is not "the description could be better", it is "the description was already improved for precisely this reason and a session still skipped it." That is what moves the remaining fix from *phrasing* to *enforcement*. Three skips, one of which occurred after a targeted description rewrite, is enough evidence that a `description` cannot carry a situation-shaped mandate on its own. ### Priority: medium-high, and I'd act on it - **Recurrence:** 3 known, across independent sessions, spanning ~2.5 weeks. - **Cost per occurrence:** ~2h each for the 2026-07-21 pair (per the skill body). This one cost a wasted deploy cycle plus the diagnosis time to work out why a *successful* deploy had changed nothing — the answer being oleks/emmett#274, declared-vs-live drift, this skill's exact target class. - **Remedy already exhausted:** the cheap fix (#1) is done and did not close it. - **Effort:** small — the guard-hook pattern already ships twice in-fleet (`anxious/hooks/issuer-ladder-guard.py`, `cluster/hooks/gitea-agent-tea-write-guard.py`), so this is a copy-and-adapt, not a design from scratch. ### Suggested scope for a first cut Start with **warn-once-per-session**, not a hard block. It would have closed all three recorded occurrences, and it avoids the obvious objection that a one-line redeploy of an already-reviewed change should not need a gate. Escalate to blocking only if a warn-level hook also gets ignored — which would itself be the useful next data point. Labelled `kind/capability-gap` + `reliability`. (This repo had no labels at all; created those two to match the vocabulary `oleks/claude-plugin-cluster` already uses, rather than inventing a parallel taxonomy.)

oleks commented

2026-08-07 15:06:35 +03:00

Further occurrence — 2026-08-07, session ee47ac17 (mempalace e5-migration grind).

What the work was: deploy the maintenance-mode gate (oleks/mempalace#100) to the mempalace-native canary. Concretely — tag v3.5.0-24 → Woodpecker builds and publishes the image → edit apps/base/mempalace/native-canary.yaml in oleks/fleet → Flux reconcile → verify the pod.

That is a merge→release→deploy cycle spanning five steps, touching the fleet repo, depending on three external endpoints (Woodpecker CI, the Gitea OCI registry, the k8s API). It matches this skill's description almost verbatim — the description names "a merge→release→deploy cycle" explicitly. integration-preflight was not invoked.

This is an occurrence report, and it sharpens the issue's thesis rather than just incrementing a counter:

The trigger phrases were not the problem. The description already covers this situation in words. The user's request was deploy #100 so we can cut over — no phrase-shaped trigger fires on that, but the situation is squarely in scope. That is the "situation-shaped mandate with phrase-shaped triggers" gap this issue names, observed again. Consistent with the wider finding that description widening is a dead lever, no trigger-phrase edit would have caught this one either.

What makes this occurrence more interesting than a clean miss: the session hit the target class and handled it ad hoc. Two pieces of undeclared / out-of-band state showed up mid-rollout, both of which are exactly what "declared vs live" exists to surface:

The image the canary was actually running (3.5.0-oracle-e5-migration-arm64) could not have been produced by CI. .woodpecker/arm64.yaml derives the version with sed 's/^v//; s/-[0-9]*$//', which strips only a numeric suffix, and the result must key into versions.nix. That tag came from a manual nix run .#publish-* — out-of-band state, presented by the live cluster as if it were a normal release.
Because of the above, the live pin looked like evidence that descriptive tags are a supported release pattern. They are not: tagging v3.5.0-maintenance-mode would have failed the build with No hash for mempalace 3.5.0-maintenance-mode.

I caught both by hand — grepping versions.nix and the tag-derivation sed before tagging — which cost a couple of minutes and avoided a failed release. But that is precisely the check this gate systematizes, done informally and by luck of curiosity rather than by checklist. No damage, and that is the point: a skipped gate that happens not to bite is still a skipped gate, and next time the ad-hoc check may not happen.

Also relevant to the enforcement question: a PreToolUse hook did fire during this session's audit when a SKILL.md was read rather than invoked (decision-flow's skill-bypass-notice). It works, and it is a live demonstration that the "one layer down" remedy proposed here is buildable — but it only fires on a read, and in this occurrence the skill was never touched at all, so nothing would have fired. An enforcement hook for this gate has to key on the situation (a write to the fleet repo, or a release tag push) rather than on any interaction with the skill file.

Not proposing a fix here, and not touching labels — recording the occurrence per audit-flow's tracking rule.

**Further occurrence — 2026-08-07, session `ee47ac17` (mempalace e5-migration grind).** What the work was: deploy the maintenance-mode gate (oleks/mempalace#100) to the `mempalace-native` canary. Concretely — tag `v3.5.0-24` → Woodpecker builds and publishes the image → edit `apps/base/mempalace/native-canary.yaml` in **oleks/fleet** → Flux reconcile → verify the pod. That is a merge→release→deploy cycle spanning five steps, touching the fleet repo, depending on three external endpoints (Woodpecker CI, the Gitea OCI registry, the k8s API). It matches this skill's description almost verbatim — the description names "a merge→release→deploy cycle" explicitly. **`integration-preflight` was not invoked.** This is an occurrence report, and it sharpens the issue's thesis rather than just incrementing a counter: **The trigger phrases were not the problem.** The description already covers this situation in words. The user's request was `deploy #100 so we can cut over` — no phrase-shaped trigger fires on that, but the *situation* is squarely in scope. That is the "situation-shaped mandate with phrase-shaped triggers" gap this issue names, observed again. Consistent with the wider finding that description widening is a dead lever, no trigger-phrase edit would have caught this one either. **What makes this occurrence more interesting than a clean miss: the session hit the target class and handled it ad hoc.** Two pieces of undeclared / out-of-band state showed up mid-rollout, both of which are exactly what "declared vs live" exists to surface: 1. The image the canary was actually running (`3.5.0-oracle-e5-migration-arm64`) **could not have been produced by CI**. `.woodpecker/arm64.yaml` derives the version with `sed 's/^v//; s/-[0-9]*$//'`, which strips only a numeric suffix, and the result must key into `versions.nix`. That tag came from a manual `nix run .#publish-*` — out-of-band state, presented by the live cluster as if it were a normal release. 2. Because of the above, the live pin looked like evidence that descriptive tags are a supported release pattern. They are not: tagging `v3.5.0-maintenance-mode` would have failed the build with `No hash for mempalace 3.5.0-maintenance-mode`. I caught both by hand — grepping `versions.nix` and the tag-derivation sed *before* tagging — which cost a couple of minutes and avoided a failed release. But that is precisely the check this gate systematizes, done informally and by luck of curiosity rather than by checklist. **No damage, and that is the point: a skipped gate that happens not to bite is still a skipped gate**, and next time the ad-hoc check may not happen. Also relevant to the enforcement question: a `PreToolUse` hook *did* fire during this session's audit when a `SKILL.md` was read rather than invoked (decision-flow's `skill-bypass-notice`). It works, and it is a live demonstration that the "one layer down" remedy proposed here is buildable — but it only fires on a **read**, and in this occurrence the skill was never touched at all, so nothing would have fired. An enforcement hook for this gate has to key on the *situation* (a write to the fleet repo, or a release tag push) rather than on any interaction with the skill file. Not proposing a fix here, and not touching labels — recording the occurrence per `audit-flow`'s tracking rule.

oleks commented

2026-08-07 15:16:03 +03:00

Full session record for the occurrence above: Session Report ee47ac17 — §4 records this as the 4th occurrence and the 2nd after the #1 description rewrite, and §5 suggestion 3 argues the count is now past the threshold to build the warn-once hook.

Full session record for the occurrence above: [Session Report ee47ac17](https://git.oleks.space/kotkan/claude-plugin-decision-flow/wiki/Session-Report-ee47ac17) — §4 records this as the 4th occurrence and the 2nd after the #1 description rewrite, and §5 suggestion 3 argues the count is now past the threshold to build the warn-once hook.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: oleks/claude-plugin-fleet-integration#2