FR-4.5: evidence-stability gate — refuse to file a candidate whose confidence moved with the window

Phase 7 filed anxious/agent-wip/release-policy-derivation as `measured` at
11.94% of audited spend on a 30-day window. The same candidate over 7 days came
back `thin` at 0.00%. Only the date range differed. A human reading either issue
body alone cannot know the other exists, which is rubric.md §4a's failure mode
reached through the plugin's own headline number.

That was caught by a human running a second skeptical review — luck, not a
control. It is now mechanical, in the same register as FR-4.2's overrule-case
gate:

- bin/stability-classify matches each `file` candidate against the most recent
  prior snapshot recording it (FR-5.2 identity) and downgrades it to a boundary
  question if `measurement_strength` changed or the share crossed the 2% filing
  threshold between windows. Movement within a label is a trend, not an
  instability; no prior window is `unchecked`, not unstable (FR-2.4).
- The downgrade is a finding against THIS repo, not the target's — the target
  did not change, the auditor described it two ways. Carries an `ia-stability`
  marker so a re-run comments rather than duplicating.
- bin/ia_store.py factors the store and identity resolution out of
  audit-snapshot so both scripts hash signatures identically. A divergence there
  would make the gate match nothing and fail open.

Documented as spec FR-4.5 and rubric §5b (shipped verbatim in references/).
Wired into skills/offload-audit as step 4b, before filing-plan; steps 5-6 now
consume its output rather than classified.json.

S4 is retired in its old form — a human triage catching a bad candidate is not a
repeatable test — and re-passed as: the gate catches the real
release-policy-derivation case with no human in the loop. Asserted in
tests/stability.test.sh case (f) against the verbatim two-window Phase 7 output.

Closes kotkan/claude-plugin-inference-arbitrage#11
Suite: 178 assertions, exit 0.
This commit is contained in:
oleks
2026-07-29 20:24:31 +03:00
committed by Oleks
parent 105e9f5bae
commit dcec9cabfb
16 changed files with 1119 additions and 123 deletions
+54 -1
View File
@@ -14,7 +14,9 @@ The rubric has five parts, applied in order:
the two in the middle.
3. **The positioning principle** — given a pipeline, where exactly do you cut?
4. **Two failure modes** — named, with detection questions for each.
5. **The falsifiability triple** — the gate a candidate must pass to be filed.
5. **The falsifiability triple** — the gate a candidate's *reasoning* must pass
to be filed.
**5b. The evidence-stability gate** — the gate its *evidence* must pass.
---
@@ -310,6 +312,57 @@ precisely the behavior the rubric is trying to teach.
---
## 5b. The evidence-stability gate
The triple in §5 asks whether the *reasoning* survives scrutiny. This asks
whether the *evidence* does.
A candidate's headline is a number and a word: `measured`, 12% of audited spend.
Both come from a window — the range of transcripts the dynamic pass happened to
read — and the window is chosen by whoever ran the audit, usually by typing a
default. If the same candidate reads `measured, 12%` over thirty days and
`thin, 0%` over seven, then the audit has not measured the candidate. It has
measured the calendar, and reported the result as a property of the plugin.
The damage is specific, and it is §4a's damage arriving by a new route. A human
reading the issue sees one window's figure, with no indication that another
window exists or disagrees. They cannot discount what they are not shown. So the
number reads as a fact, gets acted on as a fact, and the correction — if it ever
comes — arrives only because someone independently re-ran the audit on a
different range and happened to notice. That is not a control. It is luck with a
review attached.
Hence the same shape of rule as §5:
> **Evidence that changes label across windows → downgrade to boundary
> question.** Compare each filable candidate against the most recent prior
> window that recorded it. If the evidence-strength label differs, or if the two
> windows land on opposite sides of the filing threshold, the candidate is not
> filed.
Two boundaries make this a usable rule rather than a blanket refusal.
**Movement is not instability.** A candidate at 12% one window and 8% the next
is stable: the human is told the same thing, and the difference is a trend, which
the snapshot diff already reports as one. The gate fires only when the
*description a human receives* would change — the label, or which side of the
threshold the value falls on. Everything else is signal about the target, and
suppressing it would throw away the trend the plugin exists to accumulate.
**Absence of history is not instability.** The first audit of a target has
nothing to compare against, and a candidate seen for the first time has no
earlier window. Those file normally, marked unchecked. Inventing an instability
finding out of missing history would mean no plugin could ever be audited once,
which contradicts the principle that a plugin with no measured history stays
auditable on its definition alone.
The instability, when it fires, is a finding **about the auditor**. The target
did not change; the tool described it two different ways. So it is reported
against this plugin's own repo, not the target's — the same self-application
discipline as §FR-8, applied to the plugin's own headline number.
---
## 6. Confidence scoring
| `boundary_confidence` | Requires |