FR-4.5: evidence-stability gate — refuse to file a candidate whose confidence moved with the window
Phase 7 filed anxious/agent-wip/release-policy-derivation as `measured` at 11.94% of audited spend on a 30-day window. The same candidate over 7 days came back `thin` at 0.00%. Only the date range differed. A human reading either issue body alone cannot know the other exists, which is rubric.md §4a's failure mode reached through the plugin's own headline number. That was caught by a human running a second skeptical review — luck, not a control. It is now mechanical, in the same register as FR-4.2's overrule-case gate: - bin/stability-classify matches each `file` candidate against the most recent prior snapshot recording it (FR-5.2 identity) and downgrades it to a boundary question if `measurement_strength` changed or the share crossed the 2% filing threshold between windows. Movement within a label is a trend, not an instability; no prior window is `unchecked`, not unstable (FR-2.4). - The downgrade is a finding against THIS repo, not the target's — the target did not change, the auditor described it two ways. Carries an `ia-stability` marker so a re-run comments rather than duplicating. - bin/ia_store.py factors the store and identity resolution out of audit-snapshot so both scripts hash signatures identically. A divergence there would make the gate match nothing and fail open. Documented as spec FR-4.5 and rubric §5b (shipped verbatim in references/). Wired into skills/offload-audit as step 4b, before filing-plan; steps 5-6 now consume its output rather than classified.json. S4 is retired in its old form — a human triage catching a bad candidate is not a repeatable test — and re-passed as: the gate catches the real release-policy-derivation case with no human in the loop. Asserted in tests/stability.test.sh case (f) against the verbatim two-window Phase 7 output. Closes kotkan/claude-plugin-inference-arbitrage#11 Suite: 178 assertions, exit 0.
This commit is contained in:
@@ -14,7 +14,9 @@ The rubric has five parts, applied in order:
|
||||
the two in the middle.
|
||||
3. **The positioning principle** — given a pipeline, where exactly do you cut?
|
||||
4. **Two failure modes** — named, with detection questions for each.
|
||||
5. **The falsifiability triple** — the gate a candidate must pass to be filed.
|
||||
5. **The falsifiability triple** — the gate a candidate's *reasoning* must pass
|
||||
to be filed.
|
||||
**5b. The evidence-stability gate** — the gate its *evidence* must pass.
|
||||
|
||||
---
|
||||
|
||||
@@ -310,6 +312,57 @@ precisely the behavior the rubric is trying to teach.
|
||||
|
||||
---
|
||||
|
||||
## 5b. The evidence-stability gate
|
||||
|
||||
The triple in §5 asks whether the *reasoning* survives scrutiny. This asks
|
||||
whether the *evidence* does.
|
||||
|
||||
A candidate's headline is a number and a word: `measured`, 12% of audited spend.
|
||||
Both come from a window — the range of transcripts the dynamic pass happened to
|
||||
read — and the window is chosen by whoever ran the audit, usually by typing a
|
||||
default. If the same candidate reads `measured, 12%` over thirty days and
|
||||
`thin, 0%` over seven, then the audit has not measured the candidate. It has
|
||||
measured the calendar, and reported the result as a property of the plugin.
|
||||
|
||||
The damage is specific, and it is §4a's damage arriving by a new route. A human
|
||||
reading the issue sees one window's figure, with no indication that another
|
||||
window exists or disagrees. They cannot discount what they are not shown. So the
|
||||
number reads as a fact, gets acted on as a fact, and the correction — if it ever
|
||||
comes — arrives only because someone independently re-ran the audit on a
|
||||
different range and happened to notice. That is not a control. It is luck with a
|
||||
review attached.
|
||||
|
||||
Hence the same shape of rule as §5:
|
||||
|
||||
> **Evidence that changes label across windows → downgrade to boundary
|
||||
> question.** Compare each filable candidate against the most recent prior
|
||||
> window that recorded it. If the evidence-strength label differs, or if the two
|
||||
> windows land on opposite sides of the filing threshold, the candidate is not
|
||||
> filed.
|
||||
|
||||
Two boundaries make this a usable rule rather than a blanket refusal.
|
||||
|
||||
**Movement is not instability.** A candidate at 12% one window and 8% the next
|
||||
is stable: the human is told the same thing, and the difference is a trend, which
|
||||
the snapshot diff already reports as one. The gate fires only when the
|
||||
*description a human receives* would change — the label, or which side of the
|
||||
threshold the value falls on. Everything else is signal about the target, and
|
||||
suppressing it would throw away the trend the plugin exists to accumulate.
|
||||
|
||||
**Absence of history is not instability.** The first audit of a target has
|
||||
nothing to compare against, and a candidate seen for the first time has no
|
||||
earlier window. Those file normally, marked unchecked. Inventing an instability
|
||||
finding out of missing history would mean no plugin could ever be audited once,
|
||||
which contradicts the principle that a plugin with no measured history stays
|
||||
auditable on its definition alone.
|
||||
|
||||
The instability, when it fires, is a finding **about the auditor**. The target
|
||||
did not change; the tool described it two different ways. So it is reported
|
||||
against this plugin's own repo, not the target's — the same self-application
|
||||
discipline as §FR-8, applied to the plugin's own headline number.
|
||||
|
||||
---
|
||||
|
||||
## 6. Confidence scoring
|
||||
|
||||
| `boundary_confidence` | Requires |
|
||||
|
||||
Reference in New Issue
Block a user