FR-4.5: evidence-stability gate — refuse to file a candidate whose confidence moved with the window
Phase 7 filed anxious/agent-wip/release-policy-derivation as `measured` at 11.94% of audited spend on a 30-day window. The same candidate over 7 days came back `thin` at 0.00%. Only the date range differed. A human reading either issue body alone cannot know the other exists, which is rubric.md §4a's failure mode reached through the plugin's own headline number. That was caught by a human running a second skeptical review — luck, not a control. It is now mechanical, in the same register as FR-4.2's overrule-case gate: - bin/stability-classify matches each `file` candidate against the most recent prior snapshot recording it (FR-5.2 identity) and downgrades it to a boundary question if `measurement_strength` changed or the share crossed the 2% filing threshold between windows. Movement within a label is a trend, not an instability; no prior window is `unchecked`, not unstable (FR-2.4). - The downgrade is a finding against THIS repo, not the target's — the target did not change, the auditor described it two ways. Carries an `ia-stability` marker so a re-run comments rather than duplicating. - bin/ia_store.py factors the store and identity resolution out of audit-snapshot so both scripts hash signatures identically. A divergence there would make the gate match nothing and fail open. Documented as spec FR-4.5 and rubric §5b (shipped verbatim in references/). Wired into skills/offload-audit as step 4b, before filing-plan; steps 5-6 now consume its output rather than classified.json. S4 is retired in its old form — a human triage catching a bad candidate is not a repeatable test — and re-passed as: the gate catches the real release-policy-derivation case with no human in the loop. Asserted in tests/stability.test.sh case (f) against the verbatim two-window Phase 7 output. Closes kotkan/claude-plugin-inference-arbitrage#11 Suite: 178 assertions, exit 0.
This commit is contained in:
@@ -158,6 +158,22 @@ Note the inverse too: if the overrule case is *common*, the candidate is not a
|
||||
position-1 script — it is position 3, and the cut belongs earlier in the
|
||||
pipeline.
|
||||
|
||||
### 4b. The second hard gate — evidence stability
|
||||
|
||||
> **Evidence-strength label or filing-threshold side changes across measurement
|
||||
> windows → downgrade to a boundary question, never file (FR-4.5).**
|
||||
|
||||
`bin/stability-classify` enforces it, against the prior snapshot, and it runs
|
||||
after `boundary-classify` and before anything is filed. The reason is the mirror
|
||||
of §4's: the triple checks that the *reasoning* survives scrutiny, this checks
|
||||
that the *evidence* does. A `measured` 12% that becomes a `thin` 0% on a
|
||||
different date range was never a measurement of the target, and a human shown
|
||||
only one of the two has no way to discount it.
|
||||
|
||||
When it fires, the finding is about **this plugin**, not the audited one — file
|
||||
it on `kotkan/claude-plugin-inference-arbitrage`. Do not caveat the instability
|
||||
into an issue body on the target's repo; a hedge a reader can skip is not a gate.
|
||||
|
||||
### 5. Report
|
||||
|
||||
- **Rank by measured `offload_value`, and say that you are doing so.** A reader
|
||||
|
||||
Reference in New Issue
Block a user