oleks
dcec9cabfb
Phase 7 filed anxious/agent-wip/release-policy-derivation as `measured` at 11.94% of audited spend on a 30-day window. The same candidate over 7 days came back `thin` at 0.00%. Only the date range differed. A human reading either issue body alone cannot know the other exists, which is rubric.md §4a's failure mode reached through the plugin's own headline number. That was caught by a human running a second skeptical review — luck, not a control. It is now mechanical, in the same register as FR-4.2's overrule-case gate: - bin/stability-classify matches each `file` candidate against the most recent prior snapshot recording it (FR-5.2 identity) and downgrades it to a boundary question if `measurement_strength` changed or the share crossed the 2% filing threshold between windows. Movement within a label is a trend, not an instability; no prior window is `unchecked`, not unstable (FR-2.4). - The downgrade is a finding against THIS repo, not the target's — the target did not change, the auditor described it two ways. Carries an `ia-stability` marker so a re-run comments rather than duplicating. - bin/ia_store.py factors the store and identity resolution out of audit-snapshot so both scripts hash signatures identically. A divergence there would make the gate match nothing and fail open. Documented as spec FR-4.5 and rubric §5b (shipped verbatim in references/). Wired into skills/offload-audit as step 4b, before filing-plan; steps 5-6 now consume its output rather than classified.json. S4 is retired in its old form — a human triage catching a bad candidate is not a repeatable test — and re-passed as: the gate catches the real release-policy-derivation case with no human in the loop. Asserted in tests/stability.test.sh case (f) against the verbatim two-window Phase 7 output. Closes kotkan/claude-plugin-inference-arbitrage#11 Suite: 178 assertions, exit 0.
Calibration fixtures — tasks.md 4.4
The load-bearing gate. Two runs against known answers; if either fails, the rubric implementation is wrong and Phases 5 and 6 do not start.
Asserted by tests/classify.test.sh, which runs in CI.
Reproducing the inputs
bin/plugin-inventory ../token-budget --json > tb-inventory.json
bin/offload-scan --plugin token-budget --days 90 > tb-scan.json
bin/plugin-inventory ../worktree-discipline --json \
| tests/calibration/mask-worktree-audit.py > wd-inventory-masked.json
bin/offload-scan --plugin worktree-discipline --days 90 > wd-scan.json
The mask never touches the worktree-discipline checkout. It removes
bin/worktree-audit from the inventory JSON, recomputes script_loc and
script_coverage, and drops the one command block that invokes it — a skill
cannot prescribe a script that does not exist.
Mask effect: script_loc 1730 → 1458, script_coverage 0.353 → 0.297. The drop
is deliberately small — worktree-discipline keeps its hooks and its two other
scripts, so S3 cannot be passed by noticing a script-starved plugin. What is left
in sweep-worktrees is the raw git plumbing the skill prescribed:
git -C "$repo" fetch --all # ALWAYS first
git for-each-ref --format='%(upstream)' refs/heads/<branch> # must be refs/remotes/…
# give stranded commits an upstream, then re-audit
That second line is the recorded correction to the @{u} trap, carried in the
skill's own annotation — which is why the cut is rediscoverable from the JSON
alone, without appealing to outside knowledge of the incident.
The judgments files
*.judgments.json record the analyst's per-candidate rubric reasoning: the five
determinism-test outcomes, the falsifiability triple, the escalation path, and
the digest schema. Those are inference. Grading them into a confidence and a
file/don't-file verdict is a lookup table, so it lives in bin/boundary-classify
rather than in agent prose (FR-8.1 — the plugin obeying its own thesis).
Expected outcomes
| Gate | Target | Must produce |
|---|---|---|
| 1 (S2) | token-budget |
zero high-confidence candidates; conclusion "nothing to offload here" |
| 2 (S3) | worktree-discipline, masked |
sweep-worktrees worktree-safety classification at high, position llm-over-script-digest |
| 3 (S4) | anxious, two windows |
release-policy-derivation downgraded to a boundary question by the FR-4.5 stability gate, with no human in the loop |
The two-window anxious fixture — gate 3
Asserted by tests/stability.test.sh. These are real Phase 7 outputs, kept
verbatim, because the point of S4 is that the gate catches a candidate that
actually got filed, not a case constructed to be caught:
anxious-windows.snapshots.jsonl— both snapshots. Run2026-07-29T00-00-00Zis the 30-day window, in whichanxious/agent-wip/release-policy-derivationgradedmeasuredat 11.94% of audited spend. Run2026-07-29T01-00-00Zis the 7-day window, in which the same candidate gradedthinat 0.00%.anxious-7d.classified.json—boundary-classify's verdict on the 7-day window, which wasfile,highconfidence,thinevidence.anxious-7d.scan.json— the 7-day scan, for signature identity. Masked shapes and counts only (FR-3.5); no transcript content.
The test excludes the 7-day run from the store so it is the run being classified, exactly as at filing time, and asserts the gate downgrades it. The comparison is symmetric: classifying the 30-day window against the 7-day snapshot downgrades it too. Both windows cannot be right, which is the finding — recorded as kotkan/claude-plugin-inference-arbitrage#11.
The candidate matches by slug, not signature: the 7-day scan produced no recurring n-gram for that skill, so its signature falls back to the slug and the hashes differ. That is FR-5.2's resolution order doing its job — an identity that survives the evidence disappearing is exactly what a stability check needs.