oleks
dcec9cabfb
Phase 7 filed anxious/agent-wip/release-policy-derivation as `measured` at 11.94% of audited spend on a 30-day window. The same candidate over 7 days came back `thin` at 0.00%. Only the date range differed. A human reading either issue body alone cannot know the other exists, which is rubric.md §4a's failure mode reached through the plugin's own headline number. That was caught by a human running a second skeptical review — luck, not a control. It is now mechanical, in the same register as FR-4.2's overrule-case gate: - bin/stability-classify matches each `file` candidate against the most recent prior snapshot recording it (FR-5.2 identity) and downgrades it to a boundary question if `measurement_strength` changed or the share crossed the 2% filing threshold between windows. Movement within a label is a trend, not an instability; no prior window is `unchecked`, not unstable (FR-2.4). - The downgrade is a finding against THIS repo, not the target's — the target did not change, the auditor described it two ways. Carries an `ia-stability` marker so a re-run comments rather than duplicating. - bin/ia_store.py factors the store and identity resolution out of audit-snapshot so both scripts hash signatures identically. A divergence there would make the gate match nothing and fail open. Documented as spec FR-4.5 and rubric §5b (shipped verbatim in references/). Wired into skills/offload-audit as step 4b, before filing-plan; steps 5-6 now consume its output rather than classified.json. S4 is retired in its old form — a human triage catching a bad candidate is not a repeatable test — and re-passed as: the gate catches the real release-policy-derivation case with no human in the loop. Asserted in tests/stability.test.sh case (f) against the verbatim two-window Phase 7 output. Closes kotkan/claude-plugin-inference-arbitrage#11 Suite: 178 assertions, exit 0.
81 lines
4.0 KiB
Markdown
81 lines
4.0 KiB
Markdown
# Calibration fixtures — tasks.md 4.4
|
|
|
|
The load-bearing gate. Two runs against known answers; if either fails, the
|
|
rubric implementation is wrong and Phases 5 and 6 do not start.
|
|
|
|
Asserted by `tests/classify.test.sh`, which runs in CI.
|
|
|
|
## Reproducing the inputs
|
|
|
|
```bash
|
|
bin/plugin-inventory ../token-budget --json > tb-inventory.json
|
|
bin/offload-scan --plugin token-budget --days 90 > tb-scan.json
|
|
|
|
bin/plugin-inventory ../worktree-discipline --json \
|
|
| tests/calibration/mask-worktree-audit.py > wd-inventory-masked.json
|
|
bin/offload-scan --plugin worktree-discipline --days 90 > wd-scan.json
|
|
```
|
|
|
|
The mask never touches the `worktree-discipline` checkout. It removes
|
|
`bin/worktree-audit` from the inventory JSON, recomputes `script_loc` and
|
|
`script_coverage`, and drops the one command block that invokes it — a skill
|
|
cannot prescribe a script that does not exist.
|
|
|
|
Mask effect: `script_loc` 1730 → 1458, `script_coverage` 0.353 → 0.297. The drop
|
|
is deliberately small — `worktree-discipline` keeps its hooks and its two other
|
|
scripts, so S3 cannot be passed by noticing a script-starved plugin. What is left
|
|
in `sweep-worktrees` is the raw git plumbing the skill prescribed:
|
|
|
|
```text
|
|
git -C "$repo" fetch --all # ALWAYS first
|
|
git for-each-ref --format='%(upstream)' refs/heads/<branch> # must be refs/remotes/…
|
|
# give stranded commits an upstream, then re-audit
|
|
```
|
|
|
|
That second line is the recorded correction to the `@{u}` trap, carried in the
|
|
skill's own annotation — which is why the cut is rediscoverable from the JSON
|
|
alone, without appealing to outside knowledge of the incident.
|
|
|
|
## The judgments files
|
|
|
|
`*.judgments.json` record the analyst's per-candidate rubric reasoning: the five
|
|
determinism-test outcomes, the falsifiability triple, the escalation path, and
|
|
the digest schema. Those are inference. Grading them into a confidence and a
|
|
file/don't-file verdict is a lookup table, so it lives in `bin/boundary-classify`
|
|
rather than in agent prose (FR-8.1 — the plugin obeying its own thesis).
|
|
|
|
## Expected outcomes
|
|
|
|
| Gate | Target | Must produce |
|
|
| --- | --- | --- |
|
|
| 1 (S2) | `token-budget` | zero `high`-confidence candidates; conclusion "nothing to offload here" |
|
|
| 2 (S3) | `worktree-discipline`, masked | `sweep-worktrees` worktree-safety classification at `high`, position `llm-over-script-digest` |
|
|
| 3 (S4) | `anxious`, two windows | `release-policy-derivation` downgraded to a boundary question by the FR-4.5 stability gate, with no human in the loop |
|
|
|
|
## The two-window `anxious` fixture — gate 3
|
|
|
|
Asserted by `tests/stability.test.sh`. These are **real** Phase 7 outputs, kept
|
|
verbatim, because the point of S4 is that the gate catches a candidate that
|
|
actually got filed, not a case constructed to be caught:
|
|
|
|
- `anxious-windows.snapshots.jsonl` — both snapshots. Run
|
|
`2026-07-29T00-00-00Z` is the 30-day window, in which
|
|
`anxious/agent-wip/release-policy-derivation` graded `measured` at 11.94% of
|
|
audited spend. Run `2026-07-29T01-00-00Z` is the 7-day window, in which the
|
|
same candidate graded `thin` at 0.00%.
|
|
- `anxious-7d.classified.json` — `boundary-classify`'s verdict on the 7-day
|
|
window, which was `file`, `high` confidence, `thin` evidence.
|
|
- `anxious-7d.scan.json` — the 7-day scan, for signature identity. Masked
|
|
shapes and counts only (FR-3.5); no transcript content.
|
|
|
|
The test excludes the 7-day run from the store so it is the run being
|
|
classified, exactly as at filing time, and asserts the gate downgrades it. The
|
|
comparison is symmetric: classifying the 30-day window against the 7-day
|
|
snapshot downgrades it too. Both windows cannot be right, which is the finding —
|
|
recorded as kotkan/claude-plugin-inference-arbitrage#11.
|
|
|
|
The candidate matches by **slug**, not signature: the 7-day scan produced no
|
|
recurring n-gram for that skill, so its signature falls back to the slug and the
|
|
hashes differ. That is FR-5.2's resolution order doing its job — an identity
|
|
that survives the evidence disappearing is exactly what a stability check needs.
|