claude-plugin-inference-arbitrage/tests/calibration/README.md

Calibration fixtures — tasks.md 4.4

The load-bearing gate. Two runs against known answers; if either fails, the rubric implementation is wrong and Phases 5 and 6 do not start.

Asserted by tests/classify.test.sh, which runs in CI.

Reproducing the inputs

bin/plugin-inventory ../token-budget --json          > tb-inventory.json
bin/offload-scan --plugin token-budget --days 90     > tb-scan.json

bin/plugin-inventory ../worktree-discipline --json \
  | tests/calibration/mask-worktree-audit.py         > wd-inventory-masked.json
bin/offload-scan --plugin worktree-discipline --days 90 > wd-scan.json

The mask never touches the worktree-discipline checkout. It removes bin/worktree-audit from the inventory JSON, recomputes script_loc and script_coverage, and drops the one command block that invokes it — a skill cannot prescribe a script that does not exist.

Mask effect: script_loc 1730 → 1458, script_coverage 0.353 → 0.297. The drop is deliberately small — worktree-discipline keeps its hooks and its two other scripts, so S3 cannot be passed by noticing a script-starved plugin. What is left in sweep-worktrees is the raw git plumbing the skill prescribed:

git -C "$repo" fetch --all                                          # ALWAYS first
git for-each-ref --format='%(upstream)' refs/heads/<branch>         # must be refs/remotes/…
# give stranded commits an upstream, then re-audit

That second line is the recorded correction to the @{u} trap, carried in the skill's own annotation — which is why the cut is rediscoverable from the JSON alone, without appealing to outside knowledge of the incident.

The judgments files

*.judgments.json record the analyst's per-candidate rubric reasoning: the five determinism-test outcomes, the falsifiability triple, the escalation path, and the digest schema. Those are inference. Grading them into a confidence and a file/don't-file verdict is a lookup table, so it lives in bin/boundary-classify rather than in agent prose (FR-8.1 — the plugin obeying its own thesis).

Expected outcomes

Gate	Target	Must produce
1 (S2)	token-budget	zero high-confidence candidates; conclusion "nothing to offload here"
2 (S3)	worktree-discipline, masked	sweep-worktrees worktree-safety classification at high, position llm-over-script-digest
3 (S4)	anxious, two windows	release-policy-derivation refused by the FR-4.5 stability gate — as insufficient-window (FR-4.5.1), the 7-day window being too narrow to contradict the 26-day measured one — with no human in the loop

The two-window anxious fixture — gate 3

Asserted by tests/stability.test.sh. These are real Phase 7 outputs, kept verbatim, because the point of S4 is that the gate catches a candidate that actually got filed, not a case constructed to be caught:

• anxious-windows.snapshots.jsonl — both snapshots. Run 2026-07-29T00-00-00Z is the 30-day window, in which anxious/agent-wip/release-policy-derivation graded measured at 11.94% of audited spend. Run 2026-07-29T01-00-00Z is the 7-day window, in which the same candidate graded thin at 0.00%.

• anxious-7d.classified.json — boundary-classify's verdict on the 7-day window, which was file, high confidence, thin evidence.

• anxious-7d.scan.json — the 7-day scan, for signature identity. Masked shapes and counts only (FR-3.5); no transcript content.

• anxious-60d.scan.json — the real 60-day scan (bin/offload-scan --plugin anxious --days 60, 2026-06-26 → 2026-07-30, 176 sessions), added while resolving kotkan/claude-plugin-inference-arbitrage#15. Same masking rules.

• anxious-60d.classified.json — the same candidate's unchanged rubric grade with the 60-day window's real measurement: 15 invocations, offload_value 4,953,520 of 35,722,973 weighted tokens = 13.87% of audited spend, measured. It carries only that one candidate, so no other row can be read as a 60-day number it was not measured over.

The test excludes the 7-day run from the store so it is the run being classified, exactly as at filing time, and asserts the gate refuses it. What it does not assert any more is that the refusal is a downgrade. The comparison is not symmetric, and treating it as symmetric was the bug behind kotkan/claude-plugin-inference-arbitrage#15: measuring two further real windows gave 0 → 2 → 12 → 15 invocations at 7, 14, 27 and 60 days, i.e. a candidate that runs about once every two days and that a 7-day window simply never sees. So the 7-day-against-26-day pairing is now insufficient-window (FR-4.5.1) with no self-report, while the 60-day-against-26-day pairing — two adequately wide windows, measured both times, same side of the 2% threshold — is stable and files normally. Both directions are asserted, so a regression in either shows up as a test failure rather than as a wrong issue on someone's tracker.

The original two-window finding, and the fact that both windows cannot be right about a candidate they sample equally, is recorded as kotkan/claude-plugin-inference-arbitrage#11.

The candidate matches by slug, not signature: the 7-day scan produced no recurring n-gram for that skill, so its signature falls back to the slug and the hashes differ. That is FR-5.2's resolution order doing its job — an identity that survives the evidence disappearing is exactly what a stability check needs.