Oleks
88fc58b2fd
ci/woodpecker/push/test Pipeline was successful
The stability gate treated every evidence flip as symmetric, and that produced a false downgrade on the candidate it was written for (kotkan/claude-plugin-inference-arbitrage#15). Re-measuring anxious/agent-wip/release-policy-derivation over four real windows with the same script gave 0 -> 2 -> 12 -> 15 invocations at 7/14/27/60 days: monotonic accumulation of a step that runs about once every two days, not a label flipping about. The 7-day window had not caught an unstable candidate, it had failed to observe a real one. So a thin/unmeasured reading now only overturns a `measured` one when its window is at least 90% as wide (observed since->until span). Below that the comparison takes a third verdict, `insufficient-window`: not filed, not downgraded to a boundary question, no FR-6.5 self-report, and carried forward with the wider snapshot left as the standing comparison point. The parity is relative rather than an absolute day count because the adequate width is a property of the candidate's invocation rate, which the auditor does not know in advance. Unchanged, and tested: the reverse direction (thin prior -> measured now) still downgrades, a flip between comparably wide windows still downgrades, and a threshold crossing with `measured` on both sides is never excused by narrowness. An unreadable window is not an exemption either — the flip stands and window_parity records that the check could not run. - bin/stability-classify: window_days/window_parity/deferrable, the insufficient-window verdict, stability_deferrals, window_parity on findings too, insufficient_window + window_parity_ratio in the gate block - bin/audit-snapshot: record deferrals in the snapshot notes so the row is visible on the wiki page instead of silently dropped - design/spec.md: FR-4.5.1 as a testable requirement; S4 restated - design/rubric.md: the third boundary in §5b - skills/offload-audit/SKILL.md: do not file and do not self-report a deferral - tests: real 60-day anxious fixture; cases (f)/(f2) on real data, (j)-(n) synthetic — 7 test files pass Verified against the live store (~/.cache/inference-arbitrage/wiki): the real 7.0d-vs-26.49d comparison now reports insufficient-window with zero findings, while the real 33.4d-vs-26.49d one reports stable at 13.87% and files normally.
102 lines
5.4 KiB
Markdown
102 lines
5.4 KiB
Markdown
# Calibration fixtures — tasks.md 4.4
|
|
|
|
The load-bearing gate. Two runs against known answers; if either fails, the
|
|
rubric implementation is wrong and Phases 5 and 6 do not start.
|
|
|
|
Asserted by `tests/classify.test.sh`, which runs in CI.
|
|
|
|
## Reproducing the inputs
|
|
|
|
```bash
|
|
bin/plugin-inventory ../token-budget --json > tb-inventory.json
|
|
bin/offload-scan --plugin token-budget --days 90 > tb-scan.json
|
|
|
|
bin/plugin-inventory ../worktree-discipline --json \
|
|
| tests/calibration/mask-worktree-audit.py > wd-inventory-masked.json
|
|
bin/offload-scan --plugin worktree-discipline --days 90 > wd-scan.json
|
|
```
|
|
|
|
The mask never touches the `worktree-discipline` checkout. It removes
|
|
`bin/worktree-audit` from the inventory JSON, recomputes `script_loc` and
|
|
`script_coverage`, and drops the one command block that invokes it — a skill
|
|
cannot prescribe a script that does not exist.
|
|
|
|
Mask effect: `script_loc` 1730 → 1458, `script_coverage` 0.353 → 0.297. The drop
|
|
is deliberately small — `worktree-discipline` keeps its hooks and its two other
|
|
scripts, so S3 cannot be passed by noticing a script-starved plugin. What is left
|
|
in `sweep-worktrees` is the raw git plumbing the skill prescribed:
|
|
|
|
```text
|
|
git -C "$repo" fetch --all # ALWAYS first
|
|
git for-each-ref --format='%(upstream)' refs/heads/<branch> # must be refs/remotes/…
|
|
# give stranded commits an upstream, then re-audit
|
|
```
|
|
|
|
That second line is the recorded correction to the `@{u}` trap, carried in the
|
|
skill's own annotation — which is why the cut is rediscoverable from the JSON
|
|
alone, without appealing to outside knowledge of the incident.
|
|
|
|
## The judgments files
|
|
|
|
`*.judgments.json` record the analyst's per-candidate rubric reasoning: the five
|
|
determinism-test outcomes, the falsifiability triple, the escalation path, and
|
|
the digest schema. Those are inference. Grading them into a confidence and a
|
|
file/don't-file verdict is a lookup table, so it lives in `bin/boundary-classify`
|
|
rather than in agent prose (FR-8.1 — the plugin obeying its own thesis).
|
|
|
|
## Expected outcomes
|
|
|
|
| Gate | Target | Must produce |
|
|
| --- | --- | --- |
|
|
| 1 (S2) | `token-budget` | zero `high`-confidence candidates; conclusion "nothing to offload here" |
|
|
| 2 (S3) | `worktree-discipline`, masked | `sweep-worktrees` worktree-safety classification at `high`, position `llm-over-script-digest` |
|
|
| 3 (S4) | `anxious`, two windows | `release-policy-derivation` refused by the FR-4.5 stability gate — as `insufficient-window` (FR-4.5.1), the 7-day window being too narrow to contradict the 26-day `measured` one — with no human in the loop |
|
|
|
|
## The two-window `anxious` fixture — gate 3
|
|
|
|
Asserted by `tests/stability.test.sh`. These are **real** Phase 7 outputs, kept
|
|
verbatim, because the point of S4 is that the gate catches a candidate that
|
|
actually got filed, not a case constructed to be caught:
|
|
|
|
- `anxious-windows.snapshots.jsonl` — both snapshots. Run
|
|
`2026-07-29T00-00-00Z` is the 30-day window, in which
|
|
`anxious/agent-wip/release-policy-derivation` graded `measured` at 11.94% of
|
|
audited spend. Run `2026-07-29T01-00-00Z` is the 7-day window, in which the
|
|
same candidate graded `thin` at 0.00%.
|
|
- `anxious-7d.classified.json` — `boundary-classify`'s verdict on the 7-day
|
|
window, which was `file`, `high` confidence, `thin` evidence.
|
|
- `anxious-7d.scan.json` — the 7-day scan, for signature identity. Masked
|
|
shapes and counts only (FR-3.5); no transcript content.
|
|
|
|
- `anxious-60d.scan.json` — the real 60-day scan
|
|
(`bin/offload-scan --plugin anxious --days 60`, 2026-06-26 → 2026-07-30, 176
|
|
sessions), added while resolving
|
|
kotkan/claude-plugin-inference-arbitrage#15. Same masking rules.
|
|
- `anxious-60d.classified.json` — the same candidate's unchanged rubric grade
|
|
with the 60-day window's real measurement: 15 invocations, `offload_value`
|
|
4,953,520 of 35,722,973 weighted tokens = 13.87% of audited spend, `measured`.
|
|
It carries only that one candidate, so no other row can be read as a 60-day
|
|
number it was not measured over.
|
|
|
|
The test excludes the 7-day run from the store so it is the run being
|
|
classified, exactly as at filing time, and asserts the gate **refuses** it. What
|
|
it does *not* assert any more is that the refusal is a downgrade. The comparison
|
|
is not symmetric, and treating it as symmetric was the bug behind
|
|
kotkan/claude-plugin-inference-arbitrage#15: measuring two further real windows
|
|
gave 0 → 2 → 12 → 15 invocations at 7, 14, 27 and 60 days, i.e. a candidate that
|
|
runs about once every two days and that a 7-day window simply never sees. So the
|
|
7-day-against-26-day pairing is now `insufficient-window` (FR-4.5.1) with no
|
|
self-report, while the 60-day-against-26-day pairing — two adequately wide
|
|
windows, `measured` both times, same side of the 2% threshold — is `stable` and
|
|
files normally. Both directions are asserted, so a regression in either shows up
|
|
as a test failure rather than as a wrong issue on someone's tracker.
|
|
|
|
The original two-window finding, and the fact that both windows cannot be right
|
|
about a candidate they sample equally, is recorded as
|
|
kotkan/claude-plugin-inference-arbitrage#11.
|
|
|
|
The candidate matches by **slug**, not signature: the 7-day scan produced no
|
|
recurring n-gram for that skill, so its signature falls back to the slug and the
|
|
hashes differ. That is FR-5.2's resolution order doing its job — an identity
|
|
that survives the evidence disappearing is exactly what a stability check needs.
|