Phase 4 — Rubric and classification #4

New Issue

2026-07-29T16:37:49+03:00

oleks commented

2026-07-29 16:37:49 +03:00

Per design/tasks.md Phase 4. Depends on Phase 3 (closed).

4.1 references/boundary-rubric.md = design/rubric.md verbatim
4.2 skills/boundary-rubric/SKILL.md — usable standalone on a single step
4.3 agents/offload-analyst.md — hard gate: no overrule case → boundary question, never filed
4.4 Calibration against known answers — load-bearing. token-budget → zero high-confidence candidates (PASS). worktree-discipline with bin/worktree-audit masked → known cut flagged high (PASS). One real iteration recorded (measurement was wrongly gating correctness, not just value — fixed).
4.5 Calibration outcome recorded in Methodology/Calibration.md

Implemented in 38e1adb, 8da16ad. Independently verified: both calibration gate fixtures inspected directly (T1-T5 judgments are rigorously argued, not rubber-stamped — e.g. the worktree overrule case is a real edge case, a clean-but-live sibling-session checkout). bash tests/run-all.sh: 67 assertions, exit 0. See comment below for full detail.

Per design/tasks.md Phase 4. Depends on Phase 3 (closed). - [x] 4.1 `references/boundary-rubric.md` = `design/rubric.md` verbatim - [x] 4.2 `skills/boundary-rubric/SKILL.md` — usable standalone on a single step - [x] 4.3 `agents/offload-analyst.md` — hard gate: no overrule case → boundary question, never filed - [x] 4.4 **Calibration against known answers — load-bearing.** token-budget → zero `high`-confidence candidates (PASS). worktree-discipline with `bin/worktree-audit` masked → known cut flagged `high` (PASS). One real iteration recorded (measurement was wrongly gating correctness, not just value — fixed). - [x] 4.5 Calibration outcome recorded in [Methodology/Calibration.md](https://git.oleks.space/kotkan/claude-plugin-inference-arbitrage/wiki/Methodology%2FCalibration) Implemented in `38e1adb`, `8da16ad`. Independently verified: both calibration gate fixtures inspected directly (T1-T5 judgments are rigorously argued, not rubber-stamped — e.g. the worktree overrule case is a real edge case, a clean-but-live sibling-session checkout). `bash tests/run-all.sh`: 67 assertions, exit 0. See comment below for full detail.

oleks added this to the v0.1.0 milestone 2026-07-29 16:37:49 +03:00

oleks added a new dependency 2026-07-29 16:38:27 +03:00

#3 Phase 3 — Dynamic pass

oleks added a new dependency 2026-07-29 16:38:29 +03:00

#5 Phase 5 — Snapshots and trend

oleks added a new dependency 2026-07-29 16:38:29 +03:00

#6 Phase 6 — Output paths

oleks commented

2026-07-29 17:28:23 +03:00

Phase 4 delivered — rubric agent + calibration gate

Commits 38e1adb, 8da16ad on main. Plugin version 0.1.0 → 0.2.0.

tasks.md checklist

4.1 references/boundary-rubric.md = design/rubric.md verbatim (diff -q clean; both files were given identical markdownlint fixes so they stay byte-identical).
4.2 skills/boundary-rubric/SKILL.md — standalone on a single step, before any code exists. Carries the five tests, four positions, four positioning questions, both failure modes with the asymmetry, the falsifiability triple, and the §6 confidence table as an operational procedure.
4.3 agents/offload-analyst.md — model: sonnet, tools: Bash, Read, Grep, Glob, Skill, Agent. Read-only w.r.t. the target; leads with coverage; names skills not UUIDs; ranks by measured offload_value; "nothing to offload here" is a complete answer; hard gate stated.
4.4 Calibration gate — both checks pass. Evidence below.
4.5 Calibration write-up produced and handed to the team lead for the Methodology/Calibration.md append (not written by me, per instruction).

New: `bin/boundary-classify`

A deviation from plan.md §2's three-script layout, and the reason matters. The determinism-test outcomes and the falsifiability triple are irreducibly inference (T2 and T4 are semantic). But grading them into a confidence and a file/don't-file verdict is a pure lookup table — rubric §6 plus the §5.2.7 threshold. Leaving that in agent prose would have the model re-derive a fixed table every invocation, which is exactly the anti-pattern this plugin reports on (FR-8.1). So it is a script, and the FR-4.2 hard gate is now mechanically enforced rather than merely asserted in prose.

Calibration evidence

Gate 1 — token-budget → zero high-confidence. PASS.

candidates: 2 | high_confidence: 0 | to_file: 0
conclusion: "nothing to offload here"

token-budget/token-audit/opening-command-sequence   -> pure-inference / low / no-offload
token-budget/token-audit/finding-to-remediation-table -> pure-inference / low / no-offload

Two candidates were surfaced and both declined on the rubric, rather than none being looked at. The opening-sequence candidate fails T4: which diagnostic to open with depends on what the user asked, and that is not in the inputs. No overrule case is producible — and the reason is rubric §5's inverse paragraph: a human would overrule it often, so it is not a position-1 script. The dispatch-table candidate fails T1/T2/T4.

Static signals that could have produced a false positive and did not: verb ratios of 0.667 and 0.714 are both above the 0.6 smell threshold. They do not fire because the smell is a conjunction — high verb ratio with no bin/ script behind it — and every command those skills prescribe is an invocation of the 606-LOC cc-tokens. The agent body states this as a conjunction explicitly.

Gate 2 — worktree-discipline with bin/worktree-audit masked → known cut rediscovered at high. PASS.

candidates: 3 | high_confidence: 1 | to_file: 1 | drift_notes: 1

worktree-discipline/sweep-worktrees/worktree-safety-classification
    -> llm-over-script-digest / HIGH / file    (T1-T5 all pass, triple complete)
worktree-discipline/dev-server-sync/read-amplification
    -> pure-inference / low / no-offload
worktree-discipline/own-worktree-creation/env-contract-restated
    -> hook-prose-drift / low / drift-note

The mask (tests/calibration/mask-worktree-audit.py, reproducible, never touches the target checkout) drops script_loc 1730 → 407 and script_coverage 0.353 → 0.083, and removes the one command block that invoked the masked script. What is left in sweep-worktrees is the raw plumbing the skill prescribed, including the recorded correction:

git for-each-ref --format='%(upstream)' refs/heads/<branch>   # must be refs/remotes/…

That annotation is why the cut is rediscoverable from the inventory JSON alone, without appealing to outside knowledge of the @{u} incident — the edge case in the triple is exactly that trap.

Two negative controls in the same run are as important as the positive one:

dev-server-sync is the loudest cost signal in the target — 76% of audited spend, read_amplification 392.5 — and the rubric declines it. The amplification is from reading the user's project files, not a procedure the plugin owns: T1/T2/T4 all fail, MTR is 0.057, no recurring n-gram. Scripting it would be premature crystallization.
Hook/prose drift is carried as its own category with a drift-note verdict and a documentation fix ("point at hooks/policy.sh, stop restating the env-var contract"), never competing on token value in the offload table.

Iteration required

One real iteration. The first passing run graded the masked worktree candidate high but returned boundary-question instead of file, because I had made thin dynamic evidence (1 invocation, 43 turns) block filing outright. That was wrong: measurement gates the value claim, not the correctness claim, and blocking on it contradicts FR-2.4, under which a plugin with zero transcript history must stay auditable on its definition alone. Changed so that measured (≥3 invocations, ≥30 attributed turns) applies the §5.2.7 2%-of-spend threshold as written, while thin/unmeasured files on the static case with the value claim explicitly marked unproven. Both gates still pass; the known-correct cut now correctly reaches file.

Coverage-denominator decision (the Phase 3 open finding)

Decided: keep the session-wide denominator, add no second one, and relabel it. A tighter "turns within invocation windows" denominator is near-tautological — an invocation window is a contiguous run of attributed turns, so that ratio is ~1.0 by construction. It would read as a reassuring number while measuring nothing.

The session-wide ratio does not measure completeness at all; it measures dilution (how much of those sessions was other work), which is the real content of the 0.021 finding against anxious. So the agent reports it as dilution and reports completeness instead via absolute counts plus named structural blind spots — hook-driven work carries no attribution, so the agent must name the target's hooks and say their cost is invisible. Rationale is recorded as a comment in the agent body, not left as a TODO.

Tests

tests/classify.test.sh — synthetic hand-computable cases for every row of the confidence table, the FR-4.2 hard gate, the 2% threshold, position-2-without-staleness-plan, plus both calibration gates asserted end to end. The mask check skips cleanly when no sibling worktree-discipline checkout exists, so CI is unaffected.

bash tests/run-all.sh → exit 0, 67 assertions, nothing from Phase 2/3 broken. shellcheck clean; all pre-push gates green (BFM ✅ RUF ✅ SHF ✅ MDL ✅ WPC ✅ BIO ✅ RUL ✅ SHC ✅ CPV ✅).

Not done, deliberately

Filing is Phase 6. The agent body instructs it to report and file nothing, including candidates boundary-classify returns with "verdict": "file", until that path exists.

Leaving this open for your review.

## Phase 4 delivered — rubric agent + calibration gate Commits `38e1adb`, `8da16ad` on `main`. Plugin version `0.1.0` → `0.2.0`. ### tasks.md checklist - [x] **4.1** `references/boundary-rubric.md` = `design/rubric.md` **verbatim** (`diff -q` clean; both files were given identical markdownlint fixes so they stay byte-identical). - [x] **4.2** `skills/boundary-rubric/SKILL.md` — standalone on a single step, before any code exists. Carries the five tests, four positions, four positioning questions, both failure modes with the asymmetry, the falsifiability triple, and the §6 confidence table as an operational procedure. - [x] **4.3** `agents/offload-analyst.md` — `model: sonnet`, `tools: Bash, Read, Grep, Glob, Skill, Agent`. Read-only w.r.t. the target; leads with coverage; names skills not UUIDs; ranks by measured `offload_value`; "nothing to offload here" is a complete answer; hard gate stated. - [x] **4.4** **Calibration gate — both checks pass.** Evidence below. - [ ] **4.5** Calibration write-up produced and handed to the team lead for the `Methodology/Calibration.md` append (not written by me, per instruction). ### New: `bin/boundary-classify` A deviation from plan.md §2's three-script layout, and the reason matters. The determinism-test outcomes and the falsifiability triple are irreducibly inference (T2 and T4 are semantic). But **grading them into a confidence and a file/don't-file verdict is a pure lookup table** — rubric §6 plus the §5.2.7 threshold. Leaving that in agent prose would have the model re-derive a fixed table every invocation, which is exactly the anti-pattern this plugin reports on (FR-8.1). So it is a script, and the FR-4.2 hard gate is now **mechanically enforced** rather than merely asserted in prose. ### Calibration evidence **Gate 1 — `token-budget` → zero `high`-confidence.** PASS. ``` candidates: 2 | high_confidence: 0 | to_file: 0 conclusion: "nothing to offload here" token-budget/token-audit/opening-command-sequence -> pure-inference / low / no-offload token-budget/token-audit/finding-to-remediation-table -> pure-inference / low / no-offload ``` Two candidates were surfaced and both declined on the rubric, rather than none being looked at. The opening-sequence candidate fails **T4**: which diagnostic to open with depends on what the user asked, and that is not in the inputs. No overrule case is producible — and the reason is rubric §5's inverse paragraph: a human would overrule it *often*, so it is not a position-1 script. The dispatch-table candidate fails T1/T2/T4. Static signals that could have produced a false positive and did not: verb ratios of **0.667** and **0.714** are both above the 0.6 smell threshold. They do not fire because the smell is a **conjunction** — high verb ratio *with no `bin/` script behind it* — and every command those skills prescribe is an invocation of the 606-LOC `cc-tokens`. The agent body states this as a conjunction explicitly. **Gate 2 — `worktree-discipline` with `bin/worktree-audit` masked → known cut rediscovered at `high`.** PASS. ``` candidates: 3 | high_confidence: 1 | to_file: 1 | drift_notes: 1 worktree-discipline/sweep-worktrees/worktree-safety-classification -> llm-over-script-digest / HIGH / file (T1-T5 all pass, triple complete) worktree-discipline/dev-server-sync/read-amplification -> pure-inference / low / no-offload worktree-discipline/own-worktree-creation/env-contract-restated -> hook-prose-drift / low / drift-note ``` The mask (`tests/calibration/mask-worktree-audit.py`, reproducible, never touches the target checkout) drops `script_loc` 1730 → 407 and `script_coverage` 0.353 → 0.083, and removes the one command block that invoked the masked script. What is left in `sweep-worktrees` is the raw plumbing the skill prescribed, including the recorded correction: ``` git for-each-ref --format='%(upstream)' refs/heads/<branch> # must be refs/remotes/… ``` That annotation is why the cut is rediscoverable **from the inventory JSON alone**, without appealing to outside knowledge of the `@{u}` incident — the edge case in the triple is exactly that trap. Two negative controls in the same run are as important as the positive one: - `dev-server-sync` is the **loudest cost signal in the target** — 76% of audited spend, `read_amplification` 392.5 — and the rubric **declines it**. The amplification is from reading the user's project files, not a procedure the plugin owns: T1/T2/T4 all fail, MTR is 0.057, no recurring n-gram. Scripting it would be premature crystallization. - Hook/prose drift is carried as **its own category** with a `drift-note` verdict and a documentation fix ("point at `hooks/policy.sh`, stop restating the env-var contract"), never competing on token value in the offload table. ### Iteration required One real iteration. The first passing run graded the masked worktree candidate `high` but returned `boundary-question` instead of `file`, because I had made thin dynamic evidence (1 invocation, 43 turns) block filing outright. That was wrong: **measurement gates the value claim, not the correctness claim**, and blocking on it contradicts FR-2.4, under which a plugin with zero transcript history must stay auditable on its definition alone. Changed so that `measured` (≥3 invocations, ≥30 attributed turns) applies the §5.2.7 2%-of-spend threshold as written, while `thin`/`unmeasured` files on the static case with the value claim explicitly marked unproven. Both gates still pass; the known-correct cut now correctly reaches `file`. ### Coverage-denominator decision (the Phase 3 open finding) **Decided: keep the session-wide denominator, add no second one, and relabel it.** A tighter "turns within invocation windows" denominator is near-tautological — an invocation window *is* a contiguous run of attributed turns, so that ratio is ~1.0 by construction. It would read as a reassuring number while measuring nothing. The session-wide ratio does not measure completeness at all; it measures **dilution** (how much of those sessions was other work), which is the real content of the 0.021 finding against `anxious`. So the agent reports it as dilution and reports completeness instead via absolute counts plus **named structural blind spots** — hook-driven work carries no attribution, so the agent must name the target's hooks and say their cost is invisible. Rationale is recorded as a comment in the agent body, not left as a TODO. ### Tests `tests/classify.test.sh` — synthetic hand-computable cases for every row of the confidence table, the FR-4.2 hard gate, the 2% threshold, position-2-without-staleness-plan, plus both calibration gates asserted end to end. The mask check skips cleanly when no sibling `worktree-discipline` checkout exists, so CI is unaffected. `bash tests/run-all.sh` → **exit 0, 67 assertions**, nothing from Phase 2/3 broken. shellcheck clean; all pre-push gates green (`BFM ✅ RUF ✅ SHF ✅ MDL ✅ WPC ✅ BIO ✅ RUL ✅ SHC ✅ CPV ✅`). ### Not done, deliberately Filing is Phase 6. The agent body instructs it to report and file **nothing**, including candidates `boundary-classify` returns with `"verdict": "file"`, until that path exists. Leaving this open for your review.

oleks closed this issue

2026-07-29 17:37:07 +03:00

issuer-agent referenced this issue

2026-07-30 05:51:44 +03:00

CI environment is missing the cc-tokens dependency, so tests/scan.test.sh always fails in Woodpecker #20

issuer-agent referenced this issue

2026-07-30 05:51:49 +03:00

kotkan/claude-plugin-inference-arbitrage has no required-status-check branch protection, so merge_when_checks_succeed merges regardless of CI outcome #21

Sign in to join this conversation.