spec-review pass: correct FR-9.2's enforcement claims, close the disclosure gap it invented

An independent spec-quality review (spec-review) found real drift and one real
gap, but also overstated FR-9.2 as an unenforced 'discipline' when three of its
four prohibitions are already mechanical (filing-plan only files verdict=='file',
requires --existing so a failed marker search can't silently duplicate, and
stability-classify rewrites rather than re-asks). Only 'what the analyst writes
into the judgments JSON' is genuinely unchecked — narrowed the claim to that.

The review also asserted an obligation — 'a run that leans on a recalled
conclusion must say so in the summary' — that didn't exist in any procedure.
Rather than strip a good idea, implemented it: skills/offload-audit/SKILL.md §7
and agents/offload-analyst.md §5 now require exactly that disclosure.

Same class of error, caught independently: FR-7.3's new stability/tracker-
separation clause describes what the analyst's spoken report can do, not what
the rendered wiki page does — snapshot_candidate carries no stability field, and
findings from stability-classify never enter the snapshot. Documented the gap
and its mechanical cause (widening the snapshot schema, not the renderer) rather
than leaving the spec's claim wrong.

FR-6.5, the FR-9.3 wing/room table, and the Q1/Q3 resolutions the review added
were checked against the shipped code and are accurate; left unchanged.

Suite: 159 assertions, exit 0.
This commit is contained in:
Oleks
2026-07-29 22:21:43 +03:00
parent 1777ee1568
commit f52bb79ea5
4 changed files with 96 additions and 28 deletions
+5
View File
@@ -221,6 +221,11 @@ into an issue body on the target's repo; a hedge a reader can skip is not a gate
- List drift notes under their own heading, with the documentation fix.
- **Emit only shapes and counts.** Never copy transcript content, file contents,
command arguments, or user prose into a report (FR-3.5).
- **Say when you leaned on memory.** If something recalled in §1b changed how
you judged a candidate, name it. The gates are pure functions of the
judgments JSON you hand them, so nothing mechanical can tell a reasoned
overrule case from a remembered one — saying so is the only control there is
(FR-9.2).
### 6. "Nothing to offload here" is a complete answer
+56 -5
View File
@@ -102,8 +102,11 @@ crux — see `rubric.md`), and keep watching over time.
- A **boundary classification** of each candidate against a published rubric.
- **On-demand runs** that each persist a snapshot, so history accumulates and
successive runs can be diffed.
- Two output paths per run: tracked issues on the **target** plugin's repo, and a
durable summary on **this** plugin's own wiki.
- Output paths per run: tracked issues on the **target** plugin's repo (FR-6), a
durable summary on **this** plugin's own wiki (FR-7), and durable context in
mempalace so a later run does not re-derive what an earlier one concluded
(FR-9). Findings about the *audit itself* go to this plugin's own tracker
instead of the target's (FR-6.5).
### 2.2 Out of scope for v1
@@ -317,8 +320,22 @@ data. Canonical store: wiki. Optional presentation: artifact.
**FR-7.3** The summary states, in this order: attribution coverage and window;
the composition headline (how much of the plugin's spend was mechanical); the
ranked candidate table with boundary classification and value; the diff against
the previous snapshot; the boundary questions (low-confidence candidates) posed
to the user; and the issues filed or updated, in full `owner/repo#num` form.
the previous snapshot; the boundary questions posed to the user — both the
low-confidence candidates and any downgraded by the FR-4.5 stability gate, the
latter marked as a defect in the auditor rather than a fact about the
target; and the issues filed or updated, in full `owner/repo#num` form,
separating those on the target's tracker (FR-6.1) from those on this plugin's
own (FR-6.5).
*Shipped as of v0.5.0, with one gap named rather than papered over:* the
analyst's spoken report (`skills/offload-audit` §7) makes both distinctions, but
the **rendered wiki page does not yet.** `audit-snapshot`'s summary renderer
lists all boundary questions uniformly and shows only each candidate's own
issue, because `snapshot_candidate` does not carry the `stability` field and
stability findings never enter the snapshot at all — `stability-classify` emits
them alongside it, which is why `memory-notes` takes `--stable` as a separate
input (FR-9.1). Closing this means widening the snapshot schema, not editing a
renderer; until then a self-filed finding reaches the page only via `--note`.
### FR-8 — Self-application
@@ -382,6 +399,35 @@ harder at the reasoning, not to short-circuit it — and a remembered "we filed
this last time" is not a filing decision at all, because the tracker, not
memory, is the FR-5.2 identity of record.
Note honestly what kind of requirement this is, clause by clause — the mix is
not uniform, and calling the whole of FR-9.2 unenforced would be as wrong as
calling it a gate.
Three of its four prohibitions **are** mechanical, and recall cannot reach them:
- *file something the gates rejected* — `bin/filing-plan` emits `action: file`
only for `verdict == "file"`, and skips everything else with the verdict named
in the reason;
- *skip the FR-6.3 marker search* — `filing-plan plan` makes `--existing`
required precisely so that a search which never ran cannot default to
"nothing exists";
- *re-grade a returned verdict* — the verdict is `boundary-classify`'s output,
and `stability-classify` rewrites rather than re-asks it.
What is left unenforced is the one clause that matters most and is the hardest
to watch: **what the analyst writes into the judgments JSON that those gates
then consume.** The gates are pure functions of their input, so recall cannot
corrupt a verdict directly — it can only corrupt the evidence handed to them, by
softening a determinism test or supplying an overrule case remembered rather
than reasoned. No script can see that, because the whole value of the read is
that it influences judgment; a read that could not move the judgments JSON would
be a read with no purpose.
An audit trail is therefore the only available control, and it is a real
obligation, not a hope: **a run that leans on a recalled conclusion says so in
its report**, so a human can see the lean and discount it. That is why FR-9.2 is
written as prohibitions rather than thresholds.
**FR-9.3 — Two placements, for two different readers.**
| | Wing | Room | Read | Write |
@@ -648,7 +694,10 @@ another plugin's `${CLAUDE_PLUGIN_ROOT}`. Options: (a) require `cc-tokens` on
(c) vendor a copy. Recommendation: (a) with (b) as fallback, and **never** (c) —
a vendored copy silently diverges on exactly the two traps that are hardest to
notice. Better still: contribute `cc-tokens attribute --by skill|plugin --json`
upstream to `token-budget`, so the money math has one home. *Decision needed.*
upstream to `token-budget`, so the money math has one home. **Resolved (TASKS
0.1): (a) with (b) as fallback**, never (c). `bin/offload-scan` tries `PATH`,
then the cache glob, then fails loudly with an install hint. The upstream
contribution remains optional (TASKS 8.3).
**Q2 — Attribution coverage may be thin.** `attributionSkill` appears only when a
Skill was formally invoked. Hook-driven work, agent work started without a Skill
@@ -663,6 +712,8 @@ which needs no repo access. But FR-6 files issues on the target's repo, which
does. For `oleks/*` plugins this is fine. For third-party or official-marketplace
plugins there is no writable tracker, and the audit degrades to summary-only
(FR-6.4). Confirm that degradation is acceptable rather than an error.
**Resolved (TASKS 0.2): yes**`filing: unavailable` with the reason recorded,
and the run summary still written.
**Q4 — False positives are the real failure mode.** Recommending a script where
judgment is needed produces something worse than the status quo: a fast,
+23 -23
View File
@@ -26,45 +26,45 @@ These block design, not just code. Answers go into `Methodology/Calibration.md`.
(`imagex:*`) plugins are excluded from the shared wiki entirely.
- [ ] **0.4** Resolve Q5: build the Goodhart counter-metric (wontfix/revert rate)
in v1, or note it as a manual check?
- [ ] **0.5** Confirm the plugin name `inference-arbitrage` (vs. keeping the
placeholder `token-offload-audit`). The directory name must equal the
manifest name, so this decision also fixes the checkout path.
- [ ] **0.6** Choose the Gitea **org**: `oleks` or `kotkan`. Not derivable from
the name. `kotkan` already holds `token-budget`, `worktree-discipline`, and
`anti-patterns` — the three closest relatives — which argues for `kotkan`.
- [x] **0.5** Confirm the plugin name `inference-arbitrage` (vs. keeping the
placeholder `token-offload-audit`). **Resolved: `inference-arbitrage`**;
directory renamed to match the manifest name.
- [x] **0.6** Choose the Gitea **org**: `oleks` or `kotkan`. **Resolved:
`kotkan`** — `kotkan/claude-plugin-inference-arbitrage`, alongside
`token-budget`, `worktree-discipline`, and `anti-patterns`.
## Phase 1 — Repo scaffold
- [ ] **1.1** Rename `~/projects/claude-plugins/token-offload-audit/` to match the
- [x] **1.1** Rename `~/projects/claude-plugins/token-offload-audit/` to match the
confirmed manifest name (workspace convention: directory == manifest
`name`), and build out the PLAN §2 layout: `plugin.json` v0.1.0, MIT
LICENSE, `.gitignore` with `.cache/`, `.claude/`, `*.log`. The four spec
files move with the directory and stay in-tree as the design record.
- [ ] **1.2** `git init -b main` **inside the plugin subdir** (the workspace root
- [x] **1.2** `git init -b main` **inside the plugin subdir** (the workspace root
is deliberately not a repo), create `<org>/claude-plugin-<name>` on Gitea
with no auto-init (via `cluster:gitea-agent`), push `main`.
- [ ] **1.2b** Verify `bin/` scripts carry the executable bit
- [x] **1.2b** Verify `bin/` scripts carry the executable bit
(`git ls-files -s``100755`) — three shipped scripts make this a live
concern. Re-check `git status` after every push: the global pre-push hook
auto-formats markdown/JSON and may append its own commit.
- [ ] **1.2c** Register in the local `oleks-local` marketplace
- [x] **1.2c** Register in the local `oleks-local` marketplace
(`~/projects/claude-plugins/.claude-plugin/marketplace.json`,
`"source": "./<name>"`) so Phases 27 can test via `@oleks-local` without
publishing anything publicly.
- [ ] **1.3** Initialize its wiki with `Methodology/Rubric.md` (rubric.md
verbatim, v1.0.0) and empty `Audits/` + `Data/` scaffolding.
- [ ] **1.4** `.woodpecker/test.yaml` running `tests/run-all.sh`. **CI verifies;
- [x] **1.4** `.woodpecker/test.yaml` running `tests/run-all.sh`. **CI verifies;
nothing heavy runs on emmett.**
## Phase 2 — Static pass
- [ ] **2.1** `bin/plugin-inventory`: target resolution (path or name, semver
- [x] **2.1** `bin/plugin-inventory`: target resolution (path or name, semver
pick, skip `.orphaned_at`), frontmatter parsing (~30 lines, no PyYAML).
- [ ] **2.2** Signal extraction: procedural density, verb ratio, command-block
- [x] **2.2** Signal extraction: procedural density, verb ratio, command-block
signatures, rule tables, script coverage, hook/prose drift.
- [ ] **2.3** `references/signals-catalog.md` with the verb lexicons and
- [x] **2.3** `references/signals-catalog.md` with the verb lexicons and
thresholds, loaded at runtime — tuning must be a reviewable diff.
- [ ] **2.4** `tests/inventory.test.sh` against a fixture plugin tree.
- [x] **2.4** `tests/inventory.test.sh` against a fixture plugin tree.
- [ ] **2.5** **Calibration run** against `token-budget`, `worktree-discipline`,
`anxious`, `memory`. Record the actual numbers in
`Methodology/Calibration.md` and tune thresholds until the ordering matches
@@ -72,19 +72,19 @@ These block design, not just code. Answers go into `Methodology/Calibration.md`.
## Phase 3 — Dynamic pass
- [ ] **3.1** `bin/offload-scan` skeleton: streaming JSONL reader, attribution
- [x] **3.1** `bin/offload-scan` skeleton: streaming JSONL reader, attribution
filter, `(message.id, requestId)` de-dup — delegating token/cost arithmetic
to `cc-tokens` per decision 0.1. **Do not reimplement the pricing table or
the streaming-snapshot max.**
- [ ] **3.2** Invocation reconstruction (contiguous attributed runs per session).
- [ ] **3.3** Turn classification: mechanical / judgment / retry, with the
- [x] **3.2** Invocation reconstruction (contiguous attributed runs per session).
- [x] **3.3** Turn classification: mechanical / judgment / retry, with the
mechanical tool set and the output-token threshold in the signals catalog.
- [ ] **3.4** Tool-signature normalization and masking (also satisfies FR-3.5).
- [ ] **3.5** N-gram mining: recurring sequences ≥3 long in ≥3 invocations.
- [ ] **3.6** Metrics: MTR, `offload_waste`, read amplification, retry density,
- [x] **3.4** Tool-signature normalization and masking (also satisfies FR-3.5).
- [x] **3.5** N-gram mining: recurring sequences ≥3 long in ≥3 invocations.
- [x] **3.6** Metrics: MTR, `offload_waste`, read amplification, retry density,
judgment density, fan-out multiplier, composite `offload_value`.
- [ ] **3.7** Attribution coverage computation (FR-3.4).
- [ ] **3.8** `tests/scan.test.sh` against **synthetic** fixture transcripts with
- [x] **3.7** Attribution coverage computation (FR-3.4).
- [x] **3.8** `tests/scan.test.sh` against **synthetic** fixture transcripts with
hand-computed expected metrics. Do not test against real transcripts —
they contain private content and they change.
- [ ] **3.9** Verify runtime and memory on the real history: one process,
+12
View File
@@ -316,6 +316,18 @@ previous snapshot → boundary questions, as questions → drift notes, with the
documentation fix → issues filed or updated, in full `owner/repo#num` form →
the wiki page path.
Two distinctions the rendered wiki page cannot yet make, and you must: mark any
boundary question that got there via the FR-4.5 stability gate as **a defect in
this auditor, not a fact about the target**, and list issues on this plugin's
own tracker (FR-6.5) separately from those on the target's (FR-6.1).
**Say when you leaned on memory.** If a step 2b result changed how you judged a
candidate — you weighed a determinism test differently, or reached for a
remembered overrule case — name it in the report. Nothing mechanical can catch
this (FR-9.2): the gates are pure functions of the judgments JSON you hand them,
so the only control on a recalled conclusion quietly becoming evidence is that
you say it out loud where a human can discount it.
Rank by measured `offload_value` and say that you are doing so. Name skills and
agents, **never UUIDs or session ids**. Emit **only shapes and counts** — no
transcript content, file contents, command arguments, or user prose ever enters