spec-review pass: correct FR-9.2's enforcement claims, close the disclosure gap it invented
An independent spec-quality review (spec-review) found real drift and one real gap, but also overstated FR-9.2 as an unenforced 'discipline' when three of its four prohibitions are already mechanical (filing-plan only files verdict=='file', requires --existing so a failed marker search can't silently duplicate, and stability-classify rewrites rather than re-asks). Only 'what the analyst writes into the judgments JSON' is genuinely unchecked — narrowed the claim to that. The review also asserted an obligation — 'a run that leans on a recalled conclusion must say so in the summary' — that didn't exist in any procedure. Rather than strip a good idea, implemented it: skills/offload-audit/SKILL.md §7 and agents/offload-analyst.md §5 now require exactly that disclosure. Same class of error, caught independently: FR-7.3's new stability/tracker- separation clause describes what the analyst's spoken report can do, not what the rendered wiki page does — snapshot_candidate carries no stability field, and findings from stability-classify never enter the snapshot. Documented the gap and its mechanical cause (widening the snapshot schema, not the renderer) rather than leaving the spec's claim wrong. FR-6.5, the FR-9.3 wing/room table, and the Q1/Q3 resolutions the review added were checked against the shipped code and are accurate; left unchanged. Suite: 159 assertions, exit 0.
This commit is contained in:
@@ -221,6 +221,11 @@ into an issue body on the target's repo; a hedge a reader can skip is not a gate
|
||||
- List drift notes under their own heading, with the documentation fix.
|
||||
- **Emit only shapes and counts.** Never copy transcript content, file contents,
|
||||
command arguments, or user prose into a report (FR-3.5).
|
||||
- **Say when you leaned on memory.** If something recalled in §1b changed how
|
||||
you judged a candidate, name it. The gates are pure functions of the
|
||||
judgments JSON you hand them, so nothing mechanical can tell a reasoned
|
||||
overrule case from a remembered one — saying so is the only control there is
|
||||
(FR-9.2).
|
||||
|
||||
### 6. "Nothing to offload here" is a complete answer
|
||||
|
||||
|
||||
+56
-5
@@ -102,8 +102,11 @@ crux — see `rubric.md`), and keep watching over time.
|
||||
- A **boundary classification** of each candidate against a published rubric.
|
||||
- **On-demand runs** that each persist a snapshot, so history accumulates and
|
||||
successive runs can be diffed.
|
||||
- Two output paths per run: tracked issues on the **target** plugin's repo, and a
|
||||
durable summary on **this** plugin's own wiki.
|
||||
- Output paths per run: tracked issues on the **target** plugin's repo (FR-6), a
|
||||
durable summary on **this** plugin's own wiki (FR-7), and durable context in
|
||||
mempalace so a later run does not re-derive what an earlier one concluded
|
||||
(FR-9). Findings about the *audit itself* go to this plugin's own tracker
|
||||
instead of the target's (FR-6.5).
|
||||
|
||||
### 2.2 Out of scope for v1
|
||||
|
||||
@@ -317,8 +320,22 @@ data. Canonical store: wiki. Optional presentation: artifact.
|
||||
**FR-7.3** The summary states, in this order: attribution coverage and window;
|
||||
the composition headline (how much of the plugin's spend was mechanical); the
|
||||
ranked candidate table with boundary classification and value; the diff against
|
||||
the previous snapshot; the boundary questions (low-confidence candidates) posed
|
||||
to the user; and the issues filed or updated, in full `owner/repo#num` form.
|
||||
the previous snapshot; the boundary questions posed to the user — both the
|
||||
low-confidence candidates and any downgraded by the FR-4.5 stability gate, the
|
||||
latter marked as a defect in the auditor rather than a fact about the
|
||||
target; and the issues filed or updated, in full `owner/repo#num` form,
|
||||
separating those on the target's tracker (FR-6.1) from those on this plugin's
|
||||
own (FR-6.5).
|
||||
|
||||
*Shipped as of v0.5.0, with one gap named rather than papered over:* the
|
||||
analyst's spoken report (`skills/offload-audit` §7) makes both distinctions, but
|
||||
the **rendered wiki page does not yet.** `audit-snapshot`'s summary renderer
|
||||
lists all boundary questions uniformly and shows only each candidate's own
|
||||
issue, because `snapshot_candidate` does not carry the `stability` field and
|
||||
stability findings never enter the snapshot at all — `stability-classify` emits
|
||||
them alongside it, which is why `memory-notes` takes `--stable` as a separate
|
||||
input (FR-9.1). Closing this means widening the snapshot schema, not editing a
|
||||
renderer; until then a self-filed finding reaches the page only via `--note`.
|
||||
|
||||
### FR-8 — Self-application
|
||||
|
||||
@@ -382,6 +399,35 @@ harder at the reasoning, not to short-circuit it — and a remembered "we filed
|
||||
this last time" is not a filing decision at all, because the tracker, not
|
||||
memory, is the FR-5.2 identity of record.
|
||||
|
||||
Note honestly what kind of requirement this is, clause by clause — the mix is
|
||||
not uniform, and calling the whole of FR-9.2 unenforced would be as wrong as
|
||||
calling it a gate.
|
||||
|
||||
Three of its four prohibitions **are** mechanical, and recall cannot reach them:
|
||||
|
||||
- *file something the gates rejected* — `bin/filing-plan` emits `action: file`
|
||||
only for `verdict == "file"`, and skips everything else with the verdict named
|
||||
in the reason;
|
||||
- *skip the FR-6.3 marker search* — `filing-plan plan` makes `--existing`
|
||||
required precisely so that a search which never ran cannot default to
|
||||
"nothing exists";
|
||||
- *re-grade a returned verdict* — the verdict is `boundary-classify`'s output,
|
||||
and `stability-classify` rewrites rather than re-asks it.
|
||||
|
||||
What is left unenforced is the one clause that matters most and is the hardest
|
||||
to watch: **what the analyst writes into the judgments JSON that those gates
|
||||
then consume.** The gates are pure functions of their input, so recall cannot
|
||||
corrupt a verdict directly — it can only corrupt the evidence handed to them, by
|
||||
softening a determinism test or supplying an overrule case remembered rather
|
||||
than reasoned. No script can see that, because the whole value of the read is
|
||||
that it influences judgment; a read that could not move the judgments JSON would
|
||||
be a read with no purpose.
|
||||
|
||||
An audit trail is therefore the only available control, and it is a real
|
||||
obligation, not a hope: **a run that leans on a recalled conclusion says so in
|
||||
its report**, so a human can see the lean and discount it. That is why FR-9.2 is
|
||||
written as prohibitions rather than thresholds.
|
||||
|
||||
**FR-9.3 — Two placements, for two different readers.**
|
||||
|
||||
| | Wing | Room | Read | Write |
|
||||
@@ -648,7 +694,10 @@ another plugin's `${CLAUDE_PLUGIN_ROOT}`. Options: (a) require `cc-tokens` on
|
||||
(c) vendor a copy. Recommendation: (a) with (b) as fallback, and **never** (c) —
|
||||
a vendored copy silently diverges on exactly the two traps that are hardest to
|
||||
notice. Better still: contribute `cc-tokens attribute --by skill|plugin --json`
|
||||
upstream to `token-budget`, so the money math has one home. *Decision needed.*
|
||||
upstream to `token-budget`, so the money math has one home. **Resolved (TASKS
|
||||
0.1): (a) with (b) as fallback**, never (c). `bin/offload-scan` tries `PATH`,
|
||||
then the cache glob, then fails loudly with an install hint. The upstream
|
||||
contribution remains optional (TASKS 8.3).
|
||||
|
||||
**Q2 — Attribution coverage may be thin.** `attributionSkill` appears only when a
|
||||
Skill was formally invoked. Hook-driven work, agent work started without a Skill
|
||||
@@ -663,6 +712,8 @@ which needs no repo access. But FR-6 files issues on the target's repo, which
|
||||
does. For `oleks/*` plugins this is fine. For third-party or official-marketplace
|
||||
plugins there is no writable tracker, and the audit degrades to summary-only
|
||||
(FR-6.4). Confirm that degradation is acceptable rather than an error.
|
||||
**Resolved (TASKS 0.2): yes** — `filing: unavailable` with the reason recorded,
|
||||
and the run summary still written.
|
||||
|
||||
**Q4 — False positives are the real failure mode.** Recommending a script where
|
||||
judgment is needed produces something worse than the status quo: a fast,
|
||||
|
||||
+23
-23
@@ -26,45 +26,45 @@ These block design, not just code. Answers go into `Methodology/Calibration.md`.
|
||||
(`imagex:*`) plugins are excluded from the shared wiki entirely.
|
||||
- [ ] **0.4** Resolve Q5: build the Goodhart counter-metric (wontfix/revert rate)
|
||||
in v1, or note it as a manual check?
|
||||
- [ ] **0.5** Confirm the plugin name `inference-arbitrage` (vs. keeping the
|
||||
placeholder `token-offload-audit`). The directory name must equal the
|
||||
manifest name, so this decision also fixes the checkout path.
|
||||
- [ ] **0.6** Choose the Gitea **org**: `oleks` or `kotkan`. Not derivable from
|
||||
the name. `kotkan` already holds `token-budget`, `worktree-discipline`, and
|
||||
`anti-patterns` — the three closest relatives — which argues for `kotkan`.
|
||||
- [x] **0.5** Confirm the plugin name `inference-arbitrage` (vs. keeping the
|
||||
placeholder `token-offload-audit`). **Resolved: `inference-arbitrage`**;
|
||||
directory renamed to match the manifest name.
|
||||
- [x] **0.6** Choose the Gitea **org**: `oleks` or `kotkan`. **Resolved:
|
||||
`kotkan`** — `kotkan/claude-plugin-inference-arbitrage`, alongside
|
||||
`token-budget`, `worktree-discipline`, and `anti-patterns`.
|
||||
|
||||
## Phase 1 — Repo scaffold
|
||||
|
||||
- [ ] **1.1** Rename `~/projects/claude-plugins/token-offload-audit/` to match the
|
||||
- [x] **1.1** Rename `~/projects/claude-plugins/token-offload-audit/` to match the
|
||||
confirmed manifest name (workspace convention: directory == manifest
|
||||
`name`), and build out the PLAN §2 layout: `plugin.json` v0.1.0, MIT
|
||||
LICENSE, `.gitignore` with `.cache/`, `.claude/`, `*.log`. The four spec
|
||||
files move with the directory and stay in-tree as the design record.
|
||||
- [ ] **1.2** `git init -b main` **inside the plugin subdir** (the workspace root
|
||||
- [x] **1.2** `git init -b main` **inside the plugin subdir** (the workspace root
|
||||
is deliberately not a repo), create `<org>/claude-plugin-<name>` on Gitea
|
||||
with no auto-init (via `cluster:gitea-agent`), push `main`.
|
||||
- [ ] **1.2b** Verify `bin/` scripts carry the executable bit
|
||||
- [x] **1.2b** Verify `bin/` scripts carry the executable bit
|
||||
(`git ls-files -s` → `100755`) — three shipped scripts make this a live
|
||||
concern. Re-check `git status` after every push: the global pre-push hook
|
||||
auto-formats markdown/JSON and may append its own commit.
|
||||
- [ ] **1.2c** Register in the local `oleks-local` marketplace
|
||||
- [x] **1.2c** Register in the local `oleks-local` marketplace
|
||||
(`~/projects/claude-plugins/.claude-plugin/marketplace.json`,
|
||||
`"source": "./<name>"`) so Phases 2–7 can test via `@oleks-local` without
|
||||
publishing anything publicly.
|
||||
- [ ] **1.3** Initialize its wiki with `Methodology/Rubric.md` (rubric.md
|
||||
verbatim, v1.0.0) and empty `Audits/` + `Data/` scaffolding.
|
||||
- [ ] **1.4** `.woodpecker/test.yaml` running `tests/run-all.sh`. **CI verifies;
|
||||
- [x] **1.4** `.woodpecker/test.yaml` running `tests/run-all.sh`. **CI verifies;
|
||||
nothing heavy runs on emmett.**
|
||||
|
||||
## Phase 2 — Static pass
|
||||
|
||||
- [ ] **2.1** `bin/plugin-inventory`: target resolution (path or name, semver
|
||||
- [x] **2.1** `bin/plugin-inventory`: target resolution (path or name, semver
|
||||
pick, skip `.orphaned_at`), frontmatter parsing (~30 lines, no PyYAML).
|
||||
- [ ] **2.2** Signal extraction: procedural density, verb ratio, command-block
|
||||
- [x] **2.2** Signal extraction: procedural density, verb ratio, command-block
|
||||
signatures, rule tables, script coverage, hook/prose drift.
|
||||
- [ ] **2.3** `references/signals-catalog.md` with the verb lexicons and
|
||||
- [x] **2.3** `references/signals-catalog.md` with the verb lexicons and
|
||||
thresholds, loaded at runtime — tuning must be a reviewable diff.
|
||||
- [ ] **2.4** `tests/inventory.test.sh` against a fixture plugin tree.
|
||||
- [x] **2.4** `tests/inventory.test.sh` against a fixture plugin tree.
|
||||
- [ ] **2.5** **Calibration run** against `token-budget`, `worktree-discipline`,
|
||||
`anxious`, `memory`. Record the actual numbers in
|
||||
`Methodology/Calibration.md` and tune thresholds until the ordering matches
|
||||
@@ -72,19 +72,19 @@ These block design, not just code. Answers go into `Methodology/Calibration.md`.
|
||||
|
||||
## Phase 3 — Dynamic pass
|
||||
|
||||
- [ ] **3.1** `bin/offload-scan` skeleton: streaming JSONL reader, attribution
|
||||
- [x] **3.1** `bin/offload-scan` skeleton: streaming JSONL reader, attribution
|
||||
filter, `(message.id, requestId)` de-dup — delegating token/cost arithmetic
|
||||
to `cc-tokens` per decision 0.1. **Do not reimplement the pricing table or
|
||||
the streaming-snapshot max.**
|
||||
- [ ] **3.2** Invocation reconstruction (contiguous attributed runs per session).
|
||||
- [ ] **3.3** Turn classification: mechanical / judgment / retry, with the
|
||||
- [x] **3.2** Invocation reconstruction (contiguous attributed runs per session).
|
||||
- [x] **3.3** Turn classification: mechanical / judgment / retry, with the
|
||||
mechanical tool set and the output-token threshold in the signals catalog.
|
||||
- [ ] **3.4** Tool-signature normalization and masking (also satisfies FR-3.5).
|
||||
- [ ] **3.5** N-gram mining: recurring sequences ≥3 long in ≥3 invocations.
|
||||
- [ ] **3.6** Metrics: MTR, `offload_waste`, read amplification, retry density,
|
||||
- [x] **3.4** Tool-signature normalization and masking (also satisfies FR-3.5).
|
||||
- [x] **3.5** N-gram mining: recurring sequences ≥3 long in ≥3 invocations.
|
||||
- [x] **3.6** Metrics: MTR, `offload_waste`, read amplification, retry density,
|
||||
judgment density, fan-out multiplier, composite `offload_value`.
|
||||
- [ ] **3.7** Attribution coverage computation (FR-3.4).
|
||||
- [ ] **3.8** `tests/scan.test.sh` against **synthetic** fixture transcripts with
|
||||
- [x] **3.7** Attribution coverage computation (FR-3.4).
|
||||
- [x] **3.8** `tests/scan.test.sh` against **synthetic** fixture transcripts with
|
||||
hand-computed expected metrics. Do not test against real transcripts —
|
||||
they contain private content and they change.
|
||||
- [ ] **3.9** Verify runtime and memory on the real history: one process,
|
||||
|
||||
@@ -316,6 +316,18 @@ previous snapshot → boundary questions, as questions → drift notes, with the
|
||||
documentation fix → issues filed or updated, in full `owner/repo#num` form →
|
||||
the wiki page path.
|
||||
|
||||
Two distinctions the rendered wiki page cannot yet make, and you must: mark any
|
||||
boundary question that got there via the FR-4.5 stability gate as **a defect in
|
||||
this auditor, not a fact about the target**, and list issues on this plugin's
|
||||
own tracker (FR-6.5) separately from those on the target's (FR-6.1).
|
||||
|
||||
**Say when you leaned on memory.** If a step 2b result changed how you judged a
|
||||
candidate — you weighed a determinism test differently, or reached for a
|
||||
remembered overrule case — name it in the report. Nothing mechanical can catch
|
||||
this (FR-9.2): the gates are pure functions of the judgments JSON you hand them,
|
||||
so the only control on a recalled conclusion quietly becoming evidence is that
|
||||
you say it out loud where a human can discount it.
|
||||
|
||||
Rank by measured `offload_value` and say that you are doing so. Name skills and
|
||||
agents, **never UUIDs or session ids**. Emit **only shapes and counts** — no
|
||||
transcript content, file contents, command arguments, or user prose ever enters
|
||||
|
||||
Reference in New Issue
Block a user