diff --git a/agents/offload-analyst.md b/agents/offload-analyst.md index 368cfdb..c72213a 100644 --- a/agents/offload-analyst.md +++ b/agents/offload-analyst.md @@ -221,6 +221,11 @@ into an issue body on the target's repo; a hedge a reader can skip is not a gate - List drift notes under their own heading, with the documentation fix. - **Emit only shapes and counts.** Never copy transcript content, file contents, command arguments, or user prose into a report (FR-3.5). +- **Say when you leaned on memory.** If something recalled in §1b changed how + you judged a candidate, name it. The gates are pure functions of the + judgments JSON you hand them, so nothing mechanical can tell a reasoned + overrule case from a remembered one — saying so is the only control there is + (FR-9.2). ### 6. "Nothing to offload here" is a complete answer diff --git a/design/spec.md b/design/spec.md index 54d6001..50dad22 100644 --- a/design/spec.md +++ b/design/spec.md @@ -102,8 +102,11 @@ crux — see `rubric.md`), and keep watching over time. - A **boundary classification** of each candidate against a published rubric. - **On-demand runs** that each persist a snapshot, so history accumulates and successive runs can be diffed. -- Two output paths per run: tracked issues on the **target** plugin's repo, and a - durable summary on **this** plugin's own wiki. +- Output paths per run: tracked issues on the **target** plugin's repo (FR-6), a + durable summary on **this** plugin's own wiki (FR-7), and durable context in + mempalace so a later run does not re-derive what an earlier one concluded + (FR-9). Findings about the *audit itself* go to this plugin's own tracker + instead of the target's (FR-6.5). ### 2.2 Out of scope for v1 @@ -317,8 +320,22 @@ data. Canonical store: wiki. Optional presentation: artifact. **FR-7.3** The summary states, in this order: attribution coverage and window; the composition headline (how much of the plugin's spend was mechanical); the ranked candidate table with boundary classification and value; the diff against -the previous snapshot; the boundary questions (low-confidence candidates) posed -to the user; and the issues filed or updated, in full `owner/repo#num` form. +the previous snapshot; the boundary questions posed to the user — both the +low-confidence candidates and any downgraded by the FR-4.5 stability gate, the +latter marked as a defect in the auditor rather than a fact about the +target; and the issues filed or updated, in full `owner/repo#num` form, +separating those on the target's tracker (FR-6.1) from those on this plugin's +own (FR-6.5). + +*Shipped as of v0.5.0, with one gap named rather than papered over:* the +analyst's spoken report (`skills/offload-audit` §7) makes both distinctions, but +the **rendered wiki page does not yet.** `audit-snapshot`'s summary renderer +lists all boundary questions uniformly and shows only each candidate's own +issue, because `snapshot_candidate` does not carry the `stability` field and +stability findings never enter the snapshot at all — `stability-classify` emits +them alongside it, which is why `memory-notes` takes `--stable` as a separate +input (FR-9.1). Closing this means widening the snapshot schema, not editing a +renderer; until then a self-filed finding reaches the page only via `--note`. ### FR-8 — Self-application @@ -382,6 +399,35 @@ harder at the reasoning, not to short-circuit it — and a remembered "we filed this last time" is not a filing decision at all, because the tracker, not memory, is the FR-5.2 identity of record. +Note honestly what kind of requirement this is, clause by clause — the mix is +not uniform, and calling the whole of FR-9.2 unenforced would be as wrong as +calling it a gate. + +Three of its four prohibitions **are** mechanical, and recall cannot reach them: + +- *file something the gates rejected* — `bin/filing-plan` emits `action: file` + only for `verdict == "file"`, and skips everything else with the verdict named + in the reason; +- *skip the FR-6.3 marker search* — `filing-plan plan` makes `--existing` + required precisely so that a search which never ran cannot default to + "nothing exists"; +- *re-grade a returned verdict* — the verdict is `boundary-classify`'s output, + and `stability-classify` rewrites rather than re-asks it. + +What is left unenforced is the one clause that matters most and is the hardest +to watch: **what the analyst writes into the judgments JSON that those gates +then consume.** The gates are pure functions of their input, so recall cannot +corrupt a verdict directly — it can only corrupt the evidence handed to them, by +softening a determinism test or supplying an overrule case remembered rather +than reasoned. No script can see that, because the whole value of the read is +that it influences judgment; a read that could not move the judgments JSON would +be a read with no purpose. + +An audit trail is therefore the only available control, and it is a real +obligation, not a hope: **a run that leans on a recalled conclusion says so in +its report**, so a human can see the lean and discount it. That is why FR-9.2 is +written as prohibitions rather than thresholds. + **FR-9.3 — Two placements, for two different readers.** | | Wing | Room | Read | Write | @@ -648,7 +694,10 @@ another plugin's `${CLAUDE_PLUGIN_ROOT}`. Options: (a) require `cc-tokens` on (c) vendor a copy. Recommendation: (a) with (b) as fallback, and **never** (c) — a vendored copy silently diverges on exactly the two traps that are hardest to notice. Better still: contribute `cc-tokens attribute --by skill|plugin --json` -upstream to `token-budget`, so the money math has one home. *Decision needed.* +upstream to `token-budget`, so the money math has one home. **Resolved (TASKS +0.1): (a) with (b) as fallback**, never (c). `bin/offload-scan` tries `PATH`, +then the cache glob, then fails loudly with an install hint. The upstream +contribution remains optional (TASKS 8.3). **Q2 — Attribution coverage may be thin.** `attributionSkill` appears only when a Skill was formally invoked. Hook-driven work, agent work started without a Skill @@ -663,6 +712,8 @@ which needs no repo access. But FR-6 files issues on the target's repo, which does. For `oleks/*` plugins this is fine. For third-party or official-marketplace plugins there is no writable tracker, and the audit degrades to summary-only (FR-6.4). Confirm that degradation is acceptable rather than an error. +**Resolved (TASKS 0.2): yes** — `filing: unavailable` with the reason recorded, +and the run summary still written. **Q4 — False positives are the real failure mode.** Recommending a script where judgment is needed produces something worse than the status quo: a fast, diff --git a/design/tasks.md b/design/tasks.md index ccb81a6..2ffe39a 100644 --- a/design/tasks.md +++ b/design/tasks.md @@ -26,45 +26,45 @@ These block design, not just code. Answers go into `Methodology/Calibration.md`. (`imagex:*`) plugins are excluded from the shared wiki entirely. - [ ] **0.4** Resolve Q5: build the Goodhart counter-metric (wontfix/revert rate) in v1, or note it as a manual check? -- [ ] **0.5** Confirm the plugin name `inference-arbitrage` (vs. keeping the - placeholder `token-offload-audit`). The directory name must equal the - manifest name, so this decision also fixes the checkout path. -- [ ] **0.6** Choose the Gitea **org**: `oleks` or `kotkan`. Not derivable from - the name. `kotkan` already holds `token-budget`, `worktree-discipline`, and - `anti-patterns` — the three closest relatives — which argues for `kotkan`. +- [x] **0.5** Confirm the plugin name `inference-arbitrage` (vs. keeping the + placeholder `token-offload-audit`). **Resolved: `inference-arbitrage`**; + directory renamed to match the manifest name. +- [x] **0.6** Choose the Gitea **org**: `oleks` or `kotkan`. **Resolved: + `kotkan`** — `kotkan/claude-plugin-inference-arbitrage`, alongside + `token-budget`, `worktree-discipline`, and `anti-patterns`. ## Phase 1 — Repo scaffold -- [ ] **1.1** Rename `~/projects/claude-plugins/token-offload-audit/` to match the +- [x] **1.1** Rename `~/projects/claude-plugins/token-offload-audit/` to match the confirmed manifest name (workspace convention: directory == manifest `name`), and build out the PLAN §2 layout: `plugin.json` v0.1.0, MIT LICENSE, `.gitignore` with `.cache/`, `.claude/`, `*.log`. The four spec files move with the directory and stay in-tree as the design record. -- [ ] **1.2** `git init -b main` **inside the plugin subdir** (the workspace root +- [x] **1.2** `git init -b main` **inside the plugin subdir** (the workspace root is deliberately not a repo), create `/claude-plugin-` on Gitea with no auto-init (via `cluster:gitea-agent`), push `main`. -- [ ] **1.2b** Verify `bin/` scripts carry the executable bit +- [x] **1.2b** Verify `bin/` scripts carry the executable bit (`git ls-files -s` → `100755`) — three shipped scripts make this a live concern. Re-check `git status` after every push: the global pre-push hook auto-formats markdown/JSON and may append its own commit. -- [ ] **1.2c** Register in the local `oleks-local` marketplace +- [x] **1.2c** Register in the local `oleks-local` marketplace (`~/projects/claude-plugins/.claude-plugin/marketplace.json`, `"source": "./"`) so Phases 2–7 can test via `@oleks-local` without publishing anything publicly. - [ ] **1.3** Initialize its wiki with `Methodology/Rubric.md` (rubric.md verbatim, v1.0.0) and empty `Audits/` + `Data/` scaffolding. -- [ ] **1.4** `.woodpecker/test.yaml` running `tests/run-all.sh`. **CI verifies; +- [x] **1.4** `.woodpecker/test.yaml` running `tests/run-all.sh`. **CI verifies; nothing heavy runs on emmett.** ## Phase 2 — Static pass -- [ ] **2.1** `bin/plugin-inventory`: target resolution (path or name, semver +- [x] **2.1** `bin/plugin-inventory`: target resolution (path or name, semver pick, skip `.orphaned_at`), frontmatter parsing (~30 lines, no PyYAML). -- [ ] **2.2** Signal extraction: procedural density, verb ratio, command-block +- [x] **2.2** Signal extraction: procedural density, verb ratio, command-block signatures, rule tables, script coverage, hook/prose drift. -- [ ] **2.3** `references/signals-catalog.md` with the verb lexicons and +- [x] **2.3** `references/signals-catalog.md` with the verb lexicons and thresholds, loaded at runtime — tuning must be a reviewable diff. -- [ ] **2.4** `tests/inventory.test.sh` against a fixture plugin tree. +- [x] **2.4** `tests/inventory.test.sh` against a fixture plugin tree. - [ ] **2.5** **Calibration run** against `token-budget`, `worktree-discipline`, `anxious`, `memory`. Record the actual numbers in `Methodology/Calibration.md` and tune thresholds until the ordering matches @@ -72,19 +72,19 @@ These block design, not just code. Answers go into `Methodology/Calibration.md`. ## Phase 3 — Dynamic pass -- [ ] **3.1** `bin/offload-scan` skeleton: streaming JSONL reader, attribution +- [x] **3.1** `bin/offload-scan` skeleton: streaming JSONL reader, attribution filter, `(message.id, requestId)` de-dup — delegating token/cost arithmetic to `cc-tokens` per decision 0.1. **Do not reimplement the pricing table or the streaming-snapshot max.** -- [ ] **3.2** Invocation reconstruction (contiguous attributed runs per session). -- [ ] **3.3** Turn classification: mechanical / judgment / retry, with the +- [x] **3.2** Invocation reconstruction (contiguous attributed runs per session). +- [x] **3.3** Turn classification: mechanical / judgment / retry, with the mechanical tool set and the output-token threshold in the signals catalog. -- [ ] **3.4** Tool-signature normalization and masking (also satisfies FR-3.5). -- [ ] **3.5** N-gram mining: recurring sequences ≥3 long in ≥3 invocations. -- [ ] **3.6** Metrics: MTR, `offload_waste`, read amplification, retry density, +- [x] **3.4** Tool-signature normalization and masking (also satisfies FR-3.5). +- [x] **3.5** N-gram mining: recurring sequences ≥3 long in ≥3 invocations. +- [x] **3.6** Metrics: MTR, `offload_waste`, read amplification, retry density, judgment density, fan-out multiplier, composite `offload_value`. -- [ ] **3.7** Attribution coverage computation (FR-3.4). -- [ ] **3.8** `tests/scan.test.sh` against **synthetic** fixture transcripts with +- [x] **3.7** Attribution coverage computation (FR-3.4). +- [x] **3.8** `tests/scan.test.sh` against **synthetic** fixture transcripts with hand-computed expected metrics. Do not test against real transcripts — they contain private content and they change. - [ ] **3.9** Verify runtime and memory on the real history: one process, diff --git a/skills/offload-audit/SKILL.md b/skills/offload-audit/SKILL.md index 9bb6be2..918a313 100644 --- a/skills/offload-audit/SKILL.md +++ b/skills/offload-audit/SKILL.md @@ -316,6 +316,18 @@ previous snapshot → boundary questions, as questions → drift notes, with the documentation fix → issues filed or updated, in full `owner/repo#num` form → the wiki page path. +Two distinctions the rendered wiki page cannot yet make, and you must: mark any +boundary question that got there via the FR-4.5 stability gate as **a defect in +this auditor, not a fact about the target**, and list issues on this plugin's +own tracker (FR-6.5) separately from those on the target's (FR-6.1). + +**Say when you leaned on memory.** If a step 2b result changed how you judged a +candidate — you weighed a determinism test differently, or reached for a +remembered overrule case — name it in the report. Nothing mechanical can catch +this (FR-9.2): the gates are pure functions of the judgments JSON you hand them, +so the only control on a recalled conclusion quietly becoming evidence is that +you say it out loud where a human can discount it. + Rank by measured `offload_value` and say that you are doing so. Name skills and agents, **never UUIDs or session ids**. Emit **only shapes and counts** — no transcript content, file contents, command arguments, or user prose ever enters