claude-plugin-inference-arbitrage/design/spec.md

# spec.md — `inference-arbitrage`

**One-line description.** Audits a Claude Code plugin — its definition *and* its
measured transcript usage — for work that is currently paid for as LLM inference
but is really a deterministic algorithm, and proposes where to cut the boundary
between script and judgment.

**Status.** Implemented through TASKS Phase 7 (acceptance). Phase 8 (publish) is
outstanding. This document remains the requirements record; where an FR describes
behavior, that behavior ships — see the `bin/`, `skills/`, `agents/`, and
`commands/` trees, and `tests/run-all.sh` for the assertions.

**Naming note.** The proposed name **`inference-arbitrage`** (rationale in
`plan.md` §1) was accepted, and the directory renamed to match the manifest
`name` per this workspace's convention (TASKS 1.1). Nothing in the design depends
on the name.

**Target repo:** `kotkan/claude-plugin-inference-arbitrage`. The org was chosen
deliberately rather than derived — `kotkan` ("Kotkan Workload Development")
already holds the two plugins this one is most closely related to
(`kotkan/claude-plugin-token-budget`, `kotkan/claude-plugin-worktree-discipline`)
as well as `anti-patterns`. Not yet published to `claude-plugins.oleks.space`;
that is TASKS 8.2, per the `plugin-publishing` skill.

---

## 1. Why

### 1.1 The economics that make this worth building

`token-budget` established the governing fact of Claude Code cost, and this spec
inherits it rather than re-deriving it:

> ~95%+ of raw tokens are `cache_read`; well under 1% is generated output. Cost
> scales with **(session length × context size)**, largely independent of work
> produced.

The consequence for *this* plugin is sharp and non-obvious:

**Every turn the model spends emitting a mechanical tool call costs approximately
one full context re-read.** A step like "run `git worktree list`, parse it, and
decide which entries are stale" does not cost "a bit of reasoning" — it costs
~200k tokens of re-read, at 0.1× input rate, *per turn*, and it costs the same
whether the step was hard or trivial. A 6-turn deterministic loop inside a skill
is 6 full context re-reads to produce an answer a 40-line script produces for
zero tokens.

So the offload lever is not "the model writes too much." It is: **how many turns
does the model have to stay in the loop for, on steps that produce no judgment.**
That is a measurable quantity, and it is what this plugin measures.

### 1.2 The precedent — this already worked, twice, by hand

Two plugins in this environment already made the cut correctly, and both are
usable as calibration fixtures:

- **`token-budget`.** The entire transcript-parsing algorithm — JSONL streaming,
  `(message.id, requestId)` de-duplication, streaming-snapshot `output_tokens`
  maxing, the pricing table, per-session aggregation — lives in `bin/cc-tokens`
  (606 lines of stdlib Python). The skills carry only what is irreducibly
  judgment: *which* subcommand this question needs, how to frame the dollar
  caveat, which of two remediation skills a finding routes to, and whether the
  answer is "nothing is structurally wrong here." The boundary is drawn at
  exactly the right place, and nobody has to say so out loud because the split
  is structural.
- **`worktree-discipline`.** The rule that a worktree is only safe to remove if
  its commits have a *remote* upstream was being got wrong by inference: a bare
  `@{u}` check resolves even in a repo with no remote, because `git worktree
  add -b` sets up local tracking. The global `CLAUDE.md` records this explicitly
  — *"Use `worktree-audit`, which makes that call correctly; a bare `@{u}` check
  does not."* The classification moved into `bin/worktree-audit`; the skill kept
  the judgment (is this branch someone's in-progress work? is this worth
  reconciling or rescuing?).

Both were discovered by an incident. The point of this plugin is to find the
third one **before** the incident, and to find it with a number attached.

### 1.3 The user's actual framing

> "searching for the possibility points to offload algorithms into scripts from
> bare inference … not only as a one-time optimization, but also as ongoing
> efficiency monitoring with usage analysis … the problem here is to correctly
> understand a points where we need intelligence between script entities
> correctly positioned"

Three requirements fall out, and they are the three functional pillars below:
find the candidates (static + dynamic), **place the boundary correctly** (the
crux — see `rubric.md`), and keep watching over time.

---

## 2. Scope

### 2.1 In scope

- Auditing **any** Claude Code plugin, addressed by directory path or by name
  resolved against `~/.claude/plugins/cache/<source>/<name>/<version>/`.
- A **static pass** over the plugin definition (skills, agents, commands, hooks,
  references, existing scripts).
- A **dynamic pass** over local session transcripts (`~/.claude/projects/**/*.jsonl`),
  attributing measured token spend to the target plugin's skills and agents.
- A **boundary classification** of each candidate against a published rubric.
- **On-demand runs** that each persist a snapshot, so history accumulates and
  successive runs can be diffed.
- Output paths per run: tracked issues on the **target** plugin's repo (FR-6), a
  durable summary on **this** plugin's own wiki (FR-7), and durable context in
  mempalace so a later run does not re-derive what an earlier one concluded
  (FR-9). Findings about the *audit itself* go to this plugin's own tracker
  instead of the target's (FR-6.5).

### 2.2 Out of scope for v1

- **Implementing** the offload. The plugin proposes and measures; a human or a
  separate work session writes the script. Rationale: an auditor that also
  writes the code it recommends loses the ability to say "no offload here."
- Real-time hooks or a mandatory cron. Cadence is on-demand by explicit
  instruction (user or another agent). A user may wrap it in the `loop` or
  `schedule` skills if they want recurrence — that is their choice, not the
  plugin's default.
- Auditing non-plugin agent code, MCP servers, or arbitrary repos.
- Any network-dependent token accounting. The dynamic pass is offline, like
  `cc-tokens`.

### 2.3 Non-goals worth naming

- **Not a linter.** It does not enforce style on SKILL.md. It reasons about cost
  and determinism only.
- **Not a benchmark.** It never re-runs a plugin to measure it. It reads what
  already happened.
- **Not an optimizer of prose length.** Trimming a skill body is a `context-diet`
  concern and belongs to `token-budget`. This plugin is about *turn count on
  mechanical work*, which is a different and larger lever.

---

## 3. Functional requirements

### FR-1 — Target resolution

**FR-1.1** Accept a target as a filesystem path, or as a plugin name resolved
through `~/.claude/plugins/cache/*/<name>/` (picking the highest semver
directory, excluding any marked `.orphaned_at`).

**FR-1.2** Record, in every snapshot, the resolved absolute path, the plugin
`name` and `version` from `.claude-plugin/plugin.json`, and the source
marketplace. Audits of different versions must be comparable but distinguishable.

**FR-1.3** Refuse to audit a directory with no `.claude-plugin/plugin.json`, with
a clear error naming what was expected. Do not guess at a plugin-shaped layout.

### FR-2 — Static pass

**FR-2.1** Enumerate the target's surface deterministically, via a script, not by
having a model read files: `skills/*/SKILL.md` (name, description, trigger
phrases, `allowed-tools`, body word count, heading structure), `agents/*.md`
(name, `model`, `tools` allowlist, body word count), `commands/*`, `hooks/*` and
`hooks.json`, `bin/*`, `references/*`, `tests/*`.

**FR-2.2** Emit per-skill and per-agent **static signals** (defined in §5.1) as
structured JSON — never as prose. The model consumes the JSON.

**FR-2.3** Compute a **script-coverage ratio** per plugin: prose-instruction
volume vs. shipped executable volume, plus the count of *literal command blocks
appearing in more than one skill* (a duplicated command block is a missing
shared script).

**FR-2.4** The static pass must run to completion on a plugin with zero
transcript history. A brand-new plugin is auditable on definition alone; the
report must then state that no measured evidence exists and mark every candidate
`unmeasured`.

### FR-3 — Dynamic (usage) pass

**FR-3.1** Attribute measured token spend to the target plugin's skills using the
`attributionPlugin` / `attributionSkill` fields present on assistant lines in the
transcripts, and to its agents via `agentName` / `isSidechain`.

*Verified during spec research:* these fields exist and are populated
(`attributionPlugin: "memory"`, `attributionSkill: "memory:save"`, etc.), and
**every** attributed assistant line sampled carried a full `message.usage` block
(1129/1129 in a 60-transcript sample). Per-skill cost attribution is therefore
directly computable, not an estimate.

**FR-3.2** Reuse `token-budget`'s accounting rather than reimplementing it. The
two documented transcript traps — one message spanning many JSONL lines all
repeating the same `usage`, and `output_tokens` being streaming snapshots that
must be maxed per `(message.id, requestId)` — are exactly the kind of thing that
is silently got wrong on a reimplementation. See PLAN §4.3 for the mechanism and
its risk.

**FR-3.3** Compute the **offload signals** defined in §5.2 — Mechanical Turn
Ratio, repetition signature, read amplification, retry density, judgment density,
fan-out multiplier — per skill and per agent.

**FR-3.4** Report **attribution coverage** for every audit: what fraction of the
window's turns plausibly belonging to this plugin were actually attributable.
Work driven by a plugin's hooks, or by an agent invoked without a Skill call, may
carry no attribution. An audit that silently understates coverage is worse than
no audit, so coverage is a required headline field, not a footnote.

**FR-3.5** Emit only **shapes and counts** — normalized tool-call signatures,
counts, token sums. Never copy raw transcript content, file contents, command
arguments, or user prose into a snapshot, wiki page, or issue. Transcripts
contain secrets and private work. Argument literals are masked (see §5.2.2).

### FR-4 — Boundary classification

**FR-4.1** Every candidate is classified into exactly one of four positions —
`pure-script`, `script-with-compiled-judgment`, `llm-over-script-digest`,
`pure-inference` — by the rubric in `rubric.md`.

**FR-4.2** No candidate may be filed as an issue unless the agent can produce the
**falsifiability triple** (rubric §5): the proposed function signature, three
input→expected-output pairs including one edge case, and *the case where a human
would overrule the script*. Failure to produce the third element downgrades the
candidate to a **boundary question** reported to the user, never filed.

**FR-4.3** Each candidate carries a `boundary_confidence` (`high` / `medium` /
`low`) derived from which of the five determinism tests it passes, and an
`offload_value` in weighted tokens and as a share of the plugin's audited spend.

**FR-4.4** The audit must be able to conclude **"nothing to offload here"** and
say so plainly. `token-budget` is expected to produce exactly this result; if a
run against `token-budget` produces a list of confident offload candidates, the
rubric is miscalibrated and that is a bug in this plugin.

**FR-4.5** No candidate may be filed unless its **evidence is stable across
measurement windows**. Before filing, each candidate whose verdict is `file` is
matched by FR-5.2 identity against the most recent prior snapshot recording it,
and the two windows' evidence is compared. A candidate whose
`measurement_strength` differs between the windows, or whose
`share_of_audited_spend` falls on opposite sides of the §5.2.7 filing threshold,
**is downgraded to a boundary question and never filed** — and the instability is
itself reported as a finding against *this* plugin's repo (FR-6.5), because a
confidence label that moves with the calendar is a defect in the auditor, not a
fact about the target. A candidate with no prior window to compare against is
marked `unchecked` and files normally; an absence of history is not evidence of
instability, and manufacturing one from it would contradict FR-2.4.

**FR-4.5.1 — Minimum window width for a downgrade.** A weak reading may only
overturn a `measured` one when it was taken over a comparably wide window. Let
`width(w) = w.until - w.since` in days, read off the window each reading was
taken over (the observed session span the scan records, not the requested
range). A comparison is **not read as instability**, and takes a third verdict
`insufficient-window`, when all of the following hold:

1. the prior reading's `measurement_strength` is `measured`;
2. the current reading's `measurement_strength` is `thin` or `unmeasured`;
3. `width(current) < 0.9 × width(prior)`.

An `insufficient-window` candidate is **neither filed nor downgraded to a
boundary question**, emits **no FR-6.5 self-report**, and is carried forward: the
wider prior snapshot remains the standing comparison point, so the next audit
over an adequately wide window compares against it and not against this
window's under-sample. The run records the widths it decided on, the parity
floor, and the flips it suppressed.

Three cases remain downgrades, and must not be excused by narrowness: the
reverse direction (`thin`/`unmeasured` prior, `measured` current — the earlier
reading is then the weak one); any flip between windows that satisfy the 0.9
parity; and a threshold crossing in which **both** readings are `measured`,
since a window that reached `measured` observed the candidate often enough for
the share to be a claim about relative spend rather than a sampling failure.

The rule is stated as a ratio against the prior window rather than an absolute
number of days because the adequate width is a property of the candidate's
invocation rate, which the auditor does not know in advance:
`anxious/agent-wip/release-policy-derivation` runs roughly once every two days
and needs ~27 days; a once-per-hour step is fully measured in one. The ratio is
0.9 rather than 1.0 because window bounds are observed session spans, so two
nominally identical windows differ by hours. When either window does not carry
both bounds, width is not computable, the flip is **not** excused, and the run
records that the check could not be made — an unreadable window must be visible,
never a silent exemption.

### FR-5 — Snapshot accumulation and diffing

**FR-5.1** Every run appends one machine-readable snapshot record to an
append-only store on this plugin's own Gitea wiki (schema in PLAN §6).

**FR-5.2** Candidates carry a **stable identity** across runs so that "3 of 5
prior recommendations are still unaddressed" is computable. Identity resolution
order: (1) an open/closed Gitea issue number, once filed — the strongest and
preferred identity; (2) the normalized tool-sequence signature; (3) the
`(skill, slug)` pair assigned at first sighting.

**FR-5.3** A run compares against the previous snapshot for the same target and
reports per-candidate status: `new`, `persisting`, `grown`, `shrunk`, `resolved`,
`signature-drifted`, `stale` (target version changed such that the skill no
longer exists).

**FR-5.4** Trend comparison must normalize for usage volume. Absolute token
deltas across windows of different activity are meaningless — a quiet week reads
as an improvement. Primary trend metrics are **cost per invocation** and **share
of the plugin's audited spend**; absolute tokens are reported as context only.

**FR-5.5** A candidate is `resolved` only when its issue is closed **and** the
signature's measured cost has dropped below threshold in a later window with at
least N invocations. Issue-closed-without-measurement is reported as
`claimed-fixed-unconfirmed`. Committed ≠ verified, in the same spirit as this
environment's `nixos-deploy-pending` discipline.

### FR-6 — Output path A: issues on the target repo

**FR-6.1** Each `high`-confidence candidate above the value threshold is filed as
a tracked issue **on the target plugin's own repo**, by delegating to
`anxious:issuer` (which shapes and dispatches; `cluster:gitea-agent` performs the
write). This plugin never writes to Gitea directly.

**FR-6.2** Follow the existing environment conventions rather than inventing a
scheme: full `owner/repo#num` references everywhere; labels from the `anxious`
four-axis taxonomy (`kind/chore` or `kind/capability-gap`,
`area/agent-behavior`, `domain/agents`, `activity/automate`), plus one new
cross-cutting label `token-offload`. Filing is implicit — the audit does not ask
permission to file, consistent with the global bug-handling rule.

**FR-6.3** **Idempotency is mandatory.** Each issue body carries a marker line
`<!-- ia-candidate: <candidate_id> -->`. Before filing, search the target repo
for that marker; if found, comment the updated measurement on the existing issue
instead of opening a duplicate. An on-demand audit that is re-run must not spam
the tracker — this is the single most likely way for the plugin to become
hated.

**FR-6.4** If the target plugin has no writable repo (a third-party or official
marketplace plugin), skip path A entirely, record `filing: unavailable` with the
reason, and report the candidate in the summary only.

**FR-6.5** A finding **about the audit itself** — currently only the FR-4.5
instability downgrade — is filed against **this plugin's own repo**
(`kotkan/claude-plugin-inference-arbitrage`), never the target's. The target did
not change; the auditor described it two ways, so the target's tracker is the
wrong place to put it and filing it there would blame the audited plugin for the
auditor's defect. These findings obey FR-6.3's idempotency rule with their own
marker, `<!-- ia-stability: <candidate_id> -->`, kept distinct from
`ia-candidate` so that a self-finding and a target finding for the same
`candidate_id` cannot collide in a marker search. Delegation is unchanged: this
plugin never writes to Gitea directly.

### FR-7 — Output path B: the run summary

**FR-7.1** Each run writes a human-readable summary page to **this plugin's own**
Gitea wiki at `Audits/<target-plugin>/<YYYY-MM-DD>`, and updates
`Audits/<target-plugin>/Latest`.

**FR-7.2** Rationale for wiki-over-artifact, since the brief asks for one: the
summary must be durable, addressable across sessions, diffable by git, and
readable by a future agent that was not present for the run. A Gitea wiki page is
all four — the wiki is a git repo, so the snapshot JSONL and the prose report
version together and a `git log` shows the audit history for free. An Artifact is
a private, session-produced web page with no cross-run addressing and no
relationship to the repo; it cannot be the system of record. An Artifact **is**
offered as an optional rendered view — a trend chart across snapshots — when a
human explicitly asks to look at the history, generated from the same snapshot
data. Canonical store: wiki. Optional presentation: artifact.

**FR-7.3** The summary states, in this order: attribution coverage and window;
the composition headline (how much of the plugin's spend was mechanical); the
ranked candidate table with boundary classification and value; the diff against
the previous snapshot; the boundary questions posed to the user — both the
low-confidence candidates and any downgraded by the FR-4.5 stability gate, the
latter marked as a defect in the auditor rather than a fact about the
target; any candidate the gate could not judge either way (`insufficient-window`,
FR-4.5.1), stated as a window too narrow to compare rather than as a finding
about the target or the auditor; and the issues filed or updated, in full
`owner/repo#num` form,
separating those on the target's tracker (FR-6.1) from those on this plugin's
own (FR-6.5).

*Shipped as of v0.5.0, with one gap named rather than papered over:* the
analyst's spoken report (`skills/offload-audit` §7) makes both distinctions, but
the **rendered wiki page does not yet.** `audit-snapshot`'s summary renderer
lists all boundary questions uniformly and shows only each candidate's own
issue, because `snapshot_candidate` does not carry the `stability` field and
stability findings never enter the snapshot at all — `stability-classify` emits
them alongside it, which is why `memory-notes` takes `--stable` as a separate
input (FR-9.1). Closing this means widening the snapshot schema, not editing a
renderer; until then a self-filed finding reaches the page only via `--note`.

### FR-8 — Self-application

**FR-8.1** The plugin must pass its own audit. Its static pass is a script, its
transcript scan is a script, its snapshot diff is a script; the agent's job is
classification, judgment, and reporting. If any of these migrate into skill prose
executed by inference, the plugin is violating its own thesis.

**FR-8.2** Auditing `inference-arbitrage` with `inference-arbitrage` is an
acceptance test (S7, TASKS 7.7).

**FR-8.3** **The audit's delegation topology is bounded and fixed.** One audit is
one unit of work: the whole audit may be handed to a single `offload-analyst`
instance, and that hand-off happens **at most once**, from the top-level command
or skill. The analyst is the delegation target, not a further decision point — it
never spawns another `offload-analyst`, and never one agent per skill or per
candidate. Its only outbound `Agent` calls are the fixed filing chain of FR-6.1
(`anxious:issuer` → `cluster:gitea-agent`).

This is a hard requirement and not a style note, because it failed in exactly the
predictable way. The command instructs a caller to delegate a *heavy* target to
the analyst; nothing told the analyst that it *was* that delegation target, so on
a large plugin it re-applied the same heaviness criterion to itself and spawned
another instance, three to four levels deep, each restarting the same measurement
steps with no forward progress. Size is a reason to work through a target
methodically, never a reason to hand it to a fresh copy of yourself with no
memory of what has already been measured.

The rule is load-bearing twice over: it is the S8 runtime constraint (PLAN §7 —
no subagent fan-out, on a 16 GiB laptop), and it is FR-8.1 self-application, since
unbounded self-delegation is precisely the shape of waste this plugin exists to
find. A token-optimization plugin recursing on itself is the most expensive bug
it could possibly have.

### FR-9 — Memory integration

The wiki (FR-7) is this plugin's system of record; **mempalace is where an audit
becomes context for the next one.** Two distinct purposes, and they need
different placement.

**FR-9.1 — The bridge is a script, exactly like FR-7's.** `audit-snapshot
memory-queries` emits the searches to run before grading; `audit-snapshot
memory-notes` emits the `{wing, room, content, source_file, added_by}` payloads
to write after the run, plus the knowledge-graph triples. Neither makes an MCP
call: filesystem in, JSON out. The caller passes both **verbatim** to `search`
and `add_drawer` / `kg_add`.

This is FR-8.1 applied to this feature. *Which* candidates get written about is
already decided by the classify and stability verdicts, both deterministic;
*what text gets written* is a total function of the snapshot. A plugin that
narrated its own bookkeeping at inference time would be filing an issue against
itself.

**FR-9.2 — Reading is advisory, never a gate.** Prior drawers are context the
analyst may cite. They may **never** override `bin/boundary-classify` (FR-4.2)
or `bin/stability-classify` (FR-4.5), and are never a reason to file a candidate
that did not clear the gates, to skip the FR-6.3 marker search, or to re-grade a
verdict already returned. Memory says what was thought before; the gates decide
what happens now. A remembered "we declined this last time" is a reason to look
harder at the reasoning, not to short-circuit it — and a remembered "we filed
this last time" is not a filing decision at all, because the tracker, not
memory, is the FR-5.2 identity of record.

Note honestly what kind of requirement this is, clause by clause — the mix is
not uniform, and calling the whole of FR-9.2 unenforced would be as wrong as
calling it a gate.

Three of its four prohibitions **are** mechanical, and recall cannot reach them:

- *file something the gates rejected* — `bin/filing-plan` emits `action: file`
  only for `verdict == "file"`, and skips everything else with the verdict named
  in the reason;
- *skip the FR-6.3 marker search* — `filing-plan plan` makes `--existing`
  required precisely so that a search which never ran cannot default to
  "nothing exists";
- *re-grade a returned verdict* — the verdict is `boundary-classify`'s output,
  and `stability-classify` rewrites rather than re-asks it.

What is left unenforced is the one clause that matters most and is the hardest
to watch: **what the analyst writes into the judgments JSON that those gates
then consume.** The gates are pure functions of their input, so recall cannot
corrupt a verdict directly — it can only corrupt the evidence handed to them, by
softening a determinism test or supplying an overrule case remembered rather
than reasoned. No script can see that, because the whole value of the read is
that it influences judgment; a read that could not move the judgments JSON would
be a read with no purpose.

An audit trail is therefore the only available control, and it is a real
obligation, not a hope: **a run that leans on a recalled conclusion says so in
its report**, so a human can see the lean and discount it. That is why FR-9.2 is
written as prohibitions rather than thresholds.

**FR-9.3 — Two placements, for two different readers.**

| | Wing | Room | Read | Write |
| --- | --- | --- | --- | --- |
| Per-target findings | the audited plugin's manifest `name` | `inference-arbitrage-audits` | yes | yes |
| Cross-target judgment | `claude-plugins` | `inference-arbitrage-lessons` | yes | yes |

*Target-scoped* mirrors FR-6.1's philosophy — the finding belongs on the
target's own surface, not this plugin's — so a human or agent later working on
that plugin meets the offload findings while searching their own project's
memory. Two drawers per run: the per-run summary, and the qualitative judgment
calls (declined candidates and why, stability downgrades).

*Cross-target* is this plugin's own accumulated judgment — calibration
outcomes, coverage-dilution norms, the shapes that keep turning out to be false
positives — in the wing this plugin's own development history already lives in.
It is written on **every** run regardless of target and regardless of outcome; a
"nothing to offload here" result is exactly the calibration evidence worth
keeping. It is read at the start of every audit, independent of which plugin is
being audited.

**Room names are namespaced by this plugin's name, deliberately.** A wing named
for a plugin can collide with an unrelated wing that already means something
else: `cluster` (the k3s cluster) and `imagex` (corp Jira/Slack/Bitbucket work)
both exist today and mean neither the `cluster` plugin nor the `imagex` plugin.
Namespacing the *room* turns such a collision into a shelving overlap rather
than a content collision — a room-filtered search returns only audit drawers,
and every drawer opens with a line naming what wrote it. Dumping into a generic
`decisions` / `gotchas` room would have made the collision real.

**FR-9.4 — Redaction applies unchanged.** FR-3.5 governs a drawer exactly as it
governs an issue body or a wiki page: shapes, counts and prose *about* the audit
only. Never raw transcript content, file contents, command arguments, or user
prose.

The memory renderers therefore do **not** reuse the wiki renderers. A wiki page
lives in this plugin's own repo; a drawer lands in a wing shared with unrelated
projects, and `snapshot.notes` is caller-supplied free text (`--note`) that
reaches the wiki page legitimately. The memory renderers read from a fixed
**allowlist** of structured snapshot fields — ids, positions, booleans, counts,
ratios — so no free-text field crosses into a drawer at all. Allowlist, never
blocklist: a blocklist is one new snapshot field away from leaking.

**FR-9.5 — The untestable seam is one call wide.** The MCP calls cannot be
exercised by the shell test harness, and are not faked there. That is precisely
why all the judgment lives in the script: the caller's remaining job is a
verbatim tool call, so the part that cannot be tested is the part with nothing
in it to get wrong. `tests/memory.test.sh` covers wing/room resolution, query
packing, drawer selection, rendered content and redaction; the live round trip
is verified by hand (TASKS 7.9.5).

---

## 4. Success criteria

| # | Criterion | How it is verified |
| --- | --- | --- |
| S1 | Audits any plugin by path or name, with no target-specific code | Run against `token-budget`, `worktree-discipline`, `anxious`, `memory` |
| S2 | Correctly returns "nothing to offload" for a well-cut plugin | Run against `token-budget` → no `high`-confidence candidates |
| S3 | Rediscovers a known-correct historical cut | Run against `worktree-discipline` with `bin/worktree-audit` removed from the inventory → must flag the classification step as `high` confidence |
| S4 | Mechanically refuses a candidate whose evidence is a window artifact | The two-window `anxious` fixture → the FR-4.5 gate refuses `agent-wip/release-policy-derivation` (as `insufficient-window`, per FR-4.5.1, since the 7-day current window cannot fairly contradict the 26-day `measured` one), with no human in the loop; a flip between comparably wide windows is still downgraded to a boundary question |
| S5 | Re-running does not duplicate issues | Run twice; second run comments, does not file |
| S6 | Trend across ≥2 snapshots is computable and volume-normalized | Two runs on different windows; diff reports per-invocation deltas |
| S7 | Passes its own audit | FR-8.2 |
| S8 | Runs within emmett's constraints | Single stdlib process, streaming, no subagent fan-out and no self-delegation (FR-8.3), no toolchain build |

S3 is the important one. It is the only criterion that tests the rubric against a
cut *known* to be correct, made by a human, for a reason that is written down.

S4 was originally *"finds a real candidate in `anxious` that the user agrees is
genuine."* That is retired: a human triaging one candidate and pronouncing it
good is not a repeatable test, and it puts the plugin's central safety property
behind whether someone was feeling skeptical that day. What actually happened is
that a human review caught a candidate whose evidence flipped between windows —
so the criterion is now that gate, run mechanically (FR-4.5), against the real
two-window evidence that produced the original finding.

The *reason* S4's candidate is refused changed in v0.7.0, and the change is
itself instructive. Two further real windows (14 and 60 days, same script, same
candidate) showed 0 → 2 → 12 → 15 invocations as the window widened: monotonic
accumulation of a low-frequency step, not a label flipping about. The 7-day
window had not caught an unstable candidate, it had failed to observe a real one,
and calling that instability blamed the candidate for the auditor's window. So
FR-4.5.1 makes the refusal honest — `insufficient-window`, cannot tell either
way, compare again over a wider window — while keeping the property S4 exists to
test: nothing is filed off a window artifact, with no human in the loop.
(kotkan/claude-plugin-inference-arbitrage#15.)

---

## 5. Signal catalog

### 5.1 Static signals — what the definition says

These are computed by `bin/plugin-inventory` and consumed as JSON.

| Signal | Definition | Reads as |
| --- | --- | --- |
| **Procedural density** | Fraction of a skill body that is numbered/imperative steps containing literal commands, vs. discursive prose | A recipe the model re-derives every invocation |
| **Script-verb count** | Occurrences of `count`, `sum`, `rank`, `sort`, `diff`, `parse`, `validate`, `enumerate`, `dedupe`, `format`, `check exists`, `compare` | Mechanical intent stated in prose |
| **Judgment-verb count** | Occurrences of `decide`, `judge`, `explain`, `prioritize`, `name`, `write`, `weigh`, `interpret`, `for this user` | Irreducible intent stated in prose |
| **Verb ratio** | script-verbs / (script-verbs + judgment-verbs) per skill | >0.6 with no `bin/` script is the loudest static smell |
| **Duplicated command blocks** | Identical or near-identical fenced command blocks appearing in ≥2 skills | A shared script that was never written |
| **Rule tables** | Markdown tables that are pure lookup (input → output, finding → action) | A dispatch table being narrated at inference time |
| **Script coverage** | executable LOC in `bin/` + `hooks/` vs. total skill+agent body words | Structural balance; low ratio + high verb ratio is the target profile |
| **Hook/prose drift** | Logic present both in a hook script and restated in skill prose | Two sources of truth; the prose will rot |
| **Tool allowlist shape** | An agent whose `tools` are dominated by read-only mechanical tools | The agent was built to fetch, not to judge |

**Worked example — `token-budget` (expected: clean).** `skills/token-audit` has
high procedural density (it prescribes three exact commands in order) and a rule
table (`finding → remediation skill`). But script coverage is high (606 LOC of
`cc-tokens` against ~3 skill bodies), the verb ratio in the *body* is
judgment-dominated ("report", "interpret", "state the caveat", "rank the fixes by
effect"), and — decisively — the prescribed commands are invocations of a script
that already exists. The static pass should score this as **already cut**, with
the residual candidate being at most "the fixed three-command opening sequence
could be one `cc-tokens audit` subcommand," classified `low` value.

**Worked example — `anxious` (expected: real candidates).** `agents/steward.md`
sweeps *every* `oleks/*` repo and reconciles labels, milestones, and board
placement. The taxonomy it reconciles against is a **pure lookup table** with a
documented deterministic ruleset (`references/taxonomy.md` axis 3 is explicitly
described as *"steward-owned, repo-derived (deterministic)"*, and axis 1's
derivation is a four-step priority list ending in "defer to human"). A
deterministic ruleset, written down, applied per repo by an LLM turn, over N
repos, is the canonical shape: the *derivation* is a script, the *"defer to
human"* branch and the axis-2/axis-4 weak priors are judgment. Expected
classification: `llm-over-script-digest` — a script computes proposed labels and
emits only the disagreements and the no-signal cases for the model to rule on.

### 5.2 Dynamic signals — what the transcripts show

Computed by `bin/offload-scan` over `~/.claude/projects/**/*.jsonl`, filtered to
turns attributed to the target.

#### 5.2.1 The primary metric — Mechanical Turn Ratio

A turn is **mechanical** if all of:

- it contains ≥1 `tool_use` block, and every one is drawn from the mechanical
  tool set (read-only `Bash`, `Read`, `Grep`, `Glob`, `list_*` / `get_*` /
  `*_read` MCP tools, `ToolSearch`), and
- its `output_tokens` are below a threshold (default 400) — the model emitted a
  tool call and little else, and
- it is not immediately preceded by a tool error (that is a retry, counted
  separately under 5.2.4).

```
MTR(skill)             = mechanical_turns / attributed_turns
mechanical_tokens      = Σ over mechanical turns of (cache_read + cache_write + input + output)
offload_waste(skill)   = mechanical_tokens          # weighted, via cc-tokens' tier normalization
```

`offload_waste` is the honest number: **tokens spent purely to keep the model in
the loop on steps that produced no judgment.** Because `cache_read` dominates, it
is very nearly `mechanical_turns × mean_context_size` — which is exactly why turn
count, not verbosity, is the lever.

An MTR above ~0.5 on a skill carrying meaningful spend is the primary flag. A
high MTR on a skill with trivial spend is noise and is not reported.

#### 5.2.2 Repetition signature — finding the algorithm in the wild

Normalize every tool call to a signature: tool name plus argument *shape*, with
volatile literals masked — paths → `<path>`, integers → `<n>`, hex/SHA → `<sha>`,
URLs → `<url>`, quoted free text → `<str>`. (This masking is also what satisfies
the privacy requirement FR-3.5.)

Within each invocation of a skill, extract the ordered signature sequence. Across
invocations, find n-grams of length ≥3 that recur in ≥3 distinct invocations.

**A frequently-recurring identical tool sequence is an algorithm, empirically
discovered.** This is the most direct possible answer to the user's ask, because
it does not rely on reading intent out of prose — it observes the procedure being
executed the same way repeatedly and prices it. Each recurring n-gram is reported
with its occurrence count and its measured `offload_waste`, and becomes a
candidate with a ready-made proposed script boundary.

#### 5.2.3 Read amplification

```
read_amplification = tokens_pulled_into_context_by_read_tools
                     / tokens_of_that_material_referenced_in_subsequent_output
```

Approximated by comparing tool-result sizes to the model's next-turn output
length and its literal overlap with the result. High amplification means the
model is reading a haystack to find a needle — a filter/digest script belongs
upstream. This is the signature of the `llm-over-script-digest` position.

#### 5.2.4 Retry / self-correction density

Fraction of attributed turns that follow a tool error, or that repeat a
near-identical signature with adjusted arguments. Every retry is a full context
re-read. High density means the invocation is fiddly — the correct offload is
often a thin wrapper script that gets the invocation right once, rather than a
wholesale algorithm move. Cheap to build, immediately effective.

#### 5.2.5 Judgment density — the brake

Fraction of attributed turns with substantive `output_tokens` (above threshold)
and *no* `tool_use`, or containing `AskUserQuestion`. This is where the model was
actually thinking.

**Judgment density is a brake, not an accelerator.** High judgment density with
high spend means the plugin is doing what it should and must be reported as
healthy. The offload target is specifically **low judgment density × high spend**.

#### 5.2.6 Fan-out multiplier

Sidechain turns attributed to the target. Each subagent carries its own full
context, so a mechanical step performed inside a subagent costs a multiple of the
same step inline. A mechanical n-gram executing inside a fan-out is the
highest-value candidate class there is — and this environment already has a
recorded finding against one-subagent-per-tiny-step fan-out.

#### 5.2.7 Composite ranking

```
offload_value = offload_waste × (1 − judgment_density) × repetition_factor
```

where `repetition_factor` = 1 + log₂(recurrences of the dominant n-gram).
Reported in weighted tokens and as a share of the plugin's audited spend.

**Measurement strength.** Before the value threshold can be applied, the evidence
must be strong enough for the share to mean anything. Each candidate carries a
`measurement_strength`:

| Value | When |
| --- | --- |
| `measured` | ≥3 invocations **and** ≥30 attributed turns |
| `thin` | attributed, but too few invocations — the share is arithmetically real and evidentially worthless |
| `unmeasured` | no attributed usage at all (the FR-2.4 zero-history case) |

**Filing threshold.** `boundary_confidence == high` **and** the falsifiability
triple was produced (including the overrule case) **and** the FR-4.5 stability
gate passed. Given those, the value test depends on measurement strength:

- `measured` → `offload_value` must also be ≥2% of the audited window's total
  spend. Below it, the candidate is a boundary question.
- `thin` / `unmeasured` → the value test **cannot** be applied and is not
  applied. The candidate files on its **static case** alone, and the issue body
  must state that the value claim is unproven. FR-2.4 requires a plugin with no
  transcript history to stay auditable, and silently withholding a
  correctness-complete candidate because it happens to be unmeasured would make
  every brand-new plugin audit return nothing.

That asymmetry is deliberate and is the one place a candidate reaches the tracker
without a cost number behind it. It is safe only because the correctness gates —
the five determinism tests, the overrule case — are unconditional; what varies is
whether the plugin can also say *how much it is worth*. Conversely the value
threshold is deliberately conservative where it does apply: the cost of a missed
candidate is a slightly expensive plugin; the cost of a wrong candidate is a
plugin that confidently does the wrong thing, forever, silently.

Two verdicts sit outside this ladder entirely. A candidate classified
`pure-inference` (position 4) is reported as **`no-offload`** — an explicit
finding that the step is correctly done by inference, per FR-4.4, not a silent
omission. A hook/prose drift signal (§5.1) is reported as a **`drift-note`**: it
is a restated configuration contract with two sources of truth, and its fix is to
point the prose at the hook file, not to write a new script.

---

## 6. Risks and open questions

These want the user's input before or during the build.

**Q1 — Cross-plugin dependency on `cc-tokens`.** The dynamic pass should not
reimplement `token-budget`'s accounting (FR-3.2), but a plugin cannot rely on
another plugin's `${CLAUDE_PLUGIN_ROOT}`. Options: (a) require `cc-tokens` on
`PATH` and fail loudly with an install hint; (b) glob
`~/.claude/plugins/cache/*/token-budget/*/bin/cc-tokens` and pick the newest;
(c) vendor a copy. Recommendation: (a) with (b) as fallback, and **never** (c) —
a vendored copy silently diverges on exactly the two traps that are hardest to
notice. Better still: contribute `cc-tokens attribute --by skill|plugin --json`
upstream to `token-budget`, so the money math has one home. **Resolved (TASKS
0.1): (a) with (b) as fallback**, never (c). `bin/offload-scan` tries `PATH`,
then the cache glob, then fails loudly with an install hint. The upstream
contribution remains optional (TASKS 8.3).

**Q2 — Attribution coverage may be thin.** `attributionSkill` appears only when a
Skill was formally invoked. Hook-driven work, agent work started without a Skill
call, and plugin logic that runs as part of a larger session may carry no
attribution at all. A 60-transcript sample surfaced ~1100 attributed lines across
a dozen skills — real, but not obviously complete. If coverage for a given target
is low, the audit's dynamic pass is weak and must say so loudly (FR-3.4). *Open:
is there a better attribution path for agent-driven work than `agentName`?*

**Q3 — Read access to a target's repo.** The audit reads the local plugin cache,
which needs no repo access. But FR-6 files issues on the target's repo, which
does. For `oleks/*` plugins this is fine. For third-party or official-marketplace
plugins there is no writable tracker, and the audit degrades to summary-only
(FR-6.4). Confirm that degradation is acceptable rather than an error.
**Resolved (TASKS 0.2): yes** — `filing: unavailable` with the reason recorded,
and the run summary still written.

**Q4 — False positives are the real failure mode.** Recommending a script where
judgment is needed produces something worse than the status quo: a fast,
confident, wrong answer with no one watching. The rubric's five determinism
tests, the mandatory "when would a human overrule this" element, and the
conservative filing threshold are the mitigations, and the plugin never
implements its own recommendations. The residual risk is that a plausible-looking
candidate gets built by a later session that does not re-check the boundary —
which is why the issue body must carry the overrule case, not just the proposal.

**Q5 — Goodhart.** A plugin that scores plugins on token cost will, if followed
blindly, push toward brittle over-scripting. Proposed counter-metric tracked on
the trend page: the rate of `token-offload` issues later closed as `wontfix`, and
of offload scripts subsequently reverted. If that rate climbs, the rubric is too
loose. *Open: is this worth building in v1, or noted as a manual check?*

**Q6 — Privacy of the summary surface.** Snapshots and wiki pages are derived
from transcripts that contain secrets, private prose, and corp work. FR-3.5's
masking is the control. Worth confirming the masking list is sufficient, and
whether audits of `imagex:*` (corp) plugins should be excluded from the shared
wiki entirely.

**Q7 — Local compute budget.** Scanning a multi-gigabyte transcript history on
emmett must be a single streaming stdlib process, never a fan-out of one subagent
per skill. Stated as a hard constraint in PLAN §7, flagged here so it is a
conscious decision rather than an accident.

**Q8 — Rubric versioning.** Snapshots record `rubric_version`. When the rubric
changes, old candidate classifications are not directly comparable. Proposal:
diffs across a rubric-version boundary are annotated, not suppressed. Confirm.

---

## 7. Incidental finding

While surveying the plugin cache for this spec, two leaked git worktrees were
found inside the installed plugin cache itself:

```
~/.claude/plugins/cache/oleks-local/worktree-discipline/1.12.0/.claude/worktrees/wf_fdb5df0f-811-2/
~/.claude/plugins/cache/oleks-local/worktree-discipline/1.12.0/.claude/worktrees/wf_fdb5df0f-811-3/
```

Each carries a full copy of the plugin tree. These are unnamed-Workflow-subagent
worktrees (`wf_*`) that leaked into a *cache* directory — a location `sweep-worktrees`
is unlikely to be pointed at, and one that a plugin reinstall would silently
orphan. Filed as **kotkan/claude-plugin-worktree-discipline#9**; not in scope for
this spec.