claude-plugin-inference-arbitrage/design/spec.md

spec.md — inference-arbitrage

One-line description. Audits a Claude Code plugin — its definition and its measured transcript usage — for work that is currently paid for as LLM inference but is really a deterministic algorithm, and proposes where to cut the boundary between script and judgment.

Status. Spec only. No code exists.

Naming note. This spec proposes the plugin be named inference-arbitrage (rationale in plan.md §1). The checkout currently lives at ~/projects/claude-plugins/token-offload-audit/ under a placeholder directory name. Per this workspace's convention the directory name must equal the manifest name, so if the proposed name is accepted the directory is renamed at scaffold time (TASKS 1.1) — the spec files simply move with it. If the user prefers token-offload-audit, everything here holds with that substituted; nothing in the design depends on the name.

Target repo (to be created): claude-plugin-<name>, in either the oleks or the kotkan org. The org is not derivable from the name and must be chosen deliberately — kotkan ("Kotkan Workload Development") already holds the two plugins this one is most closely related to (kotkan/claude-plugin-token-budget, kotkan/claude-plugin-worktree-discipline) as well as anti-patterns, which argues for kotkan. Published to claude-plugins.oleks.space per the plugin-publishing skill.

---

1. Why

1.1 The economics that make this worth building

token-budget established the governing fact of Claude Code cost, and this spec inherits it rather than re-deriving it:

~95%+ of raw tokens are cache_read; well under 1% is generated output. Cost scales with (session length × context size), largely independent of work produced.

The consequence for this plugin is sharp and non-obvious:

Every turn the model spends emitting a mechanical tool call costs approximately one full context re-read. A step like "run git worktree list, parse it, and decide which entries are stale" does not cost "a bit of reasoning" — it costs ~200k tokens of re-read, at 0.1× input rate, per turn, and it costs the same whether the step was hard or trivial. A 6-turn deterministic loop inside a skill is 6 full context re-reads to produce an answer a 40-line script produces for zero tokens.

So the offload lever is not "the model writes too much." It is: how many turns does the model have to stay in the loop for, on steps that produce no judgment. That is a measurable quantity, and it is what this plugin measures.

1.2 The precedent — this already worked, twice, by hand

Two plugins in this environment already made the cut correctly, and both are usable as calibration fixtures:

• token-budget. The entire transcript-parsing algorithm — JSONL streaming, (message.id, requestId) de-duplication, streaming-snapshot output_tokens maxing, the pricing table, per-session aggregation — lives in bin/cc-tokens (606 lines of stdlib Python). The skills carry only what is irreducibly judgment: which subcommand this question needs, how to frame the dollar caveat, which of two remediation skills a finding routes to, and whether the answer is "nothing is structurally wrong here." The boundary is drawn at exactly the right place, and nobody has to say so out loud because the split is structural.
• worktree-discipline. The rule that a worktree is only safe to remove if its commits have a remote upstream was being got wrong by inference: a bare @{u} check resolves even in a repo with no remote, because git worktree add -b sets up local tracking. The global CLAUDE.md records this explicitly — "Use worktree-audit, which makes that call correctly; a bare @{u} check does not." The classification moved into bin/worktree-audit; the skill kept the judgment (is this branch someone's in-progress work? is this worth reconciling or rescuing?).

Both were discovered by an incident. The point of this plugin is to find the third one before the incident, and to find it with a number attached.

1.3 The user's actual framing

"searching for the possibility points to offload algorithms into scripts from bare inference … not only as a one-time optimization, but also as ongoing efficiency monitoring with usage analysis … the problem here is to correctly understand a points where we need intelligence between script entities correctly positioned"

Three requirements fall out, and they are the three functional pillars below: find the candidates (static + dynamic), place the boundary correctly (the crux — see rubric.md), and keep watching over time.

---

2. Scope

2.1 In scope

• Auditing any Claude Code plugin, addressed by directory path or by name resolved against ~/.claude/plugins/cache/<source>/<name>/<version>/.
• A static pass over the plugin definition (skills, agents, commands, hooks, references, existing scripts).
• A dynamic pass over local session transcripts (~/.claude/projects/**/*.jsonl), attributing measured token spend to the target plugin's skills and agents.
• A boundary classification of each candidate against a published rubric.
• On-demand runs that each persist a snapshot, so history accumulates and successive runs can be diffed.
• Two output paths per run: tracked issues on the target plugin's repo, and a durable summary on this plugin's own wiki.

2.2 Out of scope for v1

• Implementing the offload. The plugin proposes and measures; a human or a separate work session writes the script. Rationale: an auditor that also writes the code it recommends loses the ability to say "no offload here."
• Real-time hooks or a mandatory cron. Cadence is on-demand by explicit instruction (user or another agent). A user may wrap it in the loop or schedule skills if they want recurrence — that is their choice, not the plugin's default.
• Auditing non-plugin agent code, MCP servers, or arbitrary repos.
• Any network-dependent token accounting. The dynamic pass is offline, like cc-tokens.

2.3 Non-goals worth naming

• Not a linter. It does not enforce style on SKILL.md. It reasons about cost and determinism only.
• Not a benchmark. It never re-runs a plugin to measure it. It reads what already happened.
• Not an optimizer of prose length. Trimming a skill body is a context-diet concern and belongs to token-budget. This plugin is about turn count on mechanical work, which is a different and larger lever.

---

3. Functional requirements

FR-1 — Target resolution

FR-1.1 Accept a target as a filesystem path, or as a plugin name resolved through ~/.claude/plugins/cache/*/<name>/ (picking the highest semver directory, excluding any marked .orphaned_at).

FR-1.2 Record, in every snapshot, the resolved absolute path, the plugin name and version from .claude-plugin/plugin.json, and the source marketplace. Audits of different versions must be comparable but distinguishable.

FR-1.3 Refuse to audit a directory with no .claude-plugin/plugin.json, with a clear error naming what was expected. Do not guess at a plugin-shaped layout.

FR-2 — Static pass

FR-2.1 Enumerate the target's surface deterministically, via a script, not by having a model read files: skills/*/SKILL.md (name, description, trigger phrases, allowed-tools, body word count, heading structure), agents/*.md (name, model, tools allowlist, body word count), commands/*, hooks/* and hooks.json, bin/*, references/*, tests/*.

FR-2.2 Emit per-skill and per-agent static signals (defined in §5.1) as structured JSON — never as prose. The model consumes the JSON.

FR-2.3 Compute a script-coverage ratio per plugin: prose-instruction volume vs. shipped executable volume, plus the count of literal command blocks appearing in more than one skill (a duplicated command block is a missing shared script).

FR-2.4 The static pass must run to completion on a plugin with zero transcript history. A brand-new plugin is auditable on definition alone; the report must then state that no measured evidence exists and mark every candidate unmeasured.

FR-3 — Dynamic (usage) pass

FR-3.1 Attribute measured token spend to the target plugin's skills using the attributionPlugin / attributionSkill fields present on assistant lines in the transcripts, and to its agents via agentName / isSidechain.

Verified during spec research: these fields exist and are populated (attributionPlugin: "memory", attributionSkill: "memory:save", etc.), and every attributed assistant line sampled carried a full message.usage block (1129/1129 in a 60-transcript sample). Per-skill cost attribution is therefore directly computable, not an estimate.

FR-3.2 Reuse token-budget's accounting rather than reimplementing it. The two documented transcript traps — one message spanning many JSONL lines all repeating the same usage, and output_tokens being streaming snapshots that must be maxed per (message.id, requestId) — are exactly the kind of thing that is silently got wrong on a reimplementation. See PLAN §4.3 for the mechanism and its risk.

FR-3.3 Compute the offload signals defined in §5.2 — Mechanical Turn Ratio, repetition signature, read amplification, retry density, judgment density, fan-out multiplier — per skill and per agent.

FR-3.4 Report attribution coverage for every audit: what fraction of the window's turns plausibly belonging to this plugin were actually attributable. Work driven by a plugin's hooks, or by an agent invoked without a Skill call, may carry no attribution. An audit that silently understates coverage is worse than no audit, so coverage is a required headline field, not a footnote.

FR-3.5 Emit only shapes and counts — normalized tool-call signatures, counts, token sums. Never copy raw transcript content, file contents, command arguments, or user prose into a snapshot, wiki page, or issue. Transcripts contain secrets and private work. Argument literals are masked (see §5.2.2).

FR-4 — Boundary classification

FR-4.1 Every candidate is classified into exactly one of four positions — pure-script, script-with-compiled-judgment, llm-over-script-digest, pure-inference — by the rubric in rubric.md.

FR-4.2 No candidate may be filed as an issue unless the agent can produce the falsifiability triple (rubric §5): the proposed function signature, three input→expected-output pairs including one edge case, and the case where a human would overrule the script. Failure to produce the third element downgrades the candidate to a boundary question reported to the user, never filed.

FR-4.3 Each candidate carries a boundary_confidence (high / medium / low) derived from which of the five determinism tests it passes, and an offload_value in weighted tokens and as a share of the plugin's audited spend.

FR-4.4 The audit must be able to conclude "nothing to offload here" and say so plainly. token-budget is expected to produce exactly this result; if a run against token-budget produces a list of confident offload candidates, the rubric is miscalibrated and that is a bug in this plugin.

FR-4.5 No candidate may be filed unless its evidence is stable across measurement windows. Before filing, each candidate whose verdict is file is matched by FR-5.2 identity against the most recent prior snapshot recording it, and the two windows' evidence is compared. A candidate whose measurement_strength differs between the windows, or whose share_of_audited_spend falls on opposite sides of the §5.2.7 filing threshold, is downgraded to a boundary question and never filed — and the instability is itself reported as a finding against this plugin's repo (FR-8.2), because a confidence label that moves with the calendar is a defect in the auditor, not a fact about the target. A candidate with no prior window to compare against is marked unchecked and files normally; an absence of history is not evidence of instability, and manufacturing one from it would contradict FR-2.4.

FR-5 — Snapshot accumulation and diffing

FR-5.1 Every run appends one machine-readable snapshot record to an append-only store on this plugin's own Gitea wiki (schema in PLAN §6).

FR-5.2 Candidates carry a stable identity across runs so that "3 of 5 prior recommendations are still unaddressed" is computable. Identity resolution order: (1) an open/closed Gitea issue number, once filed — the strongest and preferred identity; (2) the normalized tool-sequence signature; (3) the (skill, slug) pair assigned at first sighting.

FR-5.3 A run compares against the previous snapshot for the same target and reports per-candidate status: new, persisting, grown, shrunk, resolved, signature-drifted, stale (target version changed such that the skill no longer exists).

FR-5.4 Trend comparison must normalize for usage volume. Absolute token deltas across windows of different activity are meaningless — a quiet week reads as an improvement. Primary trend metrics are cost per invocation and share of the plugin's audited spend; absolute tokens are reported as context only.

FR-5.5 A candidate is resolved only when its issue is closed and the signature's measured cost has dropped below threshold in a later window with at least N invocations. Issue-closed-without-measurement is reported as claimed-fixed-unconfirmed. Committed ≠ verified, in the same spirit as this environment's nixos-deploy-pending discipline.

FR-6 — Output path A: issues on the target repo

FR-6.1 Each high-confidence candidate above the value threshold is filed as a tracked issue on the target plugin's own repo, by delegating to anxious:issuer (which shapes and dispatches; cluster:gitea-agent performs the write). This plugin never writes to Gitea directly.

FR-6.2 Follow the existing environment conventions rather than inventing a scheme: full owner/repo#num references everywhere; labels from the anxious four-axis taxonomy (kind/chore or kind/capability-gap, area/agent-behavior, domain/agents, activity/automate), plus one new cross-cutting label token-offload. Filing is implicit — the audit does not ask permission to file, consistent with the global bug-handling rule.

FR-6.3 Idempotency is mandatory. Each issue body carries a marker line <!-- ia-candidate: <candidate_id> -->. Before filing, search the target repo for that marker; if found, comment the updated measurement on the existing issue instead of opening a duplicate. An on-demand audit that is re-run must not spam the tracker — this is the single most likely way for the plugin to become hated.

FR-6.4 If the target plugin has no writable repo (a third-party or official marketplace plugin), skip path A entirely, record filing: unavailable with the reason, and report the candidate in the summary only.

FR-7 — Output path B: the run summary

FR-7.1 Each run writes a human-readable summary page to this plugin's own Gitea wiki at Audits/<target-plugin>/<YYYY-MM-DD>, and updates Audits/<target-plugin>/Latest.

FR-7.2 Rationale for wiki-over-artifact, since the brief asks for one: the summary must be durable, addressable across sessions, diffable by git, and readable by a future agent that was not present for the run. A Gitea wiki page is all four — the wiki is a git repo, so the snapshot JSONL and the prose report version together and a git log shows the audit history for free. An Artifact is a private, session-produced web page with no cross-run addressing and no relationship to the repo; it cannot be the system of record. An Artifact is offered as an optional rendered view — a trend chart across snapshots — when a human explicitly asks to look at the history, generated from the same snapshot data. Canonical store: wiki. Optional presentation: artifact.

FR-7.3 The summary states, in this order: attribution coverage and window; the composition headline (how much of the plugin's spend was mechanical); the ranked candidate table with boundary classification and value; the diff against the previous snapshot; the boundary questions (low-confidence candidates) posed to the user; and the issues filed or updated, in full owner/repo#num form.

FR-8 — Self-application

FR-8.1 The plugin must pass its own audit. Its static pass is a script, its transcript scan is a script, its snapshot diff is a script; the agent's job is classification, judgment, and reporting. If any of these migrate into skill prose executed by inference, the plugin is violating its own thesis.

FR-8.2 Auditing inference-arbitrage with inference-arbitrage is an acceptance test (TASKS §6).

---

4. Success criteria

#	Criterion	How it is verified
S1	Audits any plugin by path or name, with no target-specific code	Run against token-budget, worktree-discipline, anxious, memory
S2	Correctly returns "nothing to offload" for a well-cut plugin	Run against token-budget → no high-confidence candidates
S3	Rediscovers a known-correct historical cut	Run against worktree-discipline with bin/worktree-audit removed from the inventory → must flag the classification step as high confidence
S4	Finds at least one real, defensible candidate in an unaudited plugin	Run against anxious; user agrees the candidate is genuine
S5	Re-running does not duplicate issues	Run twice; second run comments, does not file
S6	Trend across ≥2 snapshots is computable and volume-normalized	Two runs on different windows; diff reports per-invocation deltas
S7	Passes its own audit	FR-8.2
S8	Runs within emmett's constraints	Single stdlib process, streaming, no subagent fan-out, no toolchain build

S3 is the important one. It is the only criterion that tests the rubric against a cut known to be correct, made by a human, for a reason that is written down.

---

5. Signal catalog

5.1 Static signals — what the definition says

These are computed by bin/plugin-inventory and consumed as JSON.

Signal	Definition	Reads as
Procedural density	Fraction of a skill body that is numbered/imperative steps containing literal commands, vs. discursive prose	A recipe the model re-derives every invocation
Script-verb count	Occurrences of count, sum, rank, sort, diff, parse, validate, enumerate, dedupe, format, check exists, compare	Mechanical intent stated in prose
Judgment-verb count	Occurrences of decide, judge, explain, prioritize, name, write, weigh, interpret, for this user	Irreducible intent stated in prose
Verb ratio	script-verbs / (script-verbs + judgment-verbs) per skill	>0.6 with no bin/ script is the loudest static smell
Duplicated command blocks	Identical or near-identical fenced command blocks appearing in ≥2 skills	A shared script that was never written
Rule tables	Markdown tables that are pure lookup (input → output, finding → action)	A dispatch table being narrated at inference time
Script coverage	executable LOC in bin/ + hooks/ vs. total skill+agent body words	Structural balance; low ratio + high verb ratio is the target profile
Hook/prose drift	Logic present both in a hook script and restated in skill prose	Two sources of truth; the prose will rot
Tool allowlist shape	An agent whose tools are dominated by read-only mechanical tools	The agent was built to fetch, not to judge

Worked example — token-budget (expected: clean). skills/token-audit has high procedural density (it prescribes three exact commands in order) and a rule table (finding → remediation skill). But script coverage is high (606 LOC of cc-tokens against ~3 skill bodies), the verb ratio in the body is judgment-dominated ("report", "interpret", "state the caveat", "rank the fixes by effect"), and — decisively — the prescribed commands are invocations of a script that already exists. The static pass should score this as already cut, with the residual candidate being at most "the fixed three-command opening sequence could be one cc-tokens audit subcommand," classified low value.

Worked example — anxious (expected: real candidates). agents/steward.md sweeps every oleks/* repo and reconciles labels, milestones, and board placement. The taxonomy it reconciles against is a pure lookup table with a documented deterministic ruleset (references/taxonomy.md axis 3 is explicitly described as "steward-owned, repo-derived (deterministic)", and axis 1's derivation is a four-step priority list ending in "defer to human"). A deterministic ruleset, written down, applied per repo by an LLM turn, over N repos, is the canonical shape: the derivation is a script, the "defer to human" branch and the axis-2/axis-4 weak priors are judgment. Expected classification: llm-over-script-digest — a script computes proposed labels and emits only the disagreements and the no-signal cases for the model to rule on.

5.2 Dynamic signals — what the transcripts show

Computed by bin/offload-scan over ~/.claude/projects/**/*.jsonl, filtered to turns attributed to the target.

5.2.1 The primary metric — Mechanical Turn Ratio

A turn is mechanical if all of:

• it contains ≥1 tool_use block, and every one is drawn from the mechanical tool set (read-only Bash, Read, Grep, Glob, list_* / get_* / *_read MCP tools, ToolSearch), and
• its output_tokens are below a threshold (default 400) — the model emitted a tool call and little else, and
• it is not immediately preceded by a tool error (that is a retry, counted separately under 5.2.4).

MTR(skill)             = mechanical_turns / attributed_turns
mechanical_tokens      = Σ over mechanical turns of (cache_read + cache_write + input + output)
offload_waste(skill)   = mechanical_tokens          # weighted, via cc-tokens' tier normalization

offload_waste is the honest number: tokens spent purely to keep the model in the loop on steps that produced no judgment. Because cache_read dominates, it is very nearly mechanical_turns × mean_context_size — which is exactly why turn count, not verbosity, is the lever.

An MTR above ~0.5 on a skill carrying meaningful spend is the primary flag. A high MTR on a skill with trivial spend is noise and is not reported.

5.2.2 Repetition signature — finding the algorithm in the wild

Normalize every tool call to a signature: tool name plus argument shape, with volatile literals masked — paths → <path>, integers → <n>, hex/SHA → <sha>, URLs → <url>, quoted free text → <str>. (This masking is also what satisfies the privacy requirement FR-3.5.)

Within each invocation of a skill, extract the ordered signature sequence. Across invocations, find n-grams of length ≥3 that recur in ≥3 distinct invocations.

A frequently-recurring identical tool sequence is an algorithm, empirically discovered. This is the most direct possible answer to the user's ask, because it does not rely on reading intent out of prose — it observes the procedure being executed the same way repeatedly and prices it. Each recurring n-gram is reported with its occurrence count and its measured offload_waste, and becomes a candidate with a ready-made proposed script boundary.

5.2.3 Read amplification

read_amplification = tokens_pulled_into_context_by_read_tools
                     / tokens_of_that_material_referenced_in_subsequent_output

Approximated by comparing tool-result sizes to the model's next-turn output length and its literal overlap with the result. High amplification means the model is reading a haystack to find a needle — a filter/digest script belongs upstream. This is the signature of the llm-over-script-digest position.

5.2.4 Retry / self-correction density

Fraction of attributed turns that follow a tool error, or that repeat a near-identical signature with adjusted arguments. Every retry is a full context re-read. High density means the invocation is fiddly — the correct offload is often a thin wrapper script that gets the invocation right once, rather than a wholesale algorithm move. Cheap to build, immediately effective.

5.2.5 Judgment density — the brake

Fraction of attributed turns with substantive output_tokens (above threshold) and no tool_use, or containing AskUserQuestion. This is where the model was actually thinking.

Judgment density is a brake, not an accelerator. High judgment density with high spend means the plugin is doing what it should and must be reported as healthy. The offload target is specifically low judgment density × high spend.

5.2.6 Fan-out multiplier

Sidechain turns attributed to the target. Each subagent carries its own full context, so a mechanical step performed inside a subagent costs a multiple of the same step inline. A mechanical n-gram executing inside a fan-out is the highest-value candidate class there is — and this environment already has a recorded finding against one-subagent-per-tiny-step fan-out.

5.2.7 Composite ranking

offload_value = offload_waste × (1 − judgment_density) × repetition_factor

where repetition_factor = 1 + log₂(recurrences of the dominant n-gram). Reported in weighted tokens and as a share of the plugin's audited spend.

Filing threshold (all three required): offload_value ≥ 2% of the audited window's total spend and boundary_confidence == high and the falsifiability triple was produced. Everything else is a boundary question or a note in the summary. The threshold is deliberately conservative: the cost of a missed candidate is a slightly expensive plugin; the cost of a wrong candidate is a plugin that confidently does the wrong thing, forever, silently.

---

6. Risks and open questions

These want the user's input before or during the build.

Q1 — Cross-plugin dependency on cc-tokens. The dynamic pass should not reimplement token-budget's accounting (FR-3.2), but a plugin cannot rely on another plugin's ${CLAUDE_PLUGIN_ROOT}. Options: (a) require cc-tokens on PATH and fail loudly with an install hint; (b) glob ~/.claude/plugins/cache/*/token-budget/*/bin/cc-tokens and pick the newest; (c) vendor a copy. Recommendation: (a) with (b) as fallback, and never (c) — a vendored copy silently diverges on exactly the two traps that are hardest to notice. Better still: contribute cc-tokens attribute --by skill|plugin --json upstream to token-budget, so the money math has one home. Decision needed.

Q2 — Attribution coverage may be thin. attributionSkill appears only when a Skill was formally invoked. Hook-driven work, agent work started without a Skill call, and plugin logic that runs as part of a larger session may carry no attribution at all. A 60-transcript sample surfaced ~1100 attributed lines across a dozen skills — real, but not obviously complete. If coverage for a given target is low, the audit's dynamic pass is weak and must say so loudly (FR-3.4). Open: is there a better attribution path for agent-driven work than agentName?

Q3 — Read access to a target's repo. The audit reads the local plugin cache, which needs no repo access. But FR-6 files issues on the target's repo, which does. For oleks/* plugins this is fine. For third-party or official-marketplace plugins there is no writable tracker, and the audit degrades to summary-only (FR-6.4). Confirm that degradation is acceptable rather than an error.

Q4 — False positives are the real failure mode. Recommending a script where judgment is needed produces something worse than the status quo: a fast, confident, wrong answer with no one watching. The rubric's five determinism tests, the mandatory "when would a human overrule this" element, and the conservative filing threshold are the mitigations, and the plugin never implements its own recommendations. The residual risk is that a plausible-looking candidate gets built by a later session that does not re-check the boundary — which is why the issue body must carry the overrule case, not just the proposal.

Q5 — Goodhart. A plugin that scores plugins on token cost will, if followed blindly, push toward brittle over-scripting. Proposed counter-metric tracked on the trend page: the rate of token-offload issues later closed as wontfix, and of offload scripts subsequently reverted. If that rate climbs, the rubric is too loose. Open: is this worth building in v1, or noted as a manual check?

Q6 — Privacy of the summary surface. Snapshots and wiki pages are derived from transcripts that contain secrets, private prose, and corp work. FR-3.5's masking is the control. Worth confirming the masking list is sufficient, and whether audits of imagex:* (corp) plugins should be excluded from the shared wiki entirely.

Q7 — Local compute budget. Scanning a multi-gigabyte transcript history on emmett must be a single streaming stdlib process, never a fan-out of one subagent per skill. Stated as a hard constraint in PLAN §7, flagged here so it is a conscious decision rather than an accident.

Q8 — Rubric versioning. Snapshots record rubric_version. When the rubric changes, old candidate classifications are not directly comparable. Proposal: diffs across a rubric-version boundary are annotated, not suppressed. Confirm.

---

7. Incidental finding

While surveying the plugin cache for this spec, two leaked git worktrees were found inside the installed plugin cache itself:

~/.claude/plugins/cache/oleks-local/worktree-discipline/1.12.0/.claude/worktrees/wf_fdb5df0f-811-2/
~/.claude/plugins/cache/oleks-local/worktree-discipline/1.12.0/.claude/worktrees/wf_fdb5df0f-811-3/

Each carries a full copy of the plugin tree. These are unnamed-Workflow-subagent worktrees (wf_*) that leaked into a cache directory — a location sweep-worktrees is unlikely to be pointed at, and one that a plugin reinstall would silently orphan. Filed as kotkan/claude-plugin-worktree-discipline#9; not in scope for this spec.