claude-plugin-inference-arbitrage/design/spec.md

spec.md — inference-arbitrage

One-line description. Audits a Claude Code plugin — its definition and its measured transcript usage — for work that is currently paid for as LLM inference but is really a deterministic algorithm, and proposes where to cut the boundary between script and judgment.

Status. Implemented through TASKS Phase 7 (acceptance). Phase 8 (publish) is outstanding. This document remains the requirements record; where an FR describes behavior, that behavior ships — see the bin/, skills/, agents/, and commands/ trees, and tests/run-all.sh for the assertions.

Naming note. The proposed name inference-arbitrage (rationale in plan.md §1) was accepted, and the directory renamed to match the manifest name per this workspace's convention (TASKS 1.1). Nothing in the design depends on the name.

Target repo: kotkan/claude-plugin-inference-arbitrage. The org was chosen deliberately rather than derived — kotkan ("Kotkan Workload Development") already holds the two plugins this one is most closely related to (kotkan/claude-plugin-token-budget, kotkan/claude-plugin-worktree-discipline) as well as anti-patterns. Not yet published to claude-plugins.oleks.space; that is TASKS 8.2, per the plugin-publishing skill.

---

1. Why

1.1 The economics that make this worth building

token-budget established the governing fact of Claude Code cost, and this spec inherits it rather than re-deriving it:

~95%+ of raw tokens are cache_read; well under 1% is generated output. Cost scales with (session length × context size), largely independent of work produced.

The consequence for this plugin is sharp and non-obvious:

Every turn the model spends emitting a mechanical tool call costs approximately one full context re-read. A step like "run git worktree list, parse it, and decide which entries are stale" does not cost "a bit of reasoning" — it costs ~200k tokens of re-read, at 0.1× input rate, per turn, and it costs the same whether the step was hard or trivial. A 6-turn deterministic loop inside a skill is 6 full context re-reads to produce an answer a 40-line script produces for zero tokens.

So the offload lever is not "the model writes too much." It is: how many turns does the model have to stay in the loop for, on steps that produce no judgment. That is a measurable quantity, and it is what this plugin measures.

1.2 The precedent — this already worked, twice, by hand

Two plugins in this environment already made the cut correctly, and both are usable as calibration fixtures:

• token-budget. The entire transcript-parsing algorithm — JSONL streaming, (message.id, requestId) de-duplication, streaming-snapshot output_tokens maxing, the pricing table, per-session aggregation — lives in bin/cc-tokens (606 lines of stdlib Python). The skills carry only what is irreducibly judgment: which subcommand this question needs, how to frame the dollar caveat, which of two remediation skills a finding routes to, and whether the answer is "nothing is structurally wrong here." The boundary is drawn at exactly the right place, and nobody has to say so out loud because the split is structural.
• worktree-discipline. The rule that a worktree is only safe to remove if its commits have a remote upstream was being got wrong by inference: a bare @{u} check resolves even in a repo with no remote, because git worktree add -b sets up local tracking. The global CLAUDE.md records this explicitly — "Use worktree-audit, which makes that call correctly; a bare @{u} check does not." The classification moved into bin/worktree-audit; the skill kept the judgment (is this branch someone's in-progress work? is this worth reconciling or rescuing?).

Both were discovered by an incident. The point of this plugin is to find the third one before the incident, and to find it with a number attached.

1.3 The user's actual framing

"searching for the possibility points to offload algorithms into scripts from bare inference … not only as a one-time optimization, but also as ongoing efficiency monitoring with usage analysis … the problem here is to correctly understand a points where we need intelligence between script entities correctly positioned"

Three requirements fall out, and they are the three functional pillars below: find the candidates (static + dynamic), place the boundary correctly (the crux — see rubric.md), and keep watching over time.

---

2. Scope

2.1 In scope

• Auditing any Claude Code plugin, addressed by directory path or by name resolved against ~/.claude/plugins/cache/<source>/<name>/<version>/.
• A static pass over the plugin definition (skills, agents, commands, hooks, references, existing scripts).
• A dynamic pass over local session transcripts (~/.claude/projects/**/*.jsonl), attributing measured token spend to the target plugin's skills and agents.
• A boundary classification of each candidate against a published rubric.
• On-demand runs that each persist a snapshot, so history accumulates and successive runs can be diffed.
• Output paths per run: tracked issues on the target plugin's repo (FR-6), a durable summary on this plugin's own wiki (FR-7), and durable context in mempalace so a later run does not re-derive what an earlier one concluded (FR-9). Findings about the audit itself go to this plugin's own tracker instead of the target's (FR-6.5).

2.2 Out of scope for v1

• Implementing the offload. The plugin proposes and measures; a human or a separate work session writes the script. Rationale: an auditor that also writes the code it recommends loses the ability to say "no offload here."
• Real-time hooks or a mandatory cron. Cadence is on-demand by explicit instruction (user or another agent). A user may wrap it in the loop or schedule skills if they want recurrence — that is their choice, not the plugin's default.
• Auditing non-plugin agent code, MCP servers, or arbitrary repos.
• Any network-dependent token accounting. The dynamic pass is offline, like cc-tokens.

2.3 Non-goals worth naming

• Not a linter. It does not enforce style on SKILL.md. It reasons about cost and determinism only.
• Not a benchmark. It never re-runs a plugin to measure it. It reads what already happened.
• Not an optimizer of prose length. Trimming a skill body is a context-diet concern and belongs to token-budget. This plugin is about turn count on mechanical work, which is a different and larger lever.

---

3. Functional requirements

FR-1 — Target resolution

FR-1.1 Accept a target as a filesystem path, or as a plugin name resolved through ~/.claude/plugins/cache/*/<name>/ (picking the highest semver directory, excluding any marked .orphaned_at).

FR-1.2 Record, in every snapshot, the resolved absolute path, the plugin name and version from .claude-plugin/plugin.json, and the source marketplace. Audits of different versions must be comparable but distinguishable.

FR-1.3 Refuse to audit a directory with no .claude-plugin/plugin.json, with a clear error naming what was expected. Do not guess at a plugin-shaped layout.

FR-2 — Static pass

FR-2.1 Enumerate the target's surface deterministically, via a script, not by having a model read files: skills/*/SKILL.md (name, description, trigger phrases, allowed-tools, body word count, heading structure), agents/*.md (name, model, tools allowlist, body word count), commands/*, hooks/* and hooks.json, bin/*, references/*, tests/*.

FR-2.2 Emit per-skill and per-agent static signals (defined in §5.1) as structured JSON — never as prose. The model consumes the JSON.

FR-2.3 Compute a script-coverage ratio per plugin: prose-instruction volume vs. shipped executable volume, plus the count of literal command blocks appearing in more than one skill (a duplicated command block is a missing shared script).

FR-2.4 The static pass must run to completion on a plugin with zero transcript history. A brand-new plugin is auditable on definition alone; the report must then state that no measured evidence exists and mark every candidate unmeasured.

FR-3 — Dynamic (usage) pass

FR-3.1 Attribute measured token spend to the target plugin's skills using the attributionPlugin / attributionSkill fields present on assistant lines in the transcripts, and to its agents via agentName / isSidechain.

Verified during spec research: these fields exist and are populated (attributionPlugin: "memory", attributionSkill: "memory:save", etc.), and every attributed assistant line sampled carried a full message.usage block (1129/1129 in a 60-transcript sample). Per-skill cost attribution is therefore directly computable, not an estimate.

FR-3.2 Reuse token-budget's accounting rather than reimplementing it. The two documented transcript traps — one message spanning many JSONL lines all repeating the same usage, and output_tokens being streaming snapshots that must be maxed per (message.id, requestId) — are exactly the kind of thing that is silently got wrong on a reimplementation. See PLAN §4.3 for the mechanism and its risk.

FR-3.3 Compute the offload signals defined in §5.2 — Mechanical Turn Ratio, repetition signature, read amplification, retry density, judgment density, fan-out multiplier — per skill and per agent.

FR-3.4 Report attribution coverage for every audit: what fraction of the window's turns plausibly belonging to this plugin were actually attributable. Work driven by a plugin's hooks, or by an agent invoked without a Skill call, may carry no attribution. An audit that silently understates coverage is worse than no audit, so coverage is a required headline field, not a footnote.

FR-3.5 Emit only shapes and counts — normalized tool-call signatures, counts, token sums. Never copy raw transcript content, file contents, command arguments, or user prose into a snapshot, wiki page, or issue. Transcripts contain secrets and private work. Argument literals are masked (see §5.2.2).

FR-4 — Boundary classification

FR-4.1 Every candidate is classified into exactly one of four positions — pure-script, script-with-compiled-judgment, llm-over-script-digest, pure-inference — by the rubric in rubric.md.

FR-4.2 No candidate may be filed as an issue unless the agent can produce the falsifiability triple (rubric §5): the proposed function signature, three input→expected-output pairs including one edge case, and the case where a human would overrule the script. Failure to produce the third element downgrades the candidate to a boundary question reported to the user, never filed.

FR-4.3 Each candidate carries a boundary_confidence (high / medium / low) derived from which of the five determinism tests it passes, and an offload_value in weighted tokens and as a share of the plugin's audited spend.

FR-4.4 The audit must be able to conclude "nothing to offload here" and say so plainly. token-budget is expected to produce exactly this result; if a run against token-budget produces a list of confident offload candidates, the rubric is miscalibrated and that is a bug in this plugin.

FR-4.5 No candidate may be filed unless its evidence is stable across measurement windows. Before filing, each candidate whose verdict is file is matched by FR-5.2 identity against the most recent prior snapshot recording it, and the two windows' evidence is compared. A candidate whose measurement_strength differs between the windows, or whose share_of_audited_spend falls on opposite sides of the §5.2.7 filing threshold, is downgraded to a boundary question and never filed — and the instability is itself reported as a finding against this plugin's repo (FR-6.5), because a confidence label that moves with the calendar is a defect in the auditor, not a fact about the target. A candidate with no prior window to compare against is marked unchecked and files normally; an absence of history is not evidence of instability, and manufacturing one from it would contradict FR-2.4.

FR-4.5.1 — Minimum window width for a downgrade. A weak reading may only overturn a measured one when it was taken over a comparably wide window. Let width(w) = w.until - w.since in days, read off the window each reading was taken over (the observed session span the scan records, not the requested range). A comparison is not read as instability, and takes a third verdict insufficient-window, when all of the following hold:

1. the prior reading's measurement_strength is measured;
2. the current reading's measurement_strength is thin or unmeasured;
3. width(current) < 0.9 × width(prior).

An insufficient-window candidate is neither filed nor downgraded to a boundary question, emits no FR-6.5 self-report, and is carried forward: the wider prior snapshot remains the standing comparison point, so the next audit over an adequately wide window compares against it and not against this window's under-sample. The run records the widths it decided on, the parity floor, and the flips it suppressed.

Three cases remain downgrades, and must not be excused by narrowness: the reverse direction (thin/unmeasured prior, measured current — the earlier reading is then the weak one); any flip between windows that satisfy the 0.9 parity; and a threshold crossing in which both readings are measured, since a window that reached measured observed the candidate often enough for the share to be a claim about relative spend rather than a sampling failure.

The rule is stated as a ratio against the prior window rather than an absolute number of days because the adequate width is a property of the candidate's invocation rate, which the auditor does not know in advance: anxious/agent-wip/release-policy-derivation runs roughly once every two days and needs ~27 days; a once-per-hour step is fully measured in one. The ratio is 0.9 rather than 1.0 because window bounds are observed session spans, so two nominally identical windows differ by hours. When either window does not carry both bounds, width is not computable, the flip is not excused, and the run records that the check could not be made — an unreadable window must be visible, never a silent exemption.

FR-5 — Snapshot accumulation and diffing

FR-5.1 Every run appends one machine-readable snapshot record to an append-only store on this plugin's own Gitea wiki (schema in PLAN §6).

FR-5.2 Candidates carry a stable identity across runs so that "3 of 5 prior recommendations are still unaddressed" is computable. Identity resolution order: (1) an open/closed Gitea issue number, once filed — the strongest and preferred identity; (2) the normalized tool-sequence signature; (3) the (skill, slug) pair assigned at first sighting.

FR-5.3 A run compares against the previous snapshot for the same target and reports per-candidate status: new, persisting, grown, shrunk, resolved, signature-drifted, stale (target version changed such that the skill no longer exists).

FR-5.4 Trend comparison must normalize for usage volume. Absolute token deltas across windows of different activity are meaningless — a quiet week reads as an improvement. Primary trend metrics are cost per invocation and share of the plugin's audited spend; absolute tokens are reported as context only.

FR-5.5 A candidate is resolved only when its issue is closed and the signature's measured cost has dropped below threshold in a later window with at least N invocations. Issue-closed-without-measurement is reported as claimed-fixed-unconfirmed. Committed ≠ verified, in the same spirit as this environment's nixos-deploy-pending discipline.

FR-6 — Output path A: issues on the target repo

FR-6.1 Each high-confidence candidate above the value threshold is filed as a tracked issue on the target plugin's own repo, by delegating to anxious:issuer (which shapes and dispatches; cluster:gitea-agent performs the write). This plugin never writes to Gitea directly.

FR-6.2 Follow the existing environment conventions rather than inventing a scheme: full owner/repo#num references everywhere; labels from the anxious four-axis taxonomy (kind/chore or kind/capability-gap, area/agent-behavior, domain/agents, activity/automate), plus one new cross-cutting label token-offload. Filing is implicit — the audit does not ask permission to file, consistent with the global bug-handling rule.

FR-6.3 Idempotency is mandatory. Each issue body carries a marker line <!-- ia-candidate: <candidate_id> -->. Before filing, search the target repo for that marker; if found, comment the updated measurement on the existing issue instead of opening a duplicate. An on-demand audit that is re-run must not spam the tracker — this is the single most likely way for the plugin to become hated.

FR-6.4 If the target plugin has no writable repo (a third-party or official marketplace plugin), skip path A entirely, record filing: unavailable with the reason, and report the candidate in the summary only.

FR-6.5 A finding about the audit itself — currently only the FR-4.5 instability downgrade — is filed against this plugin's own repo (kotkan/claude-plugin-inference-arbitrage), never the target's. The target did not change; the auditor described it two ways, so the target's tracker is the wrong place to put it and filing it there would blame the audited plugin for the auditor's defect. These findings obey FR-6.3's idempotency rule with their own marker, <!-- ia-stability: <candidate_id> -->, kept distinct from ia-candidate so that a self-finding and a target finding for the same candidate_id cannot collide in a marker search. Delegation is unchanged: this plugin never writes to Gitea directly.

FR-7 — Output path B: the run summary

FR-7.1 Each run writes a human-readable summary page to this plugin's own Gitea wiki at Audits/<target-plugin>/<YYYY-MM-DD>, and updates Audits/<target-plugin>/Latest.

FR-7.2 Rationale for wiki-over-artifact, since the brief asks for one: the summary must be durable, addressable across sessions, diffable by git, and readable by a future agent that was not present for the run. A Gitea wiki page is all four — the wiki is a git repo, so the snapshot JSONL and the prose report version together and a git log shows the audit history for free. An Artifact is a private, session-produced web page with no cross-run addressing and no relationship to the repo; it cannot be the system of record. An Artifact is offered as an optional rendered view — a trend chart across snapshots — when a human explicitly asks to look at the history, generated from the same snapshot data. Canonical store: wiki. Optional presentation: artifact.

FR-7.3 The summary states, in this order: attribution coverage and window; the composition headline (how much of the plugin's spend was mechanical); the ranked candidate table with boundary classification and value; the diff against the previous snapshot; the boundary questions posed to the user — both the low-confidence candidates and any downgraded by the FR-4.5 stability gate, the latter marked as a defect in the auditor rather than a fact about the target; any candidate the gate could not judge either way (insufficient-window, FR-4.5.1), stated as a window too narrow to compare rather than as a finding about the target or the auditor; and the issues filed or updated, in full owner/repo#num form, separating those on the target's tracker (FR-6.1) from those on this plugin's own (FR-6.5).

Shipped as of v0.5.0, with one gap named rather than papered over: the analyst's spoken report (skills/offload-audit §7) makes both distinctions, but the rendered wiki page does not yet. audit-snapshot's summary renderer lists all boundary questions uniformly and shows only each candidate's own issue, because snapshot_candidate does not carry the stability field and stability findings never enter the snapshot at all — stability-classify emits them alongside it, which is why memory-notes takes --stable as a separate input (FR-9.1). Closing this means widening the snapshot schema, not editing a renderer; until then a self-filed finding reaches the page only via --note.

FR-8 — Self-application

FR-8.1 The plugin must pass its own audit. Its static pass is a script, its transcript scan is a script, its snapshot diff is a script; the agent's job is classification, judgment, and reporting. If any of these migrate into skill prose executed by inference, the plugin is violating its own thesis.

FR-8.2 Auditing inference-arbitrage with inference-arbitrage is an acceptance test (S7, TASKS 7.7).

FR-8.3 The audit's delegation topology is bounded and fixed. One audit is one unit of work: the whole audit may be handed to a single offload-analyst instance, and that hand-off happens at most once, from the top-level command or skill. The analyst is the delegation target, not a further decision point — it never spawns another offload-analyst, and never one agent per skill or per candidate. Its only outbound Agent calls are the fixed filing chain of FR-6.1 (anxious:issuer → cluster:gitea-agent).

This is a hard requirement and not a style note, because it failed in exactly the predictable way. The command instructs a caller to delegate a heavy target to the analyst; nothing told the analyst that it was that delegation target, so on a large plugin it re-applied the same heaviness criterion to itself and spawned another instance, three to four levels deep, each restarting the same measurement steps with no forward progress. Size is a reason to work through a target methodically, never a reason to hand it to a fresh copy of yourself with no memory of what has already been measured.

The rule is load-bearing twice over: it is the S8 runtime constraint (PLAN §7 — no subagent fan-out, on a 16 GiB laptop), and it is FR-8.1 self-application, since unbounded self-delegation is precisely the shape of waste this plugin exists to find. A token-optimization plugin recursing on itself is the most expensive bug it could possibly have.

FR-9 — Memory integration

The wiki (FR-7) is this plugin's system of record; mempalace is where an audit becomes context for the next one. Two distinct purposes, and they need different placement.

FR-9.1 — The bridge is a script, exactly like FR-7's. audit-snapshot memory-queries emits the searches to run before grading; audit-snapshot memory-notes emits the {wing, room, content, source_file, added_by} payloads to write after the run, plus the knowledge-graph triples. Neither makes an MCP call: filesystem in, JSON out. The caller passes both verbatim to search and add_drawer / kg_add.

This is FR-8.1 applied to this feature. Which candidates get written about is already decided by the classify and stability verdicts, both deterministic; what text gets written is a total function of the snapshot. A plugin that narrated its own bookkeeping at inference time would be filing an issue against itself.

FR-9.2 — Reading is advisory, never a gate. Prior drawers are context the analyst may cite. They may never override bin/boundary-classify (FR-4.2) or bin/stability-classify (FR-4.5), and are never a reason to file a candidate that did not clear the gates, to skip the FR-6.3 marker search, or to re-grade a verdict already returned. Memory says what was thought before; the gates decide what happens now. A remembered "we declined this last time" is a reason to look harder at the reasoning, not to short-circuit it — and a remembered "we filed this last time" is not a filing decision at all, because the tracker, not memory, is the FR-5.2 identity of record.

Note honestly what kind of requirement this is, clause by clause — the mix is not uniform, and calling the whole of FR-9.2 unenforced would be as wrong as calling it a gate.

Three of its four prohibitions are mechanical, and recall cannot reach them:

• file something the gates rejected — bin/filing-plan emits action: file only for verdict == "file", and skips everything else with the verdict named in the reason;
• skip the FR-6.3 marker search — filing-plan plan makes --existing required precisely so that a search which never ran cannot default to "nothing exists";
• re-grade a returned verdict — the verdict is boundary-classify's output, and stability-classify rewrites rather than re-asks it.

What is left unenforced is the one clause that matters most and is the hardest to watch: what the analyst writes into the judgments JSON that those gates then consume. The gates are pure functions of their input, so recall cannot corrupt a verdict directly — it can only corrupt the evidence handed to them, by softening a determinism test or supplying an overrule case remembered rather than reasoned. No script can see that, because the whole value of the read is that it influences judgment; a read that could not move the judgments JSON would be a read with no purpose.

An audit trail is therefore the only available control, and it is a real obligation, not a hope: a run that leans on a recalled conclusion says so in its report, so a human can see the lean and discount it. That is why FR-9.2 is written as prohibitions rather than thresholds.

FR-9.3 — Two placements, for two different readers.

	Wing	Room	Read	Write
Per-target findings	the audited plugin's manifest name	inference-arbitrage-audits	yes	yes
Cross-target judgment	claude-plugins	inference-arbitrage-lessons	yes	yes

Target-scoped mirrors FR-6.1's philosophy — the finding belongs on the target's own surface, not this plugin's — so a human or agent later working on that plugin meets the offload findings while searching their own project's memory. Two drawers per run: the per-run summary, and the qualitative judgment calls (declined candidates and why, stability downgrades).

Cross-target is this plugin's own accumulated judgment — calibration outcomes, coverage-dilution norms, the shapes that keep turning out to be false positives — in the wing this plugin's own development history already lives in. It is written on every run regardless of target and regardless of outcome; a "nothing to offload here" result is exactly the calibration evidence worth keeping. It is read at the start of every audit, independent of which plugin is being audited.

Room names are namespaced by this plugin's name, deliberately. A wing named for a plugin can collide with an unrelated wing that already means something else: cluster (the k3s cluster) and imagex (corp Jira/Slack/Bitbucket work) both exist today and mean neither the cluster plugin nor the imagex plugin. Namespacing the room turns such a collision into a shelving overlap rather than a content collision — a room-filtered search returns only audit drawers, and every drawer opens with a line naming what wrote it. Dumping into a generic decisions / gotchas room would have made the collision real.

FR-9.4 — Redaction applies unchanged. FR-3.5 governs a drawer exactly as it governs an issue body or a wiki page: shapes, counts and prose about the audit only. Never raw transcript content, file contents, command arguments, or user prose.

The memory renderers therefore do not reuse the wiki renderers. A wiki page lives in this plugin's own repo; a drawer lands in a wing shared with unrelated projects, and snapshot.notes is caller-supplied free text (--note) that reaches the wiki page legitimately. The memory renderers read from a fixed allowlist of structured snapshot fields — ids, positions, booleans, counts, ratios — so no free-text field crosses into a drawer at all. Allowlist, never blocklist: a blocklist is one new snapshot field away from leaking.

FR-9.5 — The untestable seam is one call wide. The MCP calls cannot be exercised by the shell test harness, and are not faked there. That is precisely why all the judgment lives in the script: the caller's remaining job is a verbatim tool call, so the part that cannot be tested is the part with nothing in it to get wrong. tests/memory.test.sh covers wing/room resolution, query packing, drawer selection, rendered content and redaction; the live round trip is verified by hand (TASKS 7.9.5).

---

4. Success criteria

#	Criterion	How it is verified
S1	Audits any plugin by path or name, with no target-specific code	Run against token-budget, worktree-discipline, anxious, memory
S2	Correctly returns "nothing to offload" for a well-cut plugin	Run against token-budget → no high-confidence candidates
S3	Rediscovers a known-correct historical cut	Run against worktree-discipline with bin/worktree-audit removed from the inventory → must flag the classification step as high confidence
S4	Mechanically refuses a candidate whose evidence is a window artifact	The two-window anxious fixture → the FR-4.5 gate refuses agent-wip/release-policy-derivation (as insufficient-window, per FR-4.5.1, since the 7-day current window cannot fairly contradict the 26-day measured one), with no human in the loop; a flip between comparably wide windows is still downgraded to a boundary question
S5	Re-running does not duplicate issues	Run twice; second run comments, does not file
S6	Trend across ≥2 snapshots is computable and volume-normalized	Two runs on different windows; diff reports per-invocation deltas
S7	Passes its own audit	FR-8.2
S8	Runs within emmett's constraints	Single stdlib process, streaming, no subagent fan-out and no self-delegation (FR-8.3), no toolchain build

S3 is the important one. It is the only criterion that tests the rubric against a cut known to be correct, made by a human, for a reason that is written down.

S4 was originally "finds a real candidate in anxious that the user agrees is genuine." That is retired: a human triaging one candidate and pronouncing it good is not a repeatable test, and it puts the plugin's central safety property behind whether someone was feeling skeptical that day. What actually happened is that a human review caught a candidate whose evidence flipped between windows — so the criterion is now that gate, run mechanically (FR-4.5), against the real two-window evidence that produced the original finding.

The reason S4's candidate is refused changed in v0.7.0, and the change is itself instructive. Two further real windows (14 and 60 days, same script, same candidate) showed 0 → 2 → 12 → 15 invocations as the window widened: monotonic accumulation of a low-frequency step, not a label flipping about. The 7-day window had not caught an unstable candidate, it had failed to observe a real one, and calling that instability blamed the candidate for the auditor's window. So FR-4.5.1 makes the refusal honest — insufficient-window, cannot tell either way, compare again over a wider window — while keeping the property S4 exists to test: nothing is filed off a window artifact, with no human in the loop. (kotkan/claude-plugin-inference-arbitrage#15.)

---

5. Signal catalog

5.1 Static signals — what the definition says

These are computed by bin/plugin-inventory and consumed as JSON.

Signal	Definition	Reads as
Procedural density	Fraction of a skill body that is numbered/imperative steps containing literal commands, vs. discursive prose	A recipe the model re-derives every invocation
Script-verb count	Occurrences of count, sum, rank, sort, diff, parse, validate, enumerate, dedupe, format, check exists, compare	Mechanical intent stated in prose
Judgment-verb count	Occurrences of decide, judge, explain, prioritize, name, write, weigh, interpret, for this user	Irreducible intent stated in prose
Verb ratio	script-verbs / (script-verbs + judgment-verbs) per skill	>0.6 with no bin/ script is the loudest static smell
Duplicated command blocks	Identical or near-identical fenced command blocks appearing in ≥2 skills	A shared script that was never written
Rule tables	Markdown tables that are pure lookup (input → output, finding → action)	A dispatch table being narrated at inference time
Script coverage	executable LOC in bin/ + hooks/ vs. total skill+agent body words	Structural balance; low ratio + high verb ratio is the target profile
Hook/prose drift	Logic present both in a hook script and restated in skill prose	Two sources of truth; the prose will rot
Tool allowlist shape	An agent whose tools are dominated by read-only mechanical tools	The agent was built to fetch, not to judge

Worked example — token-budget (expected: clean). skills/token-audit has high procedural density (it prescribes three exact commands in order) and a rule table (finding → remediation skill). But script coverage is high (606 LOC of cc-tokens against ~3 skill bodies), the verb ratio in the body is judgment-dominated ("report", "interpret", "state the caveat", "rank the fixes by effect"), and — decisively — the prescribed commands are invocations of a script that already exists. The static pass should score this as already cut, with the residual candidate being at most "the fixed three-command opening sequence could be one cc-tokens audit subcommand," classified low value.

Worked example — anxious (expected: real candidates). agents/steward.md sweeps every oleks/* repo and reconciles labels, milestones, and board placement. The taxonomy it reconciles against is a pure lookup table with a documented deterministic ruleset (references/taxonomy.md axis 3 is explicitly described as "steward-owned, repo-derived (deterministic)", and axis 1's derivation is a four-step priority list ending in "defer to human"). A deterministic ruleset, written down, applied per repo by an LLM turn, over N repos, is the canonical shape: the derivation is a script, the "defer to human" branch and the axis-2/axis-4 weak priors are judgment. Expected classification: llm-over-script-digest — a script computes proposed labels and emits only the disagreements and the no-signal cases for the model to rule on.

5.2 Dynamic signals — what the transcripts show

Computed by bin/offload-scan over ~/.claude/projects/**/*.jsonl, filtered to turns attributed to the target.

5.2.1 The primary metric — Mechanical Turn Ratio

A turn is mechanical if all of:

• it contains ≥1 tool_use block, and every one is drawn from the mechanical tool set (read-only Bash, Read, Grep, Glob, list_* / get_* / *_read MCP tools, ToolSearch), and
• its output_tokens are below a threshold (default 400) — the model emitted a tool call and little else, and
• it is not immediately preceded by a tool error (that is a retry, counted separately under 5.2.4).

MTR(skill)             = mechanical_turns / attributed_turns
mechanical_tokens      = Σ over mechanical turns of (cache_read + cache_write + input + output)
offload_waste(skill)   = mechanical_tokens          # weighted, via cc-tokens' tier normalization

offload_waste is the honest number: tokens spent purely to keep the model in the loop on steps that produced no judgment. Because cache_read dominates, it is very nearly mechanical_turns × mean_context_size — which is exactly why turn count, not verbosity, is the lever.

An MTR above ~0.5 on a skill carrying meaningful spend is the primary flag. A high MTR on a skill with trivial spend is noise and is not reported.

5.2.2 Repetition signature — finding the algorithm in the wild

Normalize every tool call to a signature: tool name plus argument shape, with volatile literals masked — paths → <path>, integers → <n>, hex/SHA → <sha>, URLs → <url>, quoted free text → <str>. (This masking is also what satisfies the privacy requirement FR-3.5.)

Within each invocation of a skill, extract the ordered signature sequence. Across invocations, find n-grams of length ≥3 that recur in ≥3 distinct invocations.

A frequently-recurring identical tool sequence is an algorithm, empirically discovered. This is the most direct possible answer to the user's ask, because it does not rely on reading intent out of prose — it observes the procedure being executed the same way repeatedly and prices it. Each recurring n-gram is reported with its occurrence count and its measured offload_waste, and becomes a candidate with a ready-made proposed script boundary.

5.2.3 Read amplification

read_amplification = tokens_pulled_into_context_by_read_tools
                     / tokens_of_that_material_referenced_in_subsequent_output

Approximated by comparing tool-result sizes to the model's next-turn output length and its literal overlap with the result. High amplification means the model is reading a haystack to find a needle — a filter/digest script belongs upstream. This is the signature of the llm-over-script-digest position.

5.2.4 Retry / self-correction density

Fraction of attributed turns that follow a tool error, or that repeat a near-identical signature with adjusted arguments. Every retry is a full context re-read. High density means the invocation is fiddly — the correct offload is often a thin wrapper script that gets the invocation right once, rather than a wholesale algorithm move. Cheap to build, immediately effective.

5.2.5 Judgment density — the brake

Fraction of attributed turns with substantive output_tokens (above threshold) and no tool_use, or containing AskUserQuestion. This is where the model was actually thinking.

Judgment density is a brake, not an accelerator. High judgment density with high spend means the plugin is doing what it should and must be reported as healthy. The offload target is specifically low judgment density × high spend.

5.2.6 Fan-out multiplier

Sidechain turns attributed to the target. Each subagent carries its own full context, so a mechanical step performed inside a subagent costs a multiple of the same step inline. A mechanical n-gram executing inside a fan-out is the highest-value candidate class there is — and this environment already has a recorded finding against one-subagent-per-tiny-step fan-out.

5.2.7 Composite ranking

offload_value = offload_waste × (1 − judgment_density) × repetition_factor

where repetition_factor = 1 + log₂(recurrences of the dominant n-gram). Reported in weighted tokens and as a share of the plugin's audited spend.

Measurement strength. Before the value threshold can be applied, the evidence must be strong enough for the share to mean anything. Each candidate carries a measurement_strength:

Value	When
measured	≥3 invocations and ≥30 attributed turns
thin	attributed, but too few invocations — the share is arithmetically real and evidentially worthless
unmeasured	no attributed usage at all (the FR-2.4 zero-history case)

Filing threshold. boundary_confidence == high and the falsifiability triple was produced (including the overrule case) and the FR-4.5 stability gate passed. Given those, the value test depends on measurement strength:

• measured → offload_value must also be ≥2% of the audited window's total spend. Below it, the candidate is a boundary question.
• thin / unmeasured → the value test cannot be applied and is not applied. The candidate files on its static case alone, and the issue body must state that the value claim is unproven. FR-2.4 requires a plugin with no transcript history to stay auditable, and silently withholding a correctness-complete candidate because it happens to be unmeasured would make every brand-new plugin audit return nothing.

That asymmetry is deliberate and is the one place a candidate reaches the tracker without a cost number behind it. It is safe only because the correctness gates — the five determinism tests, the overrule case — are unconditional; what varies is whether the plugin can also say how much it is worth. Conversely the value threshold is deliberately conservative where it does apply: the cost of a missed candidate is a slightly expensive plugin; the cost of a wrong candidate is a plugin that confidently does the wrong thing, forever, silently.

Two verdicts sit outside this ladder entirely. A candidate classified pure-inference (position 4) is reported as no-offload — an explicit finding that the step is correctly done by inference, per FR-4.4, not a silent omission. A hook/prose drift signal (§5.1) is reported as a drift-note: it is a restated configuration contract with two sources of truth, and its fix is to point the prose at the hook file, not to write a new script.

---

6. Risks and open questions

These want the user's input before or during the build.

Q1 — Cross-plugin dependency on cc-tokens. The dynamic pass should not reimplement token-budget's accounting (FR-3.2), but a plugin cannot rely on another plugin's ${CLAUDE_PLUGIN_ROOT}. Options: (a) require cc-tokens on PATH and fail loudly with an install hint; (b) glob ~/.claude/plugins/cache/*/token-budget/*/bin/cc-tokens and pick the newest; (c) vendor a copy. Recommendation: (a) with (b) as fallback, and never (c) — a vendored copy silently diverges on exactly the two traps that are hardest to notice. Better still: contribute cc-tokens attribute --by skill|plugin --json upstream to token-budget, so the money math has one home. Resolved (TASKS 0.1): (a) with (b) as fallback, never (c). bin/offload-scan tries PATH, then the cache glob, then fails loudly with an install hint. The upstream contribution remains optional (TASKS 8.3).

Q2 — Attribution coverage may be thin. attributionSkill appears only when a Skill was formally invoked. Hook-driven work, agent work started without a Skill call, and plugin logic that runs as part of a larger session may carry no attribution at all. A 60-transcript sample surfaced ~1100 attributed lines across a dozen skills — real, but not obviously complete. If coverage for a given target is low, the audit's dynamic pass is weak and must say so loudly (FR-3.4). Open: is there a better attribution path for agent-driven work than agentName?

Q3 — Read access to a target's repo. The audit reads the local plugin cache, which needs no repo access. But FR-6 files issues on the target's repo, which does. For oleks/* plugins this is fine. For third-party or official-marketplace plugins there is no writable tracker, and the audit degrades to summary-only (FR-6.4). Confirm that degradation is acceptable rather than an error. Resolved (TASKS 0.2): yes — filing: unavailable with the reason recorded, and the run summary still written.

Q4 — False positives are the real failure mode. Recommending a script where judgment is needed produces something worse than the status quo: a fast, confident, wrong answer with no one watching. The rubric's five determinism tests, the mandatory "when would a human overrule this" element, and the conservative filing threshold are the mitigations, and the plugin never implements its own recommendations. The residual risk is that a plausible-looking candidate gets built by a later session that does not re-check the boundary — which is why the issue body must carry the overrule case, not just the proposal.

Q5 — Goodhart. A plugin that scores plugins on token cost will, if followed blindly, push toward brittle over-scripting. Proposed counter-metric tracked on the trend page: the rate of token-offload issues later closed as wontfix, and of offload scripts subsequently reverted. If that rate climbs, the rubric is too loose. Open: is this worth building in v1, or noted as a manual check?

Q6 — Privacy of the summary surface. Snapshots and wiki pages are derived from transcripts that contain secrets, private prose, and corp work. FR-3.5's masking is the control. Worth confirming the masking list is sufficient, and whether audits of imagex:* (corp) plugins should be excluded from the shared wiki entirely.

Q7 — Local compute budget. Scanning a multi-gigabyte transcript history on emmett must be a single streaming stdlib process, never a fan-out of one subagent per skill. Stated as a hard constraint in PLAN §7, flagged here so it is a conscious decision rather than an accident.

Q8 — Rubric versioning. Snapshots record rubric_version. When the rubric changes, old candidate classifications are not directly comparable. Proposal: diffs across a rubric-version boundary are annotated, not suppressed. Confirm.

---

7. Incidental finding

While surveying the plugin cache for this spec, two leaked git worktrees were found inside the installed plugin cache itself:

~/.claude/plugins/cache/oleks-local/worktree-discipline/1.12.0/.claude/worktrees/wf_fdb5df0f-811-2/
~/.claude/plugins/cache/oleks-local/worktree-discipline/1.12.0/.claude/worktrees/wf_fdb5df0f-811-3/

Each carries a full copy of the plugin tree. These are unnamed-Workflow-subagent worktrees (wf_*) that leaked into a cache directory — a location sweep-worktrees is unlikely to be pointed at, and one that a plugin reinstall would silently orphan. Filed as kotkan/claude-plugin-worktree-discipline#9; not in scope for this spec.