kotkan/claude-plugin-inference-arbitrage

Files

T

Oleks d34118ec9a Scaffold plugin: manifest, license, design docs

2026-07-29 15:32:34 +03:00

20 KiB

Raw Permalink Blame History

plan.md — `inference-arbitrage` architecture

Implements spec.md, classifying by rubric.md. Structure follows the plugin-publishing skill's layout and the worktree-discipline / anxious precedent (skills + agent + bin/ scripts + references/ + tests/ + a Woodpecker pipeline).

1. Naming

inference-arbitrage.

The name is deliberate and echoes builder-arbitrage, which already exists in this environment and means: route each unit of work to the cheapest executor capable of doing it correctly. That is exactly this plugin's thesis, with the executor choice being "script vs. model" instead of "which build node." Reusing the vocabulary means the concept arrives already understood.

Rejected alternatives: script-offload (describes the mechanism, not the judgment, and the judgment is the hard part), token-optimizer (collides conceptually with token-budget and implies it optimizes tokens directly, which it does not — it optimizes where work executes). The placeholder directory name token-offload-audit remains available if the user prefers it; nothing in the design depends on the choice.

1.1 Workspace conventions this must satisfy

From ~/projects/claude-plugins/CLAUDE.md — these are mechanical rules, not preferences:

Directory name must equal the manifest name. True for every plugin in the workspace. If inference-arbitrage is accepted, the checkout is renamed from token-offload-audit at scaffold time (TASKS 1.1).
Gitea repo name is exactly claude-plugin-<manifest-name> — no exceptions, including stuttering cases.
The org is NOT derivable from the name. Both oleks and kotkan hold plugins. kotkan already holds token-budget, worktree-discipline, and anti-patterns — the three plugins this one is most closely related to — which is the argument for putting it there. Decision 0.5 in tasks.md.
Remote protocol is mixed (SSH vs HTTPS) and signals nothing; copy whatever the sibling repos use rather than normalizing.
Branch main; .gitignore = .cache/, .claude/, *.log.
bin/ scripts must keep the executable bit (git ls-files -s → 100755). This plugin ships three of them, so it is a live concern.
A global pre-push hook auto-formats markdown/JSON and may append a style: auto-format from pre-push hooks commit during push — re-check git status and ahead/behind afterwards and push again if it committed.
Version bumps are mandatory for any behavioural change; Claude Code caches by version.

Install (once published): claude plugin add https://claude-plugins.oleks.space/plugins/inference-arbitrage.git

Note that publishing is two independent actions: adding to the local oleks-local dev marketplace (~/projects/claude-plugins/.claude-plugin/marketplace.json, "source": "./inference-arbitrage") and publishing to the public index (~/projects/claude-plugin-index/plugins.json, a separate repo). A plugin can be in one and not the other, and several here are deliberately local-only. Local-dev registration first is the right order — it allows @oleks-local testing through Phase 7 before anything becomes publicly installable.

2. Directory layout

inference-arbitrage/
├── .claude-plugin/
│   └── plugin.json
├── agents/
│   └── offload-analyst.md          # inference-arbitrage:offload-analyst-agent
├── skills/
│   ├── offload-audit/SKILL.md      # run a full audit on a target plugin
│   ├── boundary-rubric/SKILL.md    # classify ONE step; usable standalone
│   └── offload-trend/SKILL.md      # diff vs history; chase stale recommendations
├── commands/
│   └── offload-audit.md            # /offload-audit <plugin> [--days N]
├── bin/
│   ├── plugin-inventory            # static pass    → JSON
│   ├── offload-scan                # dynamic pass   → JSON
│   └── audit-snapshot              # persist / load / diff snapshots
├── references/
│   ├── boundary-rubric.md          # rubric.md, verbatim
│   ├── signals-catalog.md          # spec.md §5, expanded with thresholds
│   ├── snapshot-schema.md          # §6 below
│   └── issue-template.md           # the issue body contract
├── tests/
│   ├── run-all.sh
│   ├── inventory.test.sh
│   ├── scan.test.sh                # against synthetic fixture transcripts
│   ├── snapshot.test.sh
│   └── fixtures/
├── .woodpecker/test.yaml
├── README.md  LICENSE  .gitignore  .markdownlint.json  .shellcheckrc

2.1 The division of labour, stated as a constraint

Per FR-8, the plugin must obey its own thesis. The split is:

Layer	Owns	Never does
`bin/*`	All parsing, aggregation, metric computation, snapshot diffing, signature normalization	Classify a candidate; decide what to file
`agents/offload-analyst.md`	Rubric application, boundary questions, ranking narrative, deciding what crosses the filing threshold	Parse a transcript; count anything by hand
`skills/*`	Procedure, thresholds, reporting contract, delegation protocol	Restate logic that lives in a script

If a future change puts a loop or an aggregation into skill prose, that is a regression, and tests/ should be able to catch the coarse version of it (a skill body containing a fenced multi-command pipeline that duplicates a bin/ capability).

3. `bin/plugin-inventory` — the static pass

Language: Python 3, stdlib only, offline. (Consistent with cc-tokens; no npm, no PyYAML — a minimal frontmatter parser for the --- block is ~30 lines and avoids a dependency.)

plugin-inventory <path|name> [--json]

Resolution: a path is used directly; a bare name globs ~/.claude/plugins/cache/*/<name>/*/, filters out directories containing .orphaned_at, and picks the highest semver.

Output: one JSON document.

{
  "plugin": {"name": "...", "version": "...", "source": "oleks-local", "path": "..."},
  "skills": [{
    "name": "sweep-worktrees",
    "description_words": 61,
    "body_words": 1840,
    "allowed_tools": ["Bash", "Read"],
    "procedural_density": 0.42,      // steps-with-commands / total blocks
    "script_verbs": 14, "judgment_verbs": 9, "verb_ratio": 0.61,
    "command_blocks": [{"sig": "git worktree list --porcelain", "count": 3}],
    "rule_tables": 2,
    "references": ["extraction-notes.md"]
  }],
  "agents": [{"name": "...", "model": "sonnet", "tools": [...],
              "body_words": 900, "mechanical_tool_share": 0.8}],
  "hooks":   [{"file": "...", "event": "PreToolUse", "loc": 120}],
  "scripts": [{"file": "bin/worktree-audit", "loc": 430, "lang": "bash"}],
  "aggregate": {
    "skill_body_words": 6200, "script_loc": 980,
    "script_coverage": 0.158,
    "duplicated_command_blocks": [{"sig": "...", "skills": ["a", "b"]}],
    "hook_prose_drift": [...]
  }
}

Verb lexicons live in references/signals-catalog.md, not hardcoded, so they can be tuned without a code change — and so a tuning is a reviewable diff.

Notably, plugin-inventory is itself the proof of FR-8: reading a plugin's structure is a pure-script step by every one of the five determinism tests, and having a model do it by reading files would be the exact anti-pattern the plugin reports on.

4. `bin/offload-scan` — the dynamic pass

offload-scan --plugin <name> [--days N | --since D --until D] [--json]
             [--root ~/.claude/projects]

4.1 Pipeline

Stream ~/.claude/projects/**/*.jsonl line by line. Never load a file whole; histories are multi-gigabyte and this runs on a 16 GiB laptop.
Filter to assistant lines whose attributionPlugin == <name>, plus sidechain lines whose agentName matches one of the target's agents.
De-duplicate turns by (message.id, requestId), taking max(output_tokens) — the two documented transcript traps. See §4.3: this is delegated, not reimplemented.
Reconstruct invocations: contiguous runs of attributed turns within one sessionId, split on a gap in attributionSkill or a configurable idle gap.
Classify each turn as mechanical / judgment / retry per spec §5.2.
Normalize tool signatures and mine recurring n-grams (spec §5.2.2).
Aggregate per skill and per agent; compute MTR, offload_waste, read amplification, retry density, judgment density, fan-out multiplier, offload_value.
Compute attribution coverage (FR-3.4): attributed turns vs. turns in sessions where the target was active at all.
Emit JSON. Masked signatures and counts only — never raw content (FR-3.5).

4.2 Output shape

{
  "window": {"since": "...", "until": "...", "sessions": 34},
  "coverage": {"attributed_turns": 812, "candidate_turns": 1104, "ratio": 0.735,
               "caveat": "hook-driven and unattributed agent turns excluded"},
  "totals": {"weighted_tokens": 41200000, "cost_estimate_usd": 118.4},
  "skills": [{
    "skill": "anxious:kanban-flow",
    "invocations": 22, "turns": 190,
    "weighted_tokens": 9100000, "share_of_plugin": 0.22,
    "mtr": 0.68, "judgment_density": 0.11,
    "offload_waste": 6900000, "read_amplification": 14.2, "retry_density": 0.09,
    "fanout_multiplier": 1.0,
    "ngrams": [{"sig": ["list_issues(<str>)", "issue_read(<n>)", "label_read(<n>)"],
                "recurrences": 17, "waste": 3100000}],
    "offload_value": 8400000
  }],
  "agents": [ ... same shape ... ]
}

4.3 The `cc-tokens` dependency (open question Q1)

The pricing table, tier weighting, and the two de-duplication traps must have exactly one home. Reimplementing them here is how the numbers silently diverge — the traps are specifically the kind that produce plausible wrong answers.

Resolution order at runtime:

cc-tokens on PATH → shell out with --json for money numbers.
Otherwise glob ~/.claude/plugins/cache/*/token-budget/*/bin/cc-tokens, newest semver.
Otherwise fail loudly with an install hint. Never vendor, never approximate.

offload-scan performs its own pass for the things cc-tokens does not model — tool-sequence signatures, turn classification, n-gram mining — but takes token and cost arithmetic from cc-tokens.

Preferred long-term fix, to be proposed to the user: contribute cc-tokens attribute --by skill|plugin --json upstream to token-budget. The attribution fields are already in the transcripts; cc-tokens is already the right place for that arithmetic; and it would make offload-scan a thin consumer. Filed as a task, not assumed.

5. `bin/audit-snapshot` — persistence and diffing

audit-snapshot write   --target <name> --from <inventory.json> <scan.json> --classified <c.json>
audit-snapshot list    --target <name>
audit-snapshot diff    --target <name> [--against <run_id>]

Storage. The wiki repo of <org>/claude-plugin-inference-arbitrage, cloned to a local cache and pushed via cluster:gitea-agent (this plugin never writes Gitea directly).

Data/<target>/snapshots.jsonl      # append-only, one JSON object per run
Audits/<target>/<YYYY-MM-DD>.md    # human-readable run summary
Audits/<target>/Latest.md          # rollup + open candidate table
Methodology/Rubric.md              # versioned copy; snapshots pin rubric_version
Methodology/Calibration.md         # threshold changes, with the evidence for each

Append-only JSONL in a git-backed wiki gives durable history, a free audit trail via git log, and diffability, with no database. Latest.md is regenerated, not appended.

Candidate identity (FR-5.2) resolves in this order — issue number first, because it is the only identifier a human also uses:

Gitea issue number, once filed.
Normalized tool-sequence signature hash.
(skill, slug) assigned at first sighting.

resolved requires issue-closed and measured cost below threshold with ≥N invocations in a later window; issue-closed alone reports claimed-fixed-unconfirmed (FR-5.5).

6. Snapshot schema

{
  "run_id": "2026-07-29T14:02:11Z",
  "rubric_version": "1.0.0",
  "tool_versions": {"inference-arbitrage": "0.1.0", "cc-tokens": "0.1.0"},
  "target": {"name": "anxious", "version": "0.31.0", "source": "oleks-local",
             "repo": "oleks/claude-plugin-anxious"},
  "window": {"since": "2026-07-22", "until": "2026-07-29", "days": 7},
  "coverage": {"ratio": 0.735, "attributed_turns": 812},
  "totals": {"weighted_tokens": 41200000, "share_mechanical": 0.61},
  "candidates": [{
    "candidate_id": "anxious/steward/label-derivation",
    "signature_hash": "b41f…",
    "skill": "anxious:steward-agent",
    "position": "llm-over-script-digest",
    "boundary_confidence": "high",
    "determinism_tests": {"T1": true, "T2": true, "T3": true, "T4": true, "T5": true},
    "falsifiability": {"signature": "...", "examples": 3, "overrule_case": "..."},
    "measurement": {"invocations": 22, "offload_waste": 6900000,
                    "offload_value": 8400000, "share_of_plugin": 0.22,
                    "cost_per_invocation": 381818},
    "digest_schema": "…what crosses the boundary…",
    "escalation_path": "…what happens on surprise…",
    "issue": "oleks/claude-plugin-anxious#41",
    "status": "persisting"
  }],
  "boundary_questions": [ { …medium/low candidates, same shape, no issue… } ],
  "notes": []
}

7. Runtime constraints

Hard constraints from the environment, not preferences:

Single streaming stdlib process. No fan-out of one subagent per skill — that violates both the emmett heavy-workload rule and the recorded subagent-fan-out-cost finding, and would be an especially embarrassing way for a token-optimization plugin to burn tokens.
No compile, test, or lint toolchain locally. CI verification goes through Woodpecker (.woodpecker/test.yaml), per the global rule.
Bounded memory. Line-by-line JSONL streaming; aggregate state is O(skills × distinct signatures), not O(turns).
Offline. No network in either pass. Only the two output paths touch the network, and both go through delegation.

8. Skills

`offload-audit` — the main procedure

Triggers: "audit this plugin for token optimization", "can any of this be a script", "offload analysis", "where is this plugin wasting inference", "check for scripting opportunities".

Procedure: resolve target → plugin-inventory → offload-scan → apply the rubric to each candidate (delegating to boundary-rubric for the per-candidate classification) → threshold → file path A → write path B → report.

Carries the filing threshold, the "nothing to offload is a valid answer" rule, and the coverage-caveat reporting contract.

`boundary-rubric` — classify one step

Triggers: "should this be a script or LLM", "where does the boundary go", "is this judgment or algorithm", "can I automate this step".

Usable standalone, outside an audit, on a step someone is about to build. This is deliberate: the rubric is more valuable applied before the code exists than after, and a standalone skill lets it be reached during ordinary design work.

Body carries the five tests, the four positions, the positioning questions, and the falsifiability triple. references/boundary-rubric.md carries the full text with worked examples.

`offload-trend` — history and follow-through

Triggers: "has this plugin got worse", "offload trend", "are my recommendations being acted on", "diff the last audit", "what did we say last time".

Runs audit-snapshot diff, reports the volume-normalized trend, lists recommendations still unaddressed with their issue refs in full owner/repo#num form, and promotes any low/medium boundary question that has recurred with growing cost.

9. Agent

inference-arbitrage:offload-analyst-agent — model: sonnet (the work is structured classification over pre-computed JSON, not open synthesis; if calibration proves weak on the boundary questions, revisit).

Tools: Bash, Read, Grep, Glob, Skill, Agent (for the anxious:issuer delegation only).

Read-only with respect to the target. It never edits a plugin it audits and never implements its own recommendations — that separation is what preserves its ability to conclude "no offload here."

Behavioral rules carried in the agent body:

Lead with attribution coverage and the mechanical-share headline. A candidate list without coverage context is misleading.
Name skills, never UUIDs — borrowed directly from usage-agent's reporting discipline, which exists for the same reason.
Rank by measured value and say so; a user who acts only on item one should have captured most of the benefit.
"Nothing to offload here" is a complete and valuable answer. Do not manufacture candidates to look thorough.
Never file without the overrule case (rubric §5).

10. Output path A — filing, in detail

Per FR-6, delegation only:

offload-analyst  →  Agent(anxious:issuer-agent)  →  cluster:gitea-agent  →  Gitea

The analyst formulates; issuer decides repo, title, labels, milestone; gitea-agent writes. This plugin holds no Gitea credentials and calls no Gitea MCP tool.

Issue body contract (references/issue-template.md):

<!-- ia-candidate: anxious/steward/label-derivation -->

**Measured cost.** 8.4M weighted tokens over 22 invocations (2026-07-22 → 07-29),
22% of this plugin's audited spend. MTR 0.68, judgment density 0.11.
Recurring sequence observed 17×: list_issues → issue_read → label_read.

**Boundary position.** llm-over-script-digest — confidence high.
Determinism tests: T1 ✓ T2 ✓ T3 ✓ T4 ✓ T5 ✓

**Proposed signature.**
`derive_labels(repo: str, issue: Issue) -> tuple[Labels, list[Ambiguity]]`

**Examples.** (3, including one edge case) …

**When a human would overrule this.** …
**Escalation path.** …
**What crosses the boundary.** …

Audit: `Audits/anxious/2026-07-29` on <org>/claude-plugin-inference-arbitrage wiki.
Rubric v1.0.0.

Labels: token-offload (new, cross-cutting), plus the anxious four-axis taxonomy as issuer determines — typically kind/chore or kind/capability-gap, area/agent-behavior, domain/agents, activity/automate.

Before filing, search the target repo for the ia-candidate marker; if present, comment the updated measurement instead of opening a duplicate (FR-6.3).

11. Publishing

Per the plugin-publishing skill, and per §1.1 above:

Source tree at ~/projects/claude-plugins/<name>/ (rename from token-offload-audit once the name is settled). Note the workspace root is deliberately not a git repo — commit inside the plugin subdir, never at the top level.
git init -b main, the workspace .gitignore, commit.
Create <org>/claude-plugin-<name> on Gitea with no auto-init, push.
Optionally register in the local oleks-local marketplace for @oleks-local testing before going public — recommended for this plugin, since Phase 7 acceptance wants real runs before anyone can install it.
Add the entry to ~/projects/claude-plugin-index/plugins.json (separate repo) and push; the Gitea webhook refreshes the registry, which also acts as the git smart-HTTP proxy that makes claude plugin add work.
Verify at claude-plugins.oleks.space/api/plugins, then claude plugin add.

Every content change requires a version bump — the cache at ~/.claude/plugins/cache/<source>/<name>/<version>/ is keyed on version and a same-version push is silently ignored.

Also create the wiki on the plugin's repo (path B depends on it) and the token-offload label on each target repo as it is first audited (via issuer, implicitly).

20 KiB Raw Permalink Blame History Unescape Escape

plan.md — inference-arbitrage architecture