claude-plugin-inference-arbitrage/design/tasks.md

tasks.md — build order for inference-arbitrage

Ordered so that each phase is independently verifiable and the risky part (the rubric) is calibrated against known-answer fixtures before anything is filed to a real tracker.

Track as Gitea issues on <org>/claude-plugin-inference-arbitrage under a v0.1.0 milestone, via anxious:issuer. Reference in full owner/repo#num form.

---

Phase 0 — Decisions to close first

These block design, not just code. Answers go into Methodology/Calibration.md.

• 0.1 Resolve Q1: the cc-tokens dependency. Resolved as option (a) + (b): bin/offload-scan locates cc-tokens on PATH first, then falls back to globbing ~/.claude/plugins/cache/*/token-budget/*/bin/cc-tokens, and fails loudly with an install hint if neither resolves. Never vendored. Upstreaming cc-tokens attribute --by skill|plugin --json stays optional (8.3).
• 0.2 Resolve Q3: confirm summary-only degradation is acceptable for plugins with no writable repo (official/third-party marketplace). Resolved: yes — shipped as FR-6.4 / filing: unavailable (6.5).
• 0.3 Resolve Q6: confirm the masking list, and decide whether corp (imagex:*) plugins are excluded from the shared wiki entirely.
• 0.4 Resolve Q5: build the Goodhart counter-metric (wontfix/revert rate) in v1, or note it as a manual check?
• 0.5 Confirm the plugin name inference-arbitrage (vs. keeping the placeholder token-offload-audit). Resolved: inference-arbitrage; directory renamed to match the manifest name.
• 0.6 Choose the Gitea org: oleks or kotkan. Resolved: kotkan — kotkan/claude-plugin-inference-arbitrage, alongside token-budget, worktree-discipline, and anti-patterns.

Phase 1 — Repo scaffold

• 1.1 Rename ~/projects/claude-plugins/token-offload-audit/ to match the confirmed manifest name (workspace convention: directory == manifest name), and build out the PLAN §2 layout: plugin.json v0.1.0, MIT LICENSE, .gitignore with .cache/, .claude/, *.log. The four spec files move with the directory and stay in-tree as the design record.
• 1.2 git init -b main inside the plugin subdir (the workspace root is deliberately not a repo), create <org>/claude-plugin-<name> on Gitea with no auto-init (via cluster:gitea-agent), push main.
• 1.2b Verify bin/ scripts carry the executable bit (git ls-files -s → 100755) — three shipped scripts make this a live concern. Re-check git status after every push: the global pre-push hook auto-formats markdown/JSON and may append its own commit.
• 1.2c Register in the local oleks-local marketplace (~/projects/claude-plugins/.claude-plugin/marketplace.json, "source": "./<name>") so Phases 2–7 can test via @oleks-local without publishing anything publicly.
• 1.3 Initialize its wiki with Methodology/Rubric.md (rubric.md verbatim, v1.0.0) and empty Audits/ + Data/ scaffolding.
• 1.4 .woodpecker/test.yaml running tests/run-all.sh. CI verifies; nothing heavy runs on emmett.

Phase 2 — Static pass

• 2.1 bin/plugin-inventory: target resolution (path or name, semver pick, skip .orphaned_at), frontmatter parsing (~30 lines, no PyYAML).
• 2.2 Signal extraction: procedural density, verb ratio, command-block signatures, rule tables, script coverage, hook/prose drift.
• 2.3 references/signals-catalog.md with the verb lexicons and thresholds, loaded at runtime — tuning must be a reviewable diff.
• 2.4 tests/inventory.test.sh against a fixture plugin tree.
• 2.5 Calibration run against token-budget, worktree-discipline, anxious, memory. Record the actual numbers in Methodology/Calibration.md and tune thresholds until the ordering matches the human read: token-budget cleanest, anxious richest in candidates.

Phase 3 — Dynamic pass

• 3.1 bin/offload-scan skeleton: streaming JSONL reader, attribution filter, (message.id, requestId) de-dup — delegating token/cost arithmetic to cc-tokens per decision 0.1. Do not reimplement the pricing table or the streaming-snapshot max.
• 3.2 Invocation reconstruction (contiguous attributed runs per session).
• 3.3 Turn classification: mechanical / judgment / retry, with the mechanical tool set and the output-token threshold in the signals catalog.
• 3.4 Tool-signature normalization and masking (also satisfies FR-3.5).
• 3.5 N-gram mining: recurring sequences ≥3 long in ≥3 invocations.
• 3.6 Metrics: MTR, offload_waste, read amplification, retry density, judgment density, fan-out multiplier, composite offload_value.
• 3.7 Attribution coverage computation (FR-3.4).
• 3.8 tests/scan.test.sh against synthetic fixture transcripts with hand-computed expected metrics. Do not test against real transcripts — they contain private content and they change.
• 3.9 Verify runtime and memory on the real history: one process, streaming, bounded. Record wall time in the calibration page.

Phase 4 — Rubric and classification

• 4.1 references/boundary-rubric.md = rubric.md verbatim.
• 4.2 skills/boundary-rubric/SKILL.md — usable standalone on a single step, before any code exists.
• 4.3 agents/offload-analyst.md with the PLAN §9 behavioral rules, including the hard gate: no overrule case → boundary question, never filed.
• 4.4 Calibration against known answers — the load-bearing test. - token-budget → must yield zero high-confidence candidates (S2). - worktree-discipline with bin/worktree-audit masked out of the inventory → must flag worktree classification as high (S3). - the two-window anxious fixture → must downgrade release-policy-derivation on the stability gate (S4). If either fails, the rubric is wrong and Phase 5 does not start.
• 4.5 Record the calibration outcome and any threshold change, with its evidence, in Methodology/Calibration.md.

Phase 5 — Snapshots and trend

• 5.1 bin/audit-snapshot write|list|diff; schema per PLAN §6.
• 5.2 Candidate identity resolution (issue → signature → slug) and signature-drift detection.
• 5.3 Volume-normalized diffing: cost-per-invocation and share-of-spend primary, absolute tokens as context only (FR-5.4).
• 5.4 resolved vs claimed-fixed-unconfirmed logic (FR-5.5).
• 5.5 Wiki read/write; Latest.md regeneration. Resolved as: the script is offline and filesystem-only, mirroring the wiki tree into a local store; audit-snapshot pages emits what to publish and the caller (skill, or cluster:gitea-agent) performs the wiki writes, and audit-snapshot import seeds a cold cache from a wiki-fetched snapshots.jsonl. See the bin/audit-snapshot docstring.
• 5.6 skills/offload-trend/SKILL.md.
• 5.7 tests/snapshot.test.sh — two synthetic snapshots, assert the diff.

Phase 6 — Output paths

• 6.1 references/issue-template.md per PLAN §10.
• 6.2 anxious:issuer delegation path; create the token-offload label on a target repo on first audit (implicitly, via issuer).
• 6.3 Idempotency (FR-6.3) — ia-candidate marker search before filing; comment on the existing issue instead of duplicating. Verify by running the same audit twice (S5). This is the highest-consequence correctness detail in the whole plugin.
• 6.4 Summary page generation → Audits/<target>/<date> + Latest.
• 6.5 Graceful filing: unavailable for repo-less targets (FR-6.4).
• 6.6 skills/offload-audit/SKILL.md and commands/offload-audit.md tying the whole procedure together.

Phase 7 — Acceptance

• 7.1 S1 — audits four different plugins with no target-specific code.
• 7.2 S2 — token-budget → "nothing to offload."
• 7.3 S3 — masked worktree-discipline → rediscovers the known cut.
• 7.4 S4 — the FR-4.5 evidence-stability gate mechanically catches a real unstable candidate in anxious (agent-wip/release-policy-derivation) without human review. The original framing — "a candidate the user agrees is genuine" — is retired: a human triage catching a bad candidate is not a repeatable test, and the one that ran caught this candidate because its evidence flipped across windows. That check is now bin/stability-classify, asserted against the real two-window evidence in tests/stability.test.sh case (f).
• 7.5 S5 — double-run files no duplicate.
• 7.6 S6 — two-snapshot trend is computable and volume-normalized.
• 7.7 S7 — audit inference-arbitrage with inference-arbitrage (FR-8.2). Any parsing or aggregation found living in skill prose is a bug to fix before release.
• 7.8 S8 — confirm the run stays inside emmett's constraints.

Phase 7.9 — Memory integration (FR-9)

Numbered 7.9 rather than 8.x because it lands before publish, by explicit user decision on 2026-07-29 ("build it now, even if it delays publish"): a v0.1.0 that ships without it would publish an audit loop that forgets every run.

• 7.9.1 audit-snapshot memory-queries — the read side. Deterministic queries over both wings, packed to mempalace's 250-char limit, resolvable from the inventory alone so the read step does not depend on a pass that has not run yet.
• 7.9.2 audit-snapshot memory-notes — the write side. Per-run summary, qualitative judgment calls, cross-target lessons, plus knowledge-graph triples. Same contract as pages: filesystem in, JSON out, no MCP call.
• 7.9.3 Redaction by allowlist (FR-9.4). Memory renderers do not reuse the wiki renderers, because snapshot.notes is caller-supplied free text that legitimately reaches a wiki page and must never reach a drawer.
• 7.9.4 Wire into skills/offload-audit/SKILL.md (step 2b read, step 6b write), agents/offload-analyst.md (procedure §1b and §7), and the agent's tools: frontmatter (mcp__plugin_memory_*).
• 7.9.5 Live round trip. Verified by hand 2026-07-29 against the real anxious snapshot: add_drawer into anxious / inference-arbitrage-audits and into claude-plugins / inference-arbitrage-lessons, both retrieved by the script's own emitted queries; kg_add of anxious → audited_by → inference-arbitrage. The MCP calls are not covered by tests/memory.test.sh and are not faked there — see FR-9.5 on why the untestable seam is one verbatim call wide.
• 7.9.6 tests/memory.test.sh — wing/room resolution, query packing, drawer selection per run shape, rendered content, redaction, kg facts.

Phase 8 — Publish

• 8.1 README with the thesis, the rubric summary, and worked examples.
• 8.2 Publish per plugin-publishing, scoped to local use: pushed to main, already registered in oleks-local (1.2c). Deliberately not added to ~/projects/claude-plugin-index/plugins.json or verified on claude-plugins.oleks.space — user's explicit call, since this plugin is used only through oleks-local. Revisit if that changes.
• 8.3 Optional, from decision 0.1: PR cc-tokens attribute --by skill|plugin to token-budget.
• 8.4 Memory checkpoint — the rubric and the calibration findings into claude-plugins, room inference-arbitrage-lessons. Written against wing_claude_memory / working-practice before FR-9 existed; that predates the wing/room split FR-9.3 actually shipped with, and having two destinations for the same cross-target judgment defeats the point of a single accumulating room. Use the one bin/audit-snapshot memory-notes already writes to, so a manual checkpoint and an automated one land in the same place.

---

Dependency notes

• Phase 3 depends on decision 0.1.
• Phase 4.4 gates Phase 5 and 6. Do not file a single real issue until the rubric passes calibration — the plugin's credibility is spent on its first wrong recommendation.
• Phase 2 and Phase 3 are otherwise independent and can proceed in parallel; both feed Phase 4.
• The static pass alone (Phases 1–2 + 4) is already a shippable v0.1.0 if the dynamic pass proves harder than expected. It answers "what could be a script" without "and what is it costing you" — less compelling, but useful, and it degrades honestly by marking every candidate unmeasured (FR-2.4).