From c4389d8bb140d3b9ef9f8711bfaf33f828ebff76 Mon Sep 17 00:00:00 2001 From: Oleks Date: Thu, 30 Jul 2026 19:27:09 +0300 Subject: [PATCH] Add bin/candidate-digest: script the inv.json/scan.json join MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The offload-analyst agent was hand-reshaping bin/plugin-inventory's PluginInventory and bin/offload-scan's OffloadScan into a candidate table on every audit run — 40.09% of this plugin's own audited spend over 26 invocations (kotkan/claude-plugin-inference-arbitrage#24). candidate_digest(inv, scan) -> list[CandidateRow] joins the two JSON documents and computes shape: share_of_spend, mtr, judgment_density, read_amplification, retry_density, fanout_multiplier, dominant_ngram_recurrences, has_bin_script, and an advisory `flags` list. It reuses boundary-classify's MIN_INVOCATIONS/ MIN_ATTRIBUTED_TURNS and ia_store's MIN_SHARE_OF_SPEND rather than reinventing thresholds, and applies the judgment_density brake (new JUDGMENT_DENSITY_BRAKE=0.3) so a row with real judgment density never gets flagged even if every mechanical threshold is crossed. This is a digest, not an auto-filing script (position 3, never position 1): flags are advisory input to the analyst's own reading of the actual n-gram/tool-call shape, never a verdict on their own. bin/boundary-classify remains the only place a verdict is computed. Wired into skills/offload-audit/SKILL.md step 3 and agents/offload-analyst.md section 2 so future audit runs call the script instead of reshaping by hand. Tests: tests/candidate-digest.test.sh covers the issue's three worked examples (scripted skill suppressed via has_bin_script, a flagged agent row, and the judgment_density-brake-suppressed edge case) plus a below-MIN_SHARE_OF_SPEND case and a full CandidateRow schema check. Also verified against a real matched inv.json/scan.json pair from a prior anxious audit in /tmp. --- .claude-plugin/plugin.json | 2 +- agents/offload-analyst.md | 59 +++++++--- bin/candidate-digest | 198 +++++++++++++++++++++++++++++++++ skills/offload-audit/SKILL.md | 37 ++++-- tests/candidate-digest.test.sh | 163 +++++++++++++++++++++++++++ 5 files changed, 434 insertions(+), 25 deletions(-) create mode 100755 bin/candidate-digest create mode 100755 tests/candidate-digest.test.sh diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index 05f43a1..55eee9d 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "inference-arbitrage", - "version": "0.7.0", + "version": "0.8.0", "description": "Audits a Claude Code plugin's definitions and usage transcripts to find steps that should be a deterministic script instead of raw LLM inference, and files the well-evidenced ones as issues on the target repo.", "author": { "name": "oleks" diff --git a/agents/offload-analyst.md b/agents/offload-analyst.md index c72213a..f89ee9a 100644 --- a/agents/offload-analyst.md +++ b/agents/offload-analyst.md @@ -50,7 +50,9 @@ ${CLAUDE_PLUGIN_ROOT}/bin/offload-scan --plugin --days N --json # dynami - **Never implement your own recommendations.** You propose; a human or a separate session builds. - **Never count, parse, or aggregate by hand.** If you find yourself tallying - something, a `bin/` tool should be doing it. + something, a `bin/` tool should be doing it — including reshaping `inv.json` + and `scan.json` into a candidate table, which is now `bin/candidate-digest` + (kotkan/claude-plugin-inference-arbitrage#24), not something you do inline. - **Never grade by hand.** Judge the five tests, write the triple, then run `bin/boundary-classify`. @@ -129,22 +131,47 @@ re-proposed as if new. ### 2. Gather candidates from three distinct sources -**a. Static smells** (`plugin-inventory`). The loudest is a high verb ratio -**with no `bin/` script behind it** — treat that as a conjunction, never the -ratio alone. `token-budget` has verb ratios of 0.667 and 0.714 and is perfectly -cut, because every command its skills prescribe is an invocation of a 606-line -script that already exists. Also: `duplicated_command_blocks` (a shared script -nobody wrote), `rule_tables` (a dispatch table being narrated at inference time), -low `script_coverage`. +**Never hand-reshape `inv.json` and `scan.json` into a candidate table.** That +was this plugin's own worst offender — 40% of its audited spend +(kotkan/claude-plugin-inference-arbitrage#24) — and it is now a script: -**b. Dynamic smells** (`offload-scan`). High `mtr` with meaningful spend; a -recurring `ngram` (an algorithm observed in the wild, which is the strongest -evidence there is); high `read_amplification` (a digest belongs upstream); high -`retry_density` (often just wants a thin wrapper that gets the invocation right -once); `fanout_multiplier > 1` (mechanical work inside a subagent costs a -multiple). **`judgment_density` is a brake, not an accelerator** — high judgment -density with high spend means the plugin is doing what it should, and must be -reported as **healthy**. +```bash +${CLAUDE_PLUGIN_ROOT}/bin/candidate-digest inv.json scan.json --json +``` + +It returns one `CandidateRow` per skill/agent with the shape fields below plus +`has_bin_script` and `flags`. **It is a digest, not a verdict** (position 3): +`flags` marks a row that crosses the same thresholds `boundary-classify` +already applies — `MIN_INVOCATIONS`, `MIN_ATTRIBUTED_TURNS`, +`MIN_SHARE_OF_SPEND` — plus the `judgment_density` brake. A flagged row is +still yours to read, not yours to auto-file: the digest can only see +token/turn shape, not intent, and a row can cross every mechanical threshold +yet turn out to be this audit's own necessary two-pass discipline (read the +static inventory, then the dynamic scan, then reshape) rather than a +genuinely wasteful loop. That overrule case is exactly why the cut sits here +and not in an auto-filing script. + +**a. Static smells** (`plugin-inventory`, `has_bin_script` in the digest). The +loudest is a high verb ratio **with no `bin/` script behind it** — treat that +as a conjunction, never the ratio alone. `token-budget` has verb ratios of +0.667 and 0.714 and is perfectly cut, because every command its skills +prescribe is an invocation of a 606-line script that already exists — the +digest's `has_bin_script: true` suppresses exactly this shape. Also read +directly from `plugin-inventory` (the digest does not carry these): +`duplicated_command_blocks` (a shared script nobody wrote), `rule_tables` (a +dispatch table being narrated at inference time), low `script_coverage`. + +**b. Dynamic smells** (`offload-scan`, most already surfaced in the digest). +High `mtr` with meaningful spend; a recurring `ngram` +(`dominant_ngram_recurrences` — an algorithm observed in the wild, which is +the strongest evidence there is); high `read_amplification` (a digest belongs +upstream); high `retry_density` (often just wants a thin wrapper that gets the +invocation right once); `fanout_multiplier > 1` (mechanical work inside a +subagent costs a multiple). **`judgment_density` is a brake, not an +accelerator** — high judgment density with high spend means the plugin is +doing what it should, and must be reported as **healthy**; the digest's +`flags` already encode this (a row with `judgment_density >= 0.3` is never +flagged), but confirm it by eye on anything you file. **c. Hook/prose drift — its own category, framed differently.** Entries in `aggregate.hook_prose_drift` are **restated configuration contracts, not diff --git a/bin/candidate-digest b/bin/candidate-digest new file mode 100755 index 0000000..26a5ef6 --- /dev/null +++ b/bin/candidate-digest @@ -0,0 +1,198 @@ +#!/usr/bin/env python3 +"""Joins bin/plugin-inventory's PluginInventory and bin/offload-scan's +OffloadScan into a candidate table -- the reshaping the offload-analyst agent +used to do by hand on every single audit run (40.09% of this plugin's own +audited spend, kotkan/claude-plugin-inference-arbitrage#24). + +This is a DIGEST, not an auto-filing script: position 3 of the rubric +(references/boundary-rubric.md), never position 1. It computes SHAPE -- which +rows cross the codified mechanical thresholds -- and leaves intent to a +human/the analyst. A row can cross every mechanical threshold (share, mtr, +read-amplification, a recurring n-gram) and still turn out to be this audit's +own necessary two-pass discipline (read the static inventory, then the +dynamic scan, then reshape both) rather than a genuinely wasteful loop. The +digest cannot see intent, only shape, so `flags` is advisory input to a +human's judgment call -- never a verdict, and never wired to filing. +`bin/boundary-classify` remains the only place a verdict is computed; this +script does not import it for grading, only for its two evidence-strength +constants (see below). + +Usage: candidate-digest [--json] +""" +import argparse +import importlib.machinery +import importlib.util +import json +import os +import sys + +BIN_DIR = os.path.dirname(os.path.abspath(__file__)) + +# ia_store.py has no hyphen, so a normal import works once bin/ is on +# sys.path -- which it already is (sys.path[0]) when this file is run as +# `bin/candidate-digest`, per ia_store.py's own docstring. +from ia_store import MIN_SHARE_OF_SPEND, scan_rows # noqa: E402 + + +def _load_bin_module(filename): + """boundary-classify has a hyphenated filename and can't be `import`ed by + name -- loaded the same way offload-scan loads token-budget's cc-tokens + (resolve_cc_tokens there), so this plugin has exactly one pattern for + "import a sibling bin/ script as a module".""" + path = os.path.join(BIN_DIR, filename) + mod_name = filename.replace("-", "_") + spec = importlib.util.spec_from_loader( + mod_name, importlib.machinery.SourceFileLoader(mod_name, path)) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +_boundary_classify = _load_bin_module("boundary-classify") +# Reused, not reinvented (kotkan/claude-plugin-inference-arbitrage#24 says to +# reuse these): the same bar boundary-classify already applies to call a +# candidate's evidence `measured` rather than `thin`. +MIN_INVOCATIONS = _boundary_classify.MIN_INVOCATIONS +MIN_ATTRIBUTED_TURNS = _boundary_classify.MIN_ATTRIBUTED_TURNS + +# New in this script -- no existing catalog threshold covers "how much +# judgment_density is enough to call a recurring shape genuine work rather +# than waste". The issue's own worked example put a real non-flag case at +# judgment_density=0.55 and a real flag case at 0.034; 0.3 sits well clear of +# both and reads naturally as "under a third of this row's turns were +# judgment calls". This is the judgment_density BRAKE +# (agents/offload-analyst.md: "a brake, not an accelerator") -- it can only +# suppress a flag, never add one. +JUDGMENT_DENSITY_BRAKE = 0.3 + + +def _has_bin_script(name, kind, inv): + """True only when plugin-inventory's OWN static evidence shows this row's + documented procedure already delegates to a bin/ script. + + Skills carry `command_blocks` (the fenced code in their SKILL.md); a + block whose first line names a `bin/` path means the skill is *already* + a thin wrapper around an existing script -- the + "verb_ratio=0.71 backed by a 606-LOC script" example from the issue, + where the conjunction (high verb ratio AND no script) fails and nothing + should be surfaced. + + Agents carry no body or command-block text in plugin-inventory's JSON at + all -- only skills do -- so this is conservatively False for every agent + row. That is a real limit on what this script can see, not a judgment + call it is dodging: an analyst who knows a given agent already delegates + to bin/ overrides this field by hand, same as any other digest output. + """ + if kind != "skill": + return False + for skill in inv.get("skills", []): + if skill.get("name") == name: + return any("bin/" in b.get("sig", "") + for b in skill.get("command_blocks", [])) + return False + + +def _dominant_ngram_recurrences(row): + ngrams = row.get("ngrams") or [] + if not ngrams: + return None + return max(g.get("recurrences", 0) for g in ngrams) + + +def _flags(share, invocations, turns, judgment_density, dominant_ngram, + has_bin_script): + """The escalation path documented in kotkan/claude-plugin-inference- + arbitrage#24: flag a row when it crosses the same thresholds + boundary-classify already uses for evidence strength and filing share + (MIN_INVOCATIONS, MIN_ATTRIBUTED_TURNS, MIN_SHARE_OF_SPEND), shows a + dominant recurring n-gram, AND judgment_density sits below the brake. + + `judgment_density` is a brake, not an accelerator: a row that crosses + every other threshold but carries real judgment density is the healthy + case (this audit's own two-pass read-then-reshape discipline, not a + wasteful loop) and MUST come back with flags=[] -- that is the low_judgment + check below, and it is the whole reason this is a digest and not an + auto-filing script. + + A row already backed by a bin/ script (has_bin_script) is never flagged + at all: whatever mechanical share it carries is already offloaded, so + there is nothing left here to propose. + """ + if has_bin_script: + return [] + measured = invocations >= MIN_INVOCATIONS and turns >= MIN_ATTRIBUTED_TURNS + significant_share = share >= MIN_SHARE_OF_SPEND + recurring = dominant_ngram is not None and dominant_ngram > 1 + low_judgment = judgment_density < JUDGMENT_DENSITY_BRAKE + flags = [] + if measured and significant_share and recurring and low_judgment: + flags.append("high-share-low-judgment-recurring-ngram") + return flags + + +def candidate_digest(inv, scan): + """PluginInventory, OffloadScan -> list[CandidateRow] + (kotkan/claude-plugin-inference-arbitrage#24). + + Pure function of its two already-computed JSON documents -- no + filesystem access beyond what the caller already read, no network, no + re-derivation of anything plugin-inventory or offload-scan already + computed. This is the entire join the offload-analyst agent used to + reshape by hand every audit run. + + CandidateRow = {row_kind: 'skill'|'agent', name: str, + share_of_spend: float, mtr: float, judgment_density: float, + read_amplification: float, retry_density: float, + fanout_multiplier: float, dominant_ngram_recurrences: int|None, + has_bin_script: bool, flags: list[str]} + """ + rows = [] + for name, row in scan_rows(scan): + # scan_rows() (ia_store.py) yields every scan["skills"] row keyed by + # row["skill"] and every scan["agents"] row keyed by row["agent"]; + # the row itself still carries whichever of those two keys it came + # from, which is the only place kind is recoverable from here. + kind = "agent" if "agent" in row else "skill" + has_bin = _has_bin_script(name, kind, inv) + dominant = _dominant_ngram_recurrences(row) + share = row.get("share_of_plugin", 0.0) + judgment_density = row.get("judgment_density", 0.0) + rows.append({ + "row_kind": kind, + "name": name, + "share_of_spend": share, + "mtr": row.get("mtr", 0.0), + "judgment_density": judgment_density, + "read_amplification": row.get("read_amplification", 0.0), + "retry_density": row.get("retry_density", 0.0), + "fanout_multiplier": row.get("fanout_multiplier", 1.0), + "dominant_ngram_recurrences": dominant, + "has_bin_script": has_bin, + "flags": _flags(share, row.get("invocations", 0), + row.get("turns", 0), judgment_density, + dominant, has_bin), + }) + rows.sort(key=lambda r: -r["share_of_spend"]) + return rows + + +def main(): + ap = argparse.ArgumentParser(prog="candidate-digest") + ap.add_argument("inventory", + help="path to plugin-inventory --json output, or - for stdin") + ap.add_argument("scan", help="path to offload-scan --json output") + ap.add_argument("--json", action="store_true", help="compact single-line JSON") + args = ap.parse_args() + + with (sys.stdin if args.inventory == "-" else open(args.inventory)) as fh: + inv = json.load(fh) + with open(args.scan) as fh: + scan = json.load(fh) + + rows = candidate_digest(inv, scan) + json.dump(rows, sys.stdout, indent=None if args.json else 2) + sys.stdout.write("\n") + + +if __name__ == "__main__": + main() diff --git a/skills/offload-audit/SKILL.md b/skills/offload-audit/SKILL.md index 32b877c..1b4b7f4 100644 --- a/skills/offload-audit/SKILL.md +++ b/skills/offload-audit/SKILL.md @@ -78,15 +78,36 @@ with `--note` so the wiki page records that this run is missing from memory. ## 3. Gather candidates -From `plugin-inventory`: high verb ratio **with no `bin/` script behind it** -(a conjunction — never the ratio alone), duplicated command blocks, rule tables, -low script coverage. From `offload-scan`: high MTR with real spend, recurring -n-grams, read amplification, retry density, fan-out. `judgment_density` is a -**brake** — high judgment density with high spend means the plugin is healthy -and must be reported that way. +**Never hand-reshape `inv.json` and `scan.json` into a candidate table.** +That reshaping used to be done by inference on every single audit run and was +itself 40% of this plugin's own audited spend +(kotkan/claude-plugin-inference-arbitrage#24). Run the join instead: -`aggregate.hook_prose_drift` entries are **not offload candidates.** Pass them -with `"category": "hook-prose-drift"` so they come back as `drift-note`. +```bash +$IA/bin/candidate-digest inv.json scan.json --json > digest.json +``` + +This produces one `CandidateRow` per skill/agent (`row_kind`, `name`, +`share_of_spend`, `mtr`, `judgment_density`, `read_amplification`, +`retry_density`, `fanout_multiplier`, `dominant_ngram_recurrences`, +`has_bin_script`, `flags`). It is a **digest, not a verdict** — position 3, the +same as this whole plugin's own thesis. A `flags` entry is a shape the digest +noticed crossing the same thresholds `boundary-classify` already applies +(`MIN_INVOCATIONS`, `MIN_ATTRIBUTED_TURNS`, `MIN_SHARE_OF_SPEND`) plus the +`judgment_density` brake; it is **never** grounds to skip step 4's judgment or +to file directly. Read every flagged row's actual n-gram/tool-call shape before +deciding candidate vs. this audit's own necessary two-pass discipline — the +overrule case `bin/boundary-classify`'s hard gate (§4 below) exists for. + +Also read `plugin-inventory`'s own fields the digest does not carry: duplicated +command blocks, rule tables, low script coverage. `judgment_density` is a +**brake, not an accelerator** — a flagged row with real judgment density +(`>= 0.3`) comes back unflagged, and high judgment density with high spend +means the plugin is healthy and must be reported that way. + +`aggregate.hook_prose_drift` entries are **not offload candidates** and never +appear in `digest.json`. Pass them with `"category": "hook-prose-drift"` so +they come back as `drift-note`. ## 4. Judge, then grade mechanically diff --git a/tests/candidate-digest.test.sh b/tests/candidate-digest.test.sh new file mode 100755 index 0000000..0a41390 --- /dev/null +++ b/tests/candidate-digest.test.sh @@ -0,0 +1,163 @@ +#!/usr/bin/env bash +# bin/candidate-digest: the inv.json/scan.json join +# (kotkan/claude-plugin-inference-arbitrage#24). Synthetic fixtures for the +# three worked examples from the issue body, plus the has_bin_script suppression. +set -euo pipefail +cd "$(dirname "${BASH_SOURCE[0]}")" + +BIN=../bin/candidate-digest +tmp=$(mktemp -d) +trap 'rm -rf "$tmp"' EXIT +fail=0 + +field() { + python3 -c "import json,sys;print(json.load(open(sys.argv[1]))[int(sys.argv[2])][sys.argv[3]])" "$@" +} + +check() { + local what=$1 got=$2 want=$3 + if [ "$got" = "$want" ]; then + echo " ok $what = $want" + else + echo " FAIL $what = $got (want $want)" >&2 + fail=1 + fi +} + +# --------------------------------------------------------------------------- +# Example 1: verb_ratio=0.71 skill backed by a bin/ script -> has_bin_script +# True, flags=[] regardless of its dynamic numbers (conjunction fails). +# Example 2: agent row, mtr=0.17 share=0.40 judgment_density=0.034 with a +# dominant recurring 8-gram -> flags=['high-share-low-judgment-recurring-ngram']. +# Example 3 (EDGE): same mtr/share/ngram but judgment_density=0.55 -> flags=[] +# (the judgment_density brake suppresses the false positive). +# --------------------------------------------------------------------------- + +cat >"$tmp/inv.json" <<'EOF' +{ + "plugin": {"name": "synthetic"}, + "skills": [ + { + "name": "syn:scripted-skill", + "verb_ratio": 0.71, + "command_blocks": [{"sig": "${CLAUDE_PLUGIN_ROOT}/bin/do-the-thing --json", "count": 1}] + } + ] +} +EOF + +cat >"$tmp/scan.json" <<'EOF' +{ + "skills": [ + { + "skill": "syn:scripted-skill", + "invocations": 12, + "turns": 90, + "share_of_plugin": 0.5, + "mtr": 0.8, + "judgment_density": 0.05, + "read_amplification": 10.0, + "retry_density": 0.0, + "fanout_multiplier": 1.0, + "ngrams": [{"sig": ["a", "b", "c"], "recurrences": 9, "invocations": 5, "waste": 1000}] + } + ], + "agents": [ + { + "agent": "syn:flagged-agent", + "invocations": 26, + "turns": 208, + "share_of_plugin": 0.40, + "mtr": 0.17, + "judgment_density": 0.034, + "read_amplification": 4.0, + "retry_density": 0.02, + "fanout_multiplier": 1.0, + "ngrams": [{"sig": ["x", "y"], "recurrences": 8, "invocations": 6, "waste": 800000}] + }, + { + "agent": "syn:healthy-agent", + "invocations": 26, + "turns": 208, + "share_of_plugin": 0.40, + "mtr": 0.17, + "judgment_density": 0.55, + "read_amplification": 4.0, + "retry_density": 0.02, + "fanout_multiplier": 1.0, + "ngrams": [{"sig": ["x", "y"], "recurrences": 8, "invocations": 6, "waste": 800000}] + } + ] +} +EOF + +$BIN "$tmp/inv.json" "$tmp/scan.json" >"$tmp/out.json" + +echo "== example 1: scripted skill, high mtr/share, has_bin_script suppresses everything ==" +row=$(python3 -c " +import json +rows = json.load(open('$tmp/out.json')) +print([i for i, r in enumerate(rows) if r['name'] == 'syn:scripted-skill'][0]) +") +check "row_kind" "$(field "$tmp/out.json" "$row" row_kind)" "skill" +check "has_bin_script" "$(field "$tmp/out.json" "$row" has_bin_script)" "True" +check "flags" "$(field "$tmp/out.json" "$row" flags)" "[]" + +echo "== example 2: agent, judgment_density=0.034 -> flagged ==" +row=$(python3 -c " +import json +rows = json.load(open('$tmp/out.json')) +print([i for i, r in enumerate(rows) if r['name'] == 'syn:flagged-agent'][0]) +") +check "row_kind" "$(field "$tmp/out.json" "$row" row_kind)" "agent" +check "has_bin_script" "$(field "$tmp/out.json" "$row" has_bin_script)" "False" +check "dominant_ngram_recurrences" "$(field "$tmp/out.json" "$row" dominant_ngram_recurrences)" "8" +check "flags" "$(field "$tmp/out.json" "$row" flags)" "['high-share-low-judgment-recurring-ngram']" + +echo "== example 3 (EDGE): same mtr/share/ngram, judgment_density=0.55 -> brake suppresses ==" +row=$(python3 -c " +import json +rows = json.load(open('$tmp/out.json')) +print([i for i, r in enumerate(rows) if r['name'] == 'syn:healthy-agent'][0]) +") +check "flags" "$(field "$tmp/out.json" "$row" flags)" "[]" + +echo "== below MIN_SHARE_OF_SPEND -> never flagged even with a low judgment_density ==" +cat >"$tmp/scan-thin.json" <<'EOF' +{ + "skills": [], + "agents": [ + { + "agent": "syn:thin-agent", + "invocations": 12, + "turns": 90, + "share_of_plugin": 0.005, + "mtr": 0.9, + "judgment_density": 0.01, + "read_amplification": 3.0, + "retry_density": 0.0, + "fanout_multiplier": 1.0, + "ngrams": [{"sig": ["p", "q"], "recurrences": 5, "invocations": 4, "waste": 100}] + } + ] +} +EOF +cat >"$tmp/inv-empty.json" <<'EOF' +{"plugin": {"name": "synthetic"}, "skills": []} +EOF +$BIN "$tmp/inv-empty.json" "$tmp/scan-thin.json" >"$tmp/out-thin.json" +check "flags" "$(field "$tmp/out-thin.json" 0 flags)" "[]" + +echo "== schema: every CandidateRow key present ==" +python3 -c " +import json +rows = json.load(open('$tmp/out.json')) +want = {'row_kind', 'name', 'share_of_spend', 'mtr', 'judgment_density', + 'read_amplification', 'retry_density', 'fanout_multiplier', + 'dominant_ngram_recurrences', 'has_bin_script', 'flags'} +for r in rows: + assert set(r.keys()) == want, (r.keys(), want) +print('ok') +" + +exit "$fail"