Add bin/candidate-digest: script the inv.json/scan.json join
ci/woodpecker/push/test Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful

The offload-analyst agent was hand-reshaping bin/plugin-inventory's
PluginInventory and bin/offload-scan's OffloadScan into a candidate
table on every audit run — 40.09% of this plugin's own audited spend
over 26 invocations (kotkan/claude-plugin-inference-arbitrage#24).

candidate_digest(inv, scan) -> list[CandidateRow] joins the two JSON
documents and computes shape: share_of_spend, mtr, judgment_density,
read_amplification, retry_density, fanout_multiplier,
dominant_ngram_recurrences, has_bin_script, and an advisory `flags`
list. It reuses boundary-classify's MIN_INVOCATIONS/
MIN_ATTRIBUTED_TURNS and ia_store's MIN_SHARE_OF_SPEND rather than
reinventing thresholds, and applies the judgment_density brake
(new JUDGMENT_DENSITY_BRAKE=0.3) so a row with real judgment density
never gets flagged even if every mechanical threshold is crossed.

This is a digest, not an auto-filing script (position 3, never
position 1): flags are advisory input to the analyst's own reading of
the actual n-gram/tool-call shape, never a verdict on their own.
bin/boundary-classify remains the only place a verdict is computed.

Wired into skills/offload-audit/SKILL.md step 3 and
agents/offload-analyst.md section 2 so future audit runs call the
script instead of reshaping by hand.

Tests: tests/candidate-digest.test.sh covers the issue's three worked
examples (scripted skill suppressed via has_bin_script, a flagged
agent row, and the judgment_density-brake-suppressed edge case) plus
a below-MIN_SHARE_OF_SPEND case and a full CandidateRow schema check.
Also verified against a real matched inv.json/scan.json pair from a
prior anxious audit in /tmp.
This commit is contained in:
Oleks
2026-07-30 19:27:09 +03:00
parent 88fc58b2fd
commit c4389d8bb1
5 changed files with 434 additions and 25 deletions
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "inference-arbitrage",
"version": "0.7.0",
"version": "0.8.0",
"description": "Audits a Claude Code plugin's definitions and usage transcripts to find steps that should be a deterministic script instead of raw LLM inference, and files the well-evidenced ones as issues on the target repo.",
"author": {
"name": "oleks"
+43 -16
View File
@@ -50,7 +50,9 @@ ${CLAUDE_PLUGIN_ROOT}/bin/offload-scan --plugin <name> --days N --json # dynami
- **Never implement your own recommendations.** You propose; a human or a
separate session builds.
- **Never count, parse, or aggregate by hand.** If you find yourself tallying
something, a `bin/` tool should be doing it.
something, a `bin/` tool should be doing it — including reshaping `inv.json`
and `scan.json` into a candidate table, which is now `bin/candidate-digest`
(kotkan/claude-plugin-inference-arbitrage#24), not something you do inline.
- **Never grade by hand.** Judge the five tests, write the triple, then run
`bin/boundary-classify`.
@@ -129,22 +131,47 @@ re-proposed as if new.
### 2. Gather candidates from three distinct sources
**a. Static smells** (`plugin-inventory`). The loudest is a high verb ratio
**with no `bin/` script behind it** — treat that as a conjunction, never the
ratio alone. `token-budget` has verb ratios of 0.667 and 0.714 and is perfectly
cut, because every command its skills prescribe is an invocation of a 606-line
script that already exists. Also: `duplicated_command_blocks` (a shared script
nobody wrote), `rule_tables` (a dispatch table being narrated at inference time),
low `script_coverage`.
**Never hand-reshape `inv.json` and `scan.json` into a candidate table.** That
was this plugin's own worst offender — 40% of its audited spend
(kotkan/claude-plugin-inference-arbitrage#24) — and it is now a script:
**b. Dynamic smells** (`offload-scan`). High `mtr` with meaningful spend; a
recurring `ngram` (an algorithm observed in the wild, which is the strongest
evidence there is); high `read_amplification` (a digest belongs upstream); high
`retry_density` (often just wants a thin wrapper that gets the invocation right
once); `fanout_multiplier > 1` (mechanical work inside a subagent costs a
multiple). **`judgment_density` is a brake, not an accelerator** — high judgment
density with high spend means the plugin is doing what it should, and must be
reported as **healthy**.
```bash
${CLAUDE_PLUGIN_ROOT}/bin/candidate-digest inv.json scan.json --json
```
It returns one `CandidateRow` per skill/agent with the shape fields below plus
`has_bin_script` and `flags`. **It is a digest, not a verdict** (position 3):
`flags` marks a row that crosses the same thresholds `boundary-classify`
already applies — `MIN_INVOCATIONS`, `MIN_ATTRIBUTED_TURNS`,
`MIN_SHARE_OF_SPEND` — plus the `judgment_density` brake. A flagged row is
still yours to read, not yours to auto-file: the digest can only see
token/turn shape, not intent, and a row can cross every mechanical threshold
yet turn out to be this audit's own necessary two-pass discipline (read the
static inventory, then the dynamic scan, then reshape) rather than a
genuinely wasteful loop. That overrule case is exactly why the cut sits here
and not in an auto-filing script.
**a. Static smells** (`plugin-inventory`, `has_bin_script` in the digest). The
loudest is a high verb ratio **with no `bin/` script behind it** — treat that
as a conjunction, never the ratio alone. `token-budget` has verb ratios of
0.667 and 0.714 and is perfectly cut, because every command its skills
prescribe is an invocation of a 606-line script that already exists — the
digest's `has_bin_script: true` suppresses exactly this shape. Also read
directly from `plugin-inventory` (the digest does not carry these):
`duplicated_command_blocks` (a shared script nobody wrote), `rule_tables` (a
dispatch table being narrated at inference time), low `script_coverage`.
**b. Dynamic smells** (`offload-scan`, most already surfaced in the digest).
High `mtr` with meaningful spend; a recurring `ngram`
(`dominant_ngram_recurrences` — an algorithm observed in the wild, which is
the strongest evidence there is); high `read_amplification` (a digest belongs
upstream); high `retry_density` (often just wants a thin wrapper that gets the
invocation right once); `fanout_multiplier > 1` (mechanical work inside a
subagent costs a multiple). **`judgment_density` is a brake, not an
accelerator** — high judgment density with high spend means the plugin is
doing what it should, and must be reported as **healthy**; the digest's
`flags` already encode this (a row with `judgment_density >= 0.3` is never
flagged), but confirm it by eye on anything you file.
**c. Hook/prose drift — its own category, framed differently.** Entries in
`aggregate.hook_prose_drift` are **restated configuration contracts, not
+198
View File
@@ -0,0 +1,198 @@
#!/usr/bin/env python3
"""Joins bin/plugin-inventory's PluginInventory and bin/offload-scan's
OffloadScan into a candidate table -- the reshaping the offload-analyst agent
used to do by hand on every single audit run (40.09% of this plugin's own
audited spend, kotkan/claude-plugin-inference-arbitrage#24).
This is a DIGEST, not an auto-filing script: position 3 of the rubric
(references/boundary-rubric.md), never position 1. It computes SHAPE -- which
rows cross the codified mechanical thresholds -- and leaves intent to a
human/the analyst. A row can cross every mechanical threshold (share, mtr,
read-amplification, a recurring n-gram) and still turn out to be this audit's
own necessary two-pass discipline (read the static inventory, then the
dynamic scan, then reshape both) rather than a genuinely wasteful loop. The
digest cannot see intent, only shape, so `flags` is advisory input to a
human's judgment call -- never a verdict, and never wired to filing.
`bin/boundary-classify` remains the only place a verdict is computed; this
script does not import it for grading, only for its two evidence-strength
constants (see below).
Usage: candidate-digest <inv.json|-> <scan.json> [--json]
"""
import argparse
import importlib.machinery
import importlib.util
import json
import os
import sys
BIN_DIR = os.path.dirname(os.path.abspath(__file__))
# ia_store.py has no hyphen, so a normal import works once bin/ is on
# sys.path -- which it already is (sys.path[0]) when this file is run as
# `bin/candidate-digest`, per ia_store.py's own docstring.
from ia_store import MIN_SHARE_OF_SPEND, scan_rows # noqa: E402
def _load_bin_module(filename):
"""boundary-classify has a hyphenated filename and can't be `import`ed by
name -- loaded the same way offload-scan loads token-budget's cc-tokens
(resolve_cc_tokens there), so this plugin has exactly one pattern for
"import a sibling bin/ script as a module"."""
path = os.path.join(BIN_DIR, filename)
mod_name = filename.replace("-", "_")
spec = importlib.util.spec_from_loader(
mod_name, importlib.machinery.SourceFileLoader(mod_name, path))
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
_boundary_classify = _load_bin_module("boundary-classify")
# Reused, not reinvented (kotkan/claude-plugin-inference-arbitrage#24 says to
# reuse these): the same bar boundary-classify already applies to call a
# candidate's evidence `measured` rather than `thin`.
MIN_INVOCATIONS = _boundary_classify.MIN_INVOCATIONS
MIN_ATTRIBUTED_TURNS = _boundary_classify.MIN_ATTRIBUTED_TURNS
# New in this script -- no existing catalog threshold covers "how much
# judgment_density is enough to call a recurring shape genuine work rather
# than waste". The issue's own worked example put a real non-flag case at
# judgment_density=0.55 and a real flag case at 0.034; 0.3 sits well clear of
# both and reads naturally as "under a third of this row's turns were
# judgment calls". This is the judgment_density BRAKE
# (agents/offload-analyst.md: "a brake, not an accelerator") -- it can only
# suppress a flag, never add one.
JUDGMENT_DENSITY_BRAKE = 0.3
def _has_bin_script(name, kind, inv):
"""True only when plugin-inventory's OWN static evidence shows this row's
documented procedure already delegates to a bin/ script.
Skills carry `command_blocks` (the fenced code in their SKILL.md); a
block whose first line names a `bin/` path means the skill is *already*
a thin wrapper around an existing script -- the
"verb_ratio=0.71 backed by a 606-LOC script" example from the issue,
where the conjunction (high verb ratio AND no script) fails and nothing
should be surfaced.
Agents carry no body or command-block text in plugin-inventory's JSON at
all -- only skills do -- so this is conservatively False for every agent
row. That is a real limit on what this script can see, not a judgment
call it is dodging: an analyst who knows a given agent already delegates
to bin/ overrides this field by hand, same as any other digest output.
"""
if kind != "skill":
return False
for skill in inv.get("skills", []):
if skill.get("name") == name:
return any("bin/" in b.get("sig", "")
for b in skill.get("command_blocks", []))
return False
def _dominant_ngram_recurrences(row):
ngrams = row.get("ngrams") or []
if not ngrams:
return None
return max(g.get("recurrences", 0) for g in ngrams)
def _flags(share, invocations, turns, judgment_density, dominant_ngram,
has_bin_script):
"""The escalation path documented in kotkan/claude-plugin-inference-
arbitrage#24: flag a row when it crosses the same thresholds
boundary-classify already uses for evidence strength and filing share
(MIN_INVOCATIONS, MIN_ATTRIBUTED_TURNS, MIN_SHARE_OF_SPEND), shows a
dominant recurring n-gram, AND judgment_density sits below the brake.
`judgment_density` is a brake, not an accelerator: a row that crosses
every other threshold but carries real judgment density is the healthy
case (this audit's own two-pass read-then-reshape discipline, not a
wasteful loop) and MUST come back with flags=[] -- that is the low_judgment
check below, and it is the whole reason this is a digest and not an
auto-filing script.
A row already backed by a bin/ script (has_bin_script) is never flagged
at all: whatever mechanical share it carries is already offloaded, so
there is nothing left here to propose.
"""
if has_bin_script:
return []
measured = invocations >= MIN_INVOCATIONS and turns >= MIN_ATTRIBUTED_TURNS
significant_share = share >= MIN_SHARE_OF_SPEND
recurring = dominant_ngram is not None and dominant_ngram > 1
low_judgment = judgment_density < JUDGMENT_DENSITY_BRAKE
flags = []
if measured and significant_share and recurring and low_judgment:
flags.append("high-share-low-judgment-recurring-ngram")
return flags
def candidate_digest(inv, scan):
"""PluginInventory, OffloadScan -> list[CandidateRow]
(kotkan/claude-plugin-inference-arbitrage#24).
Pure function of its two already-computed JSON documents -- no
filesystem access beyond what the caller already read, no network, no
re-derivation of anything plugin-inventory or offload-scan already
computed. This is the entire join the offload-analyst agent used to
reshape by hand every audit run.
CandidateRow = {row_kind: 'skill'|'agent', name: str,
share_of_spend: float, mtr: float, judgment_density: float,
read_amplification: float, retry_density: float,
fanout_multiplier: float, dominant_ngram_recurrences: int|None,
has_bin_script: bool, flags: list[str]}
"""
rows = []
for name, row in scan_rows(scan):
# scan_rows() (ia_store.py) yields every scan["skills"] row keyed by
# row["skill"] and every scan["agents"] row keyed by row["agent"];
# the row itself still carries whichever of those two keys it came
# from, which is the only place kind is recoverable from here.
kind = "agent" if "agent" in row else "skill"
has_bin = _has_bin_script(name, kind, inv)
dominant = _dominant_ngram_recurrences(row)
share = row.get("share_of_plugin", 0.0)
judgment_density = row.get("judgment_density", 0.0)
rows.append({
"row_kind": kind,
"name": name,
"share_of_spend": share,
"mtr": row.get("mtr", 0.0),
"judgment_density": judgment_density,
"read_amplification": row.get("read_amplification", 0.0),
"retry_density": row.get("retry_density", 0.0),
"fanout_multiplier": row.get("fanout_multiplier", 1.0),
"dominant_ngram_recurrences": dominant,
"has_bin_script": has_bin,
"flags": _flags(share, row.get("invocations", 0),
row.get("turns", 0), judgment_density,
dominant, has_bin),
})
rows.sort(key=lambda r: -r["share_of_spend"])
return rows
def main():
ap = argparse.ArgumentParser(prog="candidate-digest")
ap.add_argument("inventory",
help="path to plugin-inventory --json output, or - for stdin")
ap.add_argument("scan", help="path to offload-scan --json output")
ap.add_argument("--json", action="store_true", help="compact single-line JSON")
args = ap.parse_args()
with (sys.stdin if args.inventory == "-" else open(args.inventory)) as fh:
inv = json.load(fh)
with open(args.scan) as fh:
scan = json.load(fh)
rows = candidate_digest(inv, scan)
json.dump(rows, sys.stdout, indent=None if args.json else 2)
sys.stdout.write("\n")
if __name__ == "__main__":
main()
+29 -8
View File
@@ -78,15 +78,36 @@ with `--note` so the wiki page records that this run is missing from memory.
## 3. Gather candidates
From `plugin-inventory`: high verb ratio **with no `bin/` script behind it**
(a conjunction — never the ratio alone), duplicated command blocks, rule tables,
low script coverage. From `offload-scan`: high MTR with real spend, recurring
n-grams, read amplification, retry density, fan-out. `judgment_density` is a
**brake** — high judgment density with high spend means the plugin is healthy
and must be reported that way.
**Never hand-reshape `inv.json` and `scan.json` into a candidate table.**
That reshaping used to be done by inference on every single audit run and was
itself 40% of this plugin's own audited spend
(kotkan/claude-plugin-inference-arbitrage#24). Run the join instead:
`aggregate.hook_prose_drift` entries are **not offload candidates.** Pass them
with `"category": "hook-prose-drift"` so they come back as `drift-note`.
```bash
$IA/bin/candidate-digest inv.json scan.json --json > digest.json
```
This produces one `CandidateRow` per skill/agent (`row_kind`, `name`,
`share_of_spend`, `mtr`, `judgment_density`, `read_amplification`,
`retry_density`, `fanout_multiplier`, `dominant_ngram_recurrences`,
`has_bin_script`, `flags`). It is a **digest, not a verdict** — position 3, the
same as this whole plugin's own thesis. A `flags` entry is a shape the digest
noticed crossing the same thresholds `boundary-classify` already applies
(`MIN_INVOCATIONS`, `MIN_ATTRIBUTED_TURNS`, `MIN_SHARE_OF_SPEND`) plus the
`judgment_density` brake; it is **never** grounds to skip step 4's judgment or
to file directly. Read every flagged row's actual n-gram/tool-call shape before
deciding candidate vs. this audit's own necessary two-pass discipline — the
overrule case `bin/boundary-classify`'s hard gate (§4 below) exists for.
Also read `plugin-inventory`'s own fields the digest does not carry: duplicated
command blocks, rule tables, low script coverage. `judgment_density` is a
**brake, not an accelerator** — a flagged row with real judgment density
(`>= 0.3`) comes back unflagged, and high judgment density with high spend
means the plugin is healthy and must be reported that way.
`aggregate.hook_prose_drift` entries are **not offload candidates** and never
appear in `digest.json`. Pass them with `"category": "hook-prose-drift"` so
they come back as `drift-note`.
## 4. Judge, then grade mechanically
+163
View File
@@ -0,0 +1,163 @@
#!/usr/bin/env bash
# bin/candidate-digest: the inv.json/scan.json join
# (kotkan/claude-plugin-inference-arbitrage#24). Synthetic fixtures for the
# three worked examples from the issue body, plus the has_bin_script suppression.
set -euo pipefail
cd "$(dirname "${BASH_SOURCE[0]}")"
BIN=../bin/candidate-digest
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
fail=0
field() {
python3 -c "import json,sys;print(json.load(open(sys.argv[1]))[int(sys.argv[2])][sys.argv[3]])" "$@"
}
check() {
local what=$1 got=$2 want=$3
if [ "$got" = "$want" ]; then
echo " ok $what = $want"
else
echo " FAIL $what = $got (want $want)" >&2
fail=1
fi
}
# ---------------------------------------------------------------------------
# Example 1: verb_ratio=0.71 skill backed by a bin/ script -> has_bin_script
# True, flags=[] regardless of its dynamic numbers (conjunction fails).
# Example 2: agent row, mtr=0.17 share=0.40 judgment_density=0.034 with a
# dominant recurring 8-gram -> flags=['high-share-low-judgment-recurring-ngram'].
# Example 3 (EDGE): same mtr/share/ngram but judgment_density=0.55 -> flags=[]
# (the judgment_density brake suppresses the false positive).
# ---------------------------------------------------------------------------
cat >"$tmp/inv.json" <<'EOF'
{
"plugin": {"name": "synthetic"},
"skills": [
{
"name": "syn:scripted-skill",
"verb_ratio": 0.71,
"command_blocks": [{"sig": "${CLAUDE_PLUGIN_ROOT}/bin/do-the-thing --json", "count": 1}]
}
]
}
EOF
cat >"$tmp/scan.json" <<'EOF'
{
"skills": [
{
"skill": "syn:scripted-skill",
"invocations": 12,
"turns": 90,
"share_of_plugin": 0.5,
"mtr": 0.8,
"judgment_density": 0.05,
"read_amplification": 10.0,
"retry_density": 0.0,
"fanout_multiplier": 1.0,
"ngrams": [{"sig": ["a", "b", "c"], "recurrences": 9, "invocations": 5, "waste": 1000}]
}
],
"agents": [
{
"agent": "syn:flagged-agent",
"invocations": 26,
"turns": 208,
"share_of_plugin": 0.40,
"mtr": 0.17,
"judgment_density": 0.034,
"read_amplification": 4.0,
"retry_density": 0.02,
"fanout_multiplier": 1.0,
"ngrams": [{"sig": ["x", "y"], "recurrences": 8, "invocations": 6, "waste": 800000}]
},
{
"agent": "syn:healthy-agent",
"invocations": 26,
"turns": 208,
"share_of_plugin": 0.40,
"mtr": 0.17,
"judgment_density": 0.55,
"read_amplification": 4.0,
"retry_density": 0.02,
"fanout_multiplier": 1.0,
"ngrams": [{"sig": ["x", "y"], "recurrences": 8, "invocations": 6, "waste": 800000}]
}
]
}
EOF
$BIN "$tmp/inv.json" "$tmp/scan.json" >"$tmp/out.json"
echo "== example 1: scripted skill, high mtr/share, has_bin_script suppresses everything =="
row=$(python3 -c "
import json
rows = json.load(open('$tmp/out.json'))
print([i for i, r in enumerate(rows) if r['name'] == 'syn:scripted-skill'][0])
")
check "row_kind" "$(field "$tmp/out.json" "$row" row_kind)" "skill"
check "has_bin_script" "$(field "$tmp/out.json" "$row" has_bin_script)" "True"
check "flags" "$(field "$tmp/out.json" "$row" flags)" "[]"
echo "== example 2: agent, judgment_density=0.034 -> flagged =="
row=$(python3 -c "
import json
rows = json.load(open('$tmp/out.json'))
print([i for i, r in enumerate(rows) if r['name'] == 'syn:flagged-agent'][0])
")
check "row_kind" "$(field "$tmp/out.json" "$row" row_kind)" "agent"
check "has_bin_script" "$(field "$tmp/out.json" "$row" has_bin_script)" "False"
check "dominant_ngram_recurrences" "$(field "$tmp/out.json" "$row" dominant_ngram_recurrences)" "8"
check "flags" "$(field "$tmp/out.json" "$row" flags)" "['high-share-low-judgment-recurring-ngram']"
echo "== example 3 (EDGE): same mtr/share/ngram, judgment_density=0.55 -> brake suppresses =="
row=$(python3 -c "
import json
rows = json.load(open('$tmp/out.json'))
print([i for i, r in enumerate(rows) if r['name'] == 'syn:healthy-agent'][0])
")
check "flags" "$(field "$tmp/out.json" "$row" flags)" "[]"
echo "== below MIN_SHARE_OF_SPEND -> never flagged even with a low judgment_density =="
cat >"$tmp/scan-thin.json" <<'EOF'
{
"skills": [],
"agents": [
{
"agent": "syn:thin-agent",
"invocations": 12,
"turns": 90,
"share_of_plugin": 0.005,
"mtr": 0.9,
"judgment_density": 0.01,
"read_amplification": 3.0,
"retry_density": 0.0,
"fanout_multiplier": 1.0,
"ngrams": [{"sig": ["p", "q"], "recurrences": 5, "invocations": 4, "waste": 100}]
}
]
}
EOF
cat >"$tmp/inv-empty.json" <<'EOF'
{"plugin": {"name": "synthetic"}, "skills": []}
EOF
$BIN "$tmp/inv-empty.json" "$tmp/scan-thin.json" >"$tmp/out-thin.json"
check "flags" "$(field "$tmp/out-thin.json" 0 flags)" "[]"
echo "== schema: every CandidateRow key present =="
python3 -c "
import json
rows = json.load(open('$tmp/out.json'))
want = {'row_kind', 'name', 'share_of_spend', 'mtr', 'judgment_density',
'read_amplification', 'retry_density', 'fanout_multiplier',
'dominant_ngram_recurrences', 'has_bin_script', 'flags'}
for r in rows:
assert set(r.keys()) == want, (r.keys(), want)
print('ok')
"
exit "$fail"