Compare commits
12 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 230a44348c | |||
| 9a25402d37 | |||
| c4389d8bb1 | |||
| 5f5462571c | |||
| 64d6c90284 | |||
| b77fbdddbb | |||
| 88fc58b2fd | |||
| 4230782aaa | |||
| eba3449e60 | |||
| 1a14456220 | |||
| 58b14cbde1 | |||
| c8943690e0 |
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "inference-arbitrage",
|
||||
"version": "0.6.3",
|
||||
"version": "0.8.1",
|
||||
"description": "Audits a Claude Code plugin's definitions and usage transcripts to find steps that should be a deterministic script instead of raw LLM inference, and files the well-evidenced ones as issues on the target repo.",
|
||||
"author": {
|
||||
"name": "oleks"
|
||||
|
||||
+15
-2
@@ -14,13 +14,26 @@ steps:
|
||||
- name: suites
|
||||
image: alpine:3
|
||||
commands:
|
||||
- apk add --no-cache bash python3 >/dev/null
|
||||
- apk add --no-cache bash python3 wget >/dev/null
|
||||
# bin/offload-scan (design decision 0.1) deliberately never vendors
|
||||
# token-budget's pricing/token-accounting logic -- it imports the real
|
||||
# cc-tokens module and fails loudly if it's missing. This bare alpine
|
||||
# container has neither the Claude Code plugin cache nor `claude
|
||||
# plugin install`, so fetch the one file it needs straight from
|
||||
# token-budget's own repo and put it on PATH. See
|
||||
# kotkan/claude-plugin-inference-arbitrage#20.
|
||||
- mkdir -p /tmp/ci-tools
|
||||
- >-
|
||||
wget -q -O /tmp/ci-tools/cc-tokens
|
||||
https://git.oleks.space/kotkan/claude-plugin-token-budget/raw/branch/main/bin/cc-tokens
|
||||
- chmod +x /tmp/ci-tools/cc-tokens
|
||||
- export PATH="/tmp/ci-tools:$PATH"
|
||||
- timeout 600 bash tests/run-all.sh
|
||||
|
||||
- name: shellcheck
|
||||
image: koalaman/shellcheck-alpine:stable
|
||||
commands:
|
||||
# bin/ mixes python and bash (plugin-inventory is python) — filter by
|
||||
# bin/ mixes python and bash (plugin-inventory is python) -- filter by
|
||||
# actual shebang, not just the executable bit or a *.sh glob, or
|
||||
# shellcheck chokes on the python entries (SC1071).
|
||||
- >-
|
||||
|
||||
+43
-16
@@ -50,7 +50,9 @@ ${CLAUDE_PLUGIN_ROOT}/bin/offload-scan --plugin <name> --days N --json # dynami
|
||||
- **Never implement your own recommendations.** You propose; a human or a
|
||||
separate session builds.
|
||||
- **Never count, parse, or aggregate by hand.** If you find yourself tallying
|
||||
something, a `bin/` tool should be doing it.
|
||||
something, a `bin/` tool should be doing it — including reshaping `inv.json`
|
||||
and `scan.json` into a candidate table, which is now `bin/candidate-digest`
|
||||
(kotkan/claude-plugin-inference-arbitrage#24), not something you do inline.
|
||||
- **Never grade by hand.** Judge the five tests, write the triple, then run
|
||||
`bin/boundary-classify`.
|
||||
|
||||
@@ -129,22 +131,47 @@ re-proposed as if new.
|
||||
|
||||
### 2. Gather candidates from three distinct sources
|
||||
|
||||
**a. Static smells** (`plugin-inventory`). The loudest is a high verb ratio
|
||||
**with no `bin/` script behind it** — treat that as a conjunction, never the
|
||||
ratio alone. `token-budget` has verb ratios of 0.667 and 0.714 and is perfectly
|
||||
cut, because every command its skills prescribe is an invocation of a 606-line
|
||||
script that already exists. Also: `duplicated_command_blocks` (a shared script
|
||||
nobody wrote), `rule_tables` (a dispatch table being narrated at inference time),
|
||||
low `script_coverage`.
|
||||
**Never hand-reshape `inv.json` and `scan.json` into a candidate table.** That
|
||||
was this plugin's own worst offender — 40% of its audited spend
|
||||
(kotkan/claude-plugin-inference-arbitrage#24) — and it is now a script:
|
||||
|
||||
**b. Dynamic smells** (`offload-scan`). High `mtr` with meaningful spend; a
|
||||
recurring `ngram` (an algorithm observed in the wild, which is the strongest
|
||||
evidence there is); high `read_amplification` (a digest belongs upstream); high
|
||||
`retry_density` (often just wants a thin wrapper that gets the invocation right
|
||||
once); `fanout_multiplier > 1` (mechanical work inside a subagent costs a
|
||||
multiple). **`judgment_density` is a brake, not an accelerator** — high judgment
|
||||
density with high spend means the plugin is doing what it should, and must be
|
||||
reported as **healthy**.
|
||||
```bash
|
||||
${CLAUDE_PLUGIN_ROOT}/bin/candidate-digest inv.json scan.json --json
|
||||
```
|
||||
|
||||
It returns one `CandidateRow` per skill/agent with the shape fields below plus
|
||||
`has_bin_script` and `flags`. **It is a digest, not a verdict** (position 3):
|
||||
`flags` marks a row that crosses the same thresholds `boundary-classify`
|
||||
already applies — `MIN_INVOCATIONS`, `MIN_ATTRIBUTED_TURNS`,
|
||||
`MIN_SHARE_OF_SPEND` — plus the `judgment_density` brake. A flagged row is
|
||||
still yours to read, not yours to auto-file: the digest can only see
|
||||
token/turn shape, not intent, and a row can cross every mechanical threshold
|
||||
yet turn out to be this audit's own necessary two-pass discipline (read the
|
||||
static inventory, then the dynamic scan, then reshape) rather than a
|
||||
genuinely wasteful loop. That overrule case is exactly why the cut sits here
|
||||
and not in an auto-filing script.
|
||||
|
||||
**a. Static smells** (`plugin-inventory`, `has_bin_script` in the digest). The
|
||||
loudest is a high verb ratio **with no `bin/` script behind it** — treat that
|
||||
as a conjunction, never the ratio alone. `token-budget` has verb ratios of
|
||||
0.667 and 0.714 and is perfectly cut, because every command its skills
|
||||
prescribe is an invocation of a 606-line script that already exists — the
|
||||
digest's `has_bin_script: true` suppresses exactly this shape. Also read
|
||||
directly from `plugin-inventory` (the digest does not carry these):
|
||||
`duplicated_command_blocks` (a shared script nobody wrote), `rule_tables` (a
|
||||
dispatch table being narrated at inference time), low `script_coverage`.
|
||||
|
||||
**b. Dynamic smells** (`offload-scan`, most already surfaced in the digest).
|
||||
High `mtr` with meaningful spend; a recurring `ngram`
|
||||
(`dominant_ngram_recurrences` — an algorithm observed in the wild, which is
|
||||
the strongest evidence there is); high `read_amplification` (a digest belongs
|
||||
upstream); high `retry_density` (often just wants a thin wrapper that gets the
|
||||
invocation right once); `fanout_multiplier > 1` (mechanical work inside a
|
||||
subagent costs a multiple). **`judgment_density` is a brake, not an
|
||||
accelerator** — high judgment density with high spend means the plugin is
|
||||
doing what it should, and must be reported as **healthy**; the digest's
|
||||
`flags` already encode this (a row with `judgment_density >= 0.3` is never
|
||||
flagged), but confirm it by eye on anything you file.
|
||||
|
||||
**c. Hook/prose drift — its own category, framed differently.** Entries in
|
||||
`aggregate.hook_prose_drift` are **restated configuration contracts, not
|
||||
|
||||
+13
-1
@@ -180,9 +180,14 @@ def build_snapshot(target, inventory, scan, classified, issue_map, explicit, run
|
||||
# the filing pass — `filing: unavailable` and why (FR-6.4) — which this
|
||||
# script cannot see and which have to reach the wiki page, not just the
|
||||
# user's terminal.
|
||||
# `insufficient-window` is in this list, not in `candidates` or
|
||||
# `boundary_questions`: an FR-4.5 deferral is neither a filed candidate nor
|
||||
# a question put to a human, and the wider snapshot it was deferred against
|
||||
# remains the standing comparison point. Recording it as a note keeps the
|
||||
# deferral visible on the wiki page instead of silently dropping the row.
|
||||
notes = [
|
||||
f"{c['candidate_id']}: {c['verdict']} — {c['reasons'][-1] if c.get('reasons') else ''}"
|
||||
for c in rows if c.get("verdict") in ("no-offload", "drift-note")
|
||||
for c in rows if c.get("verdict") in ("no-offload", "drift-note", "insufficient-window")
|
||||
] + list(extra_notes)
|
||||
|
||||
return {
|
||||
@@ -212,6 +217,7 @@ def build_snapshot(target, inventory, scan, classified, issue_map, explicit, run
|
||||
"weighted_tokens": scan_total.get("weighted_tokens", 0),
|
||||
"invocations": scan_total.get("invocations", 0),
|
||||
"share_mechanical": scan_total.get("mechanical_share", 0.0),
|
||||
"invocation_caveat": scan_total.get("invocation_caveat"),
|
||||
},
|
||||
"candidates": [snapshot_candidate(c, scan, explicit, issue_map) for c in filed],
|
||||
"boundary_questions": [snapshot_candidate(c, scan, explicit, issue_map) for c in questions],
|
||||
@@ -452,6 +458,12 @@ def render_run(snap, diffdoc):
|
||||
f"- Mechanical share of audited spend: {(snap['totals'].get('share_mechanical') or 0):.1%}",
|
||||
f"- Rubric v{snap.get('rubric_version')}, inference-arbitrage "
|
||||
f"v{snap.get('tool_versions', {}).get('inference-arbitrage')}",
|
||||
]
|
||||
invocation_caveat = snap["totals"].get("invocation_caveat")
|
||||
if invocation_caveat:
|
||||
lines.append(f"- Note: {invocation_caveat}, so per-invocation figures below are "
|
||||
"correspondingly inflated")
|
||||
lines += [
|
||||
"",
|
||||
"## Candidates",
|
||||
"",
|
||||
|
||||
Executable
+198
@@ -0,0 +1,198 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Joins bin/plugin-inventory's PluginInventory and bin/offload-scan's
|
||||
OffloadScan into a candidate table -- the reshaping the offload-analyst agent
|
||||
used to do by hand on every single audit run (40.09% of this plugin's own
|
||||
audited spend, kotkan/claude-plugin-inference-arbitrage#24).
|
||||
|
||||
This is a DIGEST, not an auto-filing script: position 3 of the rubric
|
||||
(references/boundary-rubric.md), never position 1. It computes SHAPE -- which
|
||||
rows cross the codified mechanical thresholds -- and leaves intent to a
|
||||
human/the analyst. A row can cross every mechanical threshold (share, mtr,
|
||||
read-amplification, a recurring n-gram) and still turn out to be this audit's
|
||||
own necessary two-pass discipline (read the static inventory, then the
|
||||
dynamic scan, then reshape both) rather than a genuinely wasteful loop. The
|
||||
digest cannot see intent, only shape, so `flags` is advisory input to a
|
||||
human's judgment call -- never a verdict, and never wired to filing.
|
||||
`bin/boundary-classify` remains the only place a verdict is computed; this
|
||||
script does not import it for grading, only for its two evidence-strength
|
||||
constants (see below).
|
||||
|
||||
Usage: candidate-digest <inv.json|-> <scan.json> [--json]
|
||||
"""
|
||||
import argparse
|
||||
import importlib.machinery
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
BIN_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||
|
||||
# ia_store.py has no hyphen, so a normal import works once bin/ is on
|
||||
# sys.path -- which it already is (sys.path[0]) when this file is run as
|
||||
# `bin/candidate-digest`, per ia_store.py's own docstring.
|
||||
from ia_store import MIN_SHARE_OF_SPEND, scan_rows # noqa: E402
|
||||
|
||||
|
||||
def _load_bin_module(filename):
|
||||
"""boundary-classify has a hyphenated filename and can't be `import`ed by
|
||||
name -- loaded the same way offload-scan loads token-budget's cc-tokens
|
||||
(resolve_cc_tokens there), so this plugin has exactly one pattern for
|
||||
"import a sibling bin/ script as a module"."""
|
||||
path = os.path.join(BIN_DIR, filename)
|
||||
mod_name = filename.replace("-", "_")
|
||||
spec = importlib.util.spec_from_loader(
|
||||
mod_name, importlib.machinery.SourceFileLoader(mod_name, path))
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
_boundary_classify = _load_bin_module("boundary-classify")
|
||||
# Reused, not reinvented (kotkan/claude-plugin-inference-arbitrage#24 says to
|
||||
# reuse these): the same bar boundary-classify already applies to call a
|
||||
# candidate's evidence `measured` rather than `thin`.
|
||||
MIN_INVOCATIONS = _boundary_classify.MIN_INVOCATIONS
|
||||
MIN_ATTRIBUTED_TURNS = _boundary_classify.MIN_ATTRIBUTED_TURNS
|
||||
|
||||
# New in this script -- no existing catalog threshold covers "how much
|
||||
# judgment_density is enough to call a recurring shape genuine work rather
|
||||
# than waste". The issue's own worked example put a real non-flag case at
|
||||
# judgment_density=0.55 and a real flag case at 0.034; 0.3 sits well clear of
|
||||
# both and reads naturally as "under a third of this row's turns were
|
||||
# judgment calls". This is the judgment_density BRAKE
|
||||
# (agents/offload-analyst.md: "a brake, not an accelerator") -- it can only
|
||||
# suppress a flag, never add one.
|
||||
JUDGMENT_DENSITY_BRAKE = 0.3
|
||||
|
||||
|
||||
def _has_bin_script(name, kind, inv):
|
||||
"""True only when plugin-inventory's OWN static evidence shows this row's
|
||||
documented procedure already delegates to a bin/ script.
|
||||
|
||||
Skills carry `command_blocks` (the fenced code in their SKILL.md); a
|
||||
block whose first line names a `bin/` path means the skill is *already*
|
||||
a thin wrapper around an existing script -- the
|
||||
"verb_ratio=0.71 backed by a 606-LOC script" example from the issue,
|
||||
where the conjunction (high verb ratio AND no script) fails and nothing
|
||||
should be surfaced.
|
||||
|
||||
Agents carry no body or command-block text in plugin-inventory's JSON at
|
||||
all -- only skills do -- so this is conservatively False for every agent
|
||||
row. That is a real limit on what this script can see, not a judgment
|
||||
call it is dodging: an analyst who knows a given agent already delegates
|
||||
to bin/ overrides this field by hand, same as any other digest output.
|
||||
"""
|
||||
if kind != "skill":
|
||||
return False
|
||||
for skill in inv.get("skills", []):
|
||||
if skill.get("name") == name:
|
||||
return any("bin/" in b.get("sig", "")
|
||||
for b in skill.get("command_blocks", []))
|
||||
return False
|
||||
|
||||
|
||||
def _dominant_ngram_recurrences(row):
|
||||
ngrams = row.get("ngrams") or []
|
||||
if not ngrams:
|
||||
return None
|
||||
return max(g.get("recurrences", 0) for g in ngrams)
|
||||
|
||||
|
||||
def _flags(share, invocations, turns, judgment_density, dominant_ngram,
|
||||
has_bin_script):
|
||||
"""The escalation path documented in kotkan/claude-plugin-inference-
|
||||
arbitrage#24: flag a row when it crosses the same thresholds
|
||||
boundary-classify already uses for evidence strength and filing share
|
||||
(MIN_INVOCATIONS, MIN_ATTRIBUTED_TURNS, MIN_SHARE_OF_SPEND), shows a
|
||||
dominant recurring n-gram, AND judgment_density sits below the brake.
|
||||
|
||||
`judgment_density` is a brake, not an accelerator: a row that crosses
|
||||
every other threshold but carries real judgment density is the healthy
|
||||
case (this audit's own two-pass read-then-reshape discipline, not a
|
||||
wasteful loop) and MUST come back with flags=[] -- that is the low_judgment
|
||||
check below, and it is the whole reason this is a digest and not an
|
||||
auto-filing script.
|
||||
|
||||
A row already backed by a bin/ script (has_bin_script) is never flagged
|
||||
at all: whatever mechanical share it carries is already offloaded, so
|
||||
there is nothing left here to propose.
|
||||
"""
|
||||
if has_bin_script:
|
||||
return []
|
||||
measured = invocations >= MIN_INVOCATIONS and turns >= MIN_ATTRIBUTED_TURNS
|
||||
significant_share = share >= MIN_SHARE_OF_SPEND
|
||||
recurring = dominant_ngram is not None and dominant_ngram > 1
|
||||
low_judgment = judgment_density < JUDGMENT_DENSITY_BRAKE
|
||||
flags = []
|
||||
if measured and significant_share and recurring and low_judgment:
|
||||
flags.append("high-share-low-judgment-recurring-ngram")
|
||||
return flags
|
||||
|
||||
|
||||
def candidate_digest(inv, scan):
|
||||
"""PluginInventory, OffloadScan -> list[CandidateRow]
|
||||
(kotkan/claude-plugin-inference-arbitrage#24).
|
||||
|
||||
Pure function of its two already-computed JSON documents -- no
|
||||
filesystem access beyond what the caller already read, no network, no
|
||||
re-derivation of anything plugin-inventory or offload-scan already
|
||||
computed. This is the entire join the offload-analyst agent used to
|
||||
reshape by hand every audit run.
|
||||
|
||||
CandidateRow = {row_kind: 'skill'|'agent', name: str,
|
||||
share_of_spend: float, mtr: float, judgment_density: float,
|
||||
read_amplification: float, retry_density: float,
|
||||
fanout_multiplier: float, dominant_ngram_recurrences: int|None,
|
||||
has_bin_script: bool, flags: list[str]}
|
||||
"""
|
||||
rows = []
|
||||
for name, row in scan_rows(scan):
|
||||
# scan_rows() (ia_store.py) yields every scan["skills"] row keyed by
|
||||
# row["skill"] and every scan["agents"] row keyed by row["agent"];
|
||||
# the row itself still carries whichever of those two keys it came
|
||||
# from, which is the only place kind is recoverable from here.
|
||||
kind = "agent" if "agent" in row else "skill"
|
||||
has_bin = _has_bin_script(name, kind, inv)
|
||||
dominant = _dominant_ngram_recurrences(row)
|
||||
share = row.get("share_of_plugin", 0.0)
|
||||
judgment_density = row.get("judgment_density", 0.0)
|
||||
rows.append({
|
||||
"row_kind": kind,
|
||||
"name": name,
|
||||
"share_of_spend": share,
|
||||
"mtr": row.get("mtr", 0.0),
|
||||
"judgment_density": judgment_density,
|
||||
"read_amplification": row.get("read_amplification", 0.0),
|
||||
"retry_density": row.get("retry_density", 0.0),
|
||||
"fanout_multiplier": row.get("fanout_multiplier", 1.0),
|
||||
"dominant_ngram_recurrences": dominant,
|
||||
"has_bin_script": has_bin,
|
||||
"flags": _flags(share, row.get("invocations", 0),
|
||||
row.get("turns", 0), judgment_density,
|
||||
dominant, has_bin),
|
||||
})
|
||||
rows.sort(key=lambda r: -r["share_of_spend"])
|
||||
return rows
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(prog="candidate-digest")
|
||||
ap.add_argument("inventory",
|
||||
help="path to plugin-inventory --json output, or - for stdin")
|
||||
ap.add_argument("scan", help="path to offload-scan --json output")
|
||||
ap.add_argument("--json", action="store_true", help="compact single-line JSON")
|
||||
args = ap.parse_args()
|
||||
|
||||
with (sys.stdin if args.inventory == "-" else open(args.inventory)) as fh:
|
||||
inv = json.load(fh)
|
||||
with open(args.scan) as fh:
|
||||
scan = json.load(fh)
|
||||
|
||||
rows = candidate_digest(inv, scan)
|
||||
json.dump(rows, sys.stdout, indent=None if args.json else 2)
|
||||
sys.stdout.write("\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+23
-4
@@ -85,10 +85,29 @@ def marker(candidate_id):
|
||||
|
||||
|
||||
def marker_re(candidate_id):
|
||||
"""Tolerates whitespace drift inside the comment, nothing else. The id
|
||||
itself must match exactly — a fuzzy id match would let two genuinely
|
||||
different candidates collapse onto one issue."""
|
||||
return re.compile(r"<!--\s*ia-candidate:\s*" + re.escape(candidate_id) + r"\s*-->")
|
||||
"""Tolerates whitespace drift inside the comment, and tolerates the
|
||||
HTML-comment wrapper (`<!-- -->`) being absent entirely, nothing else. The
|
||||
id itself must match exactly — a fuzzy id match would let two genuinely
|
||||
different candidates collapse onto one issue.
|
||||
|
||||
The bare form (no `<!-- -->`) is a backward-compatibility case, not the
|
||||
contract: kotkan/claude-plugin-inference-arbitrage#23 found that issues
|
||||
filed on 2026-07-29 (oleks/claude-plugin-cluster#56, #57) lost the wrapper
|
||||
somewhere between `filing-plan`'s render and the Gitea write — the
|
||||
template and this script always render the wrapped form (see `marker()`
|
||||
below and references/issue-template.md). Matching the bare line here is
|
||||
purely so a *later* audit still recognizes those already-filed issues and
|
||||
doesn't duplicate-file them; it does not license filing new issues without
|
||||
the wrapper.
|
||||
"""
|
||||
wrapped = r"<!--\s*ia-candidate:\s*" + re.escape(candidate_id) + r"\s*-->"
|
||||
# End-of-line anchored (not \b): candidate_id contains "/" and "-", so a
|
||||
# word-boundary check would treat "…label-derivation" as a match inside
|
||||
# "…label-derivation-longer" (the "n"/"-" transition IS a \b). Anchoring
|
||||
# to end-of-line (re.MULTILINE) instead requires an exact id, matching
|
||||
# what marker() actually renders as the line's sole content.
|
||||
bare = r"^ia-candidate:\s*" + re.escape(candidate_id) + r"[ \t]*$"
|
||||
return re.compile(wrapped + "|" + bare, re.MULTILINE)
|
||||
|
||||
|
||||
def find_existing(candidate_id, issues):
|
||||
|
||||
@@ -750,6 +750,17 @@ def main():
|
||||
"mechanical_share": round(
|
||||
sum(1 for t in turns if classify(t, catalog) == "mechanical")
|
||||
/ len(turns), 3) if turns else 0.0,
|
||||
# reconstruct_invocations() only sees already-attributed turns and
|
||||
# splits on a wall-clock idle gap; it has no signal for unrelated
|
||||
# activity between two genuinely separate same-skill calls, so
|
||||
# back-to-back invocations closer together than the gap collapse
|
||||
# into one reported invocation. Counts here are a LOWER BOUND on
|
||||
# the true invocation count, and per-invocation figures derived
|
||||
# from them (e.g. cost/invocation) are correspondingly inflated.
|
||||
"invocation_caveat": (
|
||||
"invocation counts are a lower bound: two same-skill calls "
|
||||
f"within {int(th['idle_gap_minutes'])} minutes of each other on the wall "
|
||||
"clock are merged into one reported invocation"),
|
||||
},
|
||||
"skills": skills,
|
||||
"agents": agents,
|
||||
|
||||
@@ -40,6 +40,66 @@ window and 5% the next is stable — same label, same side of the threshold —
|
||||
that variation is what `audit-snapshot diff` exists to report as a trend. This
|
||||
gate fires only when the *description a human receives* changes.
|
||||
|
||||
A NARROW WINDOW IS NOT EVIDENCE OF INSTABILITY EITHER
|
||||
-----------------------------------------------------
|
||||
The gate as first written treated every flip as symmetric, and that produced a
|
||||
false downgrade on the very candidate that motivated it
|
||||
(kotkan/claude-plugin-inference-arbitrage#15). Re-measuring
|
||||
`anxious/agent-wip/release-policy-derivation` across four real windows with the
|
||||
same script showed a monotonic accumulation, not a flip:
|
||||
|
||||
7 days -> 0 invocations, 0.00% of spend -> thin
|
||||
14 days -> 2 invocations, 0.72% -> thin
|
||||
27 days -> 12 invocations, 11.73% -> measured
|
||||
60 days -> 15 invocations, 13.87% -> measured
|
||||
|
||||
The candidate runs about once every two days. A 7-day window did not observe an
|
||||
unstable candidate; it failed to observe a real one. The wider the window, the
|
||||
higher the reading — the 60-day pass had every opportunity to dilute the share
|
||||
back down and instead raised it. Calling that "the confidence label moves with
|
||||
the calendar" gets the direction of the artifact backwards: the *narrow* window
|
||||
was the artifact, and the gate punished the candidate for it.
|
||||
|
||||
So a weak reading only gets to overturn a `measured` one when it was taken over
|
||||
a comparably wide window. Concretely, all three must hold:
|
||||
|
||||
* the prior reading is `measured`,
|
||||
* the current reading is `thin` or `unmeasured`,
|
||||
* the current window is narrower than WINDOW_PARITY_RATIO of the prior
|
||||
window's observed width (`until - since`),
|
||||
|
||||
and then the comparison is not read as instability at all. The candidate takes
|
||||
a third verdict, `insufficient-window`: it is neither filed nor downgraded, no
|
||||
self-report is emitted, and it is carried forward — the wider prior snapshot
|
||||
stays the standing comparison point, so the next audit run over an adequate
|
||||
window compares against it rather than against this window's under-sample.
|
||||
|
||||
The parity is relative, not an absolute day count, on purpose. "At least 21
|
||||
days" would be a guess about candidate frequency dressed up as a threshold —
|
||||
this candidate needs ~27 days, a once-a-week step would need more, and a
|
||||
once-an-hour step is fully measured in one. What the gate can know without
|
||||
guessing is that a window materially narrower than the one that produced the
|
||||
`measured` reading cannot fairly contradict it. WINDOW_PARITY_RATIO is 0.9
|
||||
rather than 1.0 because window bounds are observed session spans, not requested
|
||||
ranges, so two nominally identical windows differ by hours.
|
||||
|
||||
Three things stay downgrades, unchanged:
|
||||
|
||||
1. the reverse direction (`thin` prior, `measured` current) — a wider or
|
||||
equally wide current window that now sees the candidate genuinely
|
||||
contradicts the earlier reading, and the earlier reading is the weak one;
|
||||
2. any flip between comparably wide windows;
|
||||
3. a threshold-side flip in which BOTH windows read `measured`. If the
|
||||
current window still reached `measured`, it observed the candidate often
|
||||
enough; a share that then crosses the filing threshold is a claim about
|
||||
relative spend, not a sampling failure, and narrowness does not excuse it.
|
||||
|
||||
Width is read off the snapshot's own `window.since`/`window.until`. When either
|
||||
window does not carry both bounds the widths are not comparable, and the gate
|
||||
does NOT quietly excuse the flip: it downgrades as before and says so in the
|
||||
finding's `window_parity` block, so an unreadable window is visible rather than
|
||||
a silent exemption.
|
||||
|
||||
NO PRIOR WINDOW MEANS UNCHECKED, NOT UNSTABLE
|
||||
---------------------------------------------
|
||||
A first-ever audit of a target, or a candidate seen for the first time, has
|
||||
@@ -80,6 +140,7 @@ of `classified.json`.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import datetime
|
||||
import json
|
||||
import sys
|
||||
|
||||
@@ -92,6 +153,21 @@ GATE = "FR-4.5"
|
||||
UNCHECKED_NO_HISTORY = "unchecked — no prior window to compare"
|
||||
UNCHECKED_NO_MATCH = "unchecked — no prior window recorded this candidate"
|
||||
|
||||
# The verdict a comparison takes when it is too lopsided to read either way.
|
||||
# Distinct from `boundary-question` on purpose: a boundary question is something
|
||||
# a human is asked to judge, and this is the auditor declining to make a claim.
|
||||
INSUFFICIENT_WINDOW = "insufficient-window"
|
||||
|
||||
# Readings that mean "this window did not observe the candidate enough", as
|
||||
# opposed to `measured`, which means it did.
|
||||
WEAK_STRENGTHS = ("thin", "unmeasured")
|
||||
|
||||
# How wide the current window must be, as a fraction of the prior window that
|
||||
# produced the `measured` reading, before a weak reading may overturn it. Not
|
||||
# 1.0: window bounds are observed session spans, so two nominally equal windows
|
||||
# differ by hours and an exact-parity rule would fire on clock noise.
|
||||
WINDOW_PARITY_RATIO = 0.9
|
||||
|
||||
|
||||
def keyed(cand, scan, explicit, issue_map):
|
||||
"""The identity fields FR-5.2 matches on, computed for a candidate that is
|
||||
@@ -149,10 +225,78 @@ def flips(prior, curr_strength, curr_share):
|
||||
return out
|
||||
|
||||
|
||||
def window_days(window):
|
||||
"""Observed width of a window in days, or None when it cannot be read.
|
||||
|
||||
None is a real answer, not a default: `window_parity` reports it as
|
||||
"not comparable" and lets the flip stand, rather than treating an
|
||||
unreadable window as narrow (which would excuse every flip) or as wide
|
||||
(which would hide that the check could not run)."""
|
||||
window = window or {}
|
||||
bounds = []
|
||||
for key in ("since", "until"):
|
||||
raw = window.get(key)
|
||||
if not isinstance(raw, str):
|
||||
return None
|
||||
try:
|
||||
bounds.append(datetime.datetime.fromisoformat(raw))
|
||||
except ValueError:
|
||||
return None
|
||||
since, until = bounds
|
||||
# One bound tz-aware and the other naive cannot be subtracted, and guessing
|
||||
# a zone for the naive one would invent a width.
|
||||
if (since.tzinfo is None) != (until.tzinfo is None):
|
||||
return None
|
||||
return (until - since).total_seconds() / 86400.0
|
||||
|
||||
|
||||
def window_parity(prior_window, curr_window):
|
||||
"""Whether the current window is wide enough to contradict the prior one.
|
||||
|
||||
Always returned, whether or not it changes the outcome, so the finding
|
||||
records the widths it was decided on."""
|
||||
prior_days = window_days(prior_window)
|
||||
curr_days = window_days(curr_window)
|
||||
out = {
|
||||
"parity_ratio": WINDOW_PARITY_RATIO,
|
||||
"prior_days": round(prior_days, 2) if prior_days is not None else None,
|
||||
"current_days": round(curr_days, 2) if curr_days is not None else None,
|
||||
"required_days": (round(prior_days * WINDOW_PARITY_RATIO, 2)
|
||||
if prior_days is not None else None),
|
||||
}
|
||||
if prior_days is None or curr_days is None:
|
||||
out["comparable"] = None
|
||||
out["why"] = ("window width is unreadable — one of the two windows does not carry "
|
||||
"both `since` and `until` — so narrowness cannot be established and "
|
||||
"the flip is not excused; this is recorded rather than defaulted")
|
||||
return out
|
||||
out["comparable"] = curr_days >= out["required_days"]
|
||||
out["why"] = (
|
||||
f"this window spans {out['current_days']}d against the prior window's "
|
||||
f"{out['prior_days']}d; a weak reading needs at least "
|
||||
f"{out['required_days']}d ({WINDOW_PARITY_RATIO:.0%} of the prior width) to "
|
||||
"contradict a `measured` one"
|
||||
if not out["comparable"] else
|
||||
f"this window spans {out['current_days']}d against the prior window's "
|
||||
f"{out['prior_days']}d, at or above the {out['required_days']}d parity floor, so "
|
||||
"the two readings are comparably sampled and the flip is real")
|
||||
return out
|
||||
|
||||
|
||||
def deferrable(prior_strength, curr_strength):
|
||||
"""The only direction narrowness can excuse: a wide `measured` reading that a
|
||||
narrow window failed to observe. The reverse (`thin` prior, `measured` now)
|
||||
is a genuine contradiction in which the *earlier* reading is the weak one,
|
||||
and a `measured` -> `measured` threshold crossing means both windows saw the
|
||||
candidate, so neither is excused here."""
|
||||
return prior_strength == "measured" and curr_strength in WEAK_STRENGTHS
|
||||
|
||||
|
||||
def gate(doc, history, scan, explicit, issue_map):
|
||||
target = doc.get("target")
|
||||
window = scan.get("window") or {}
|
||||
findings = []
|
||||
deferrals = []
|
||||
checked = 0
|
||||
|
||||
for cand in doc.get("candidates", []):
|
||||
@@ -180,6 +324,54 @@ def gate(doc, history, scan, explicit, issue_map):
|
||||
f"{curr_strength}, {curr_share:.2%} of audited spend either side")
|
||||
continue
|
||||
|
||||
parity = window_parity(snap.get("window"), window)
|
||||
if deferrable(prior.get("measurement_strength"), curr_strength) \
|
||||
and parity["comparable"] is False:
|
||||
# Not filed, not downgraded, not self-reported: this comparison
|
||||
# cannot be made fairly, and saying so is the honest third answer.
|
||||
cand["verdict"] = INSUFFICIENT_WINDOW
|
||||
cand["stability"] = (
|
||||
f"insufficient-window against {snap.get('run_id')} (matched by {by}): "
|
||||
f"{parity['current_days']}d window read {curr_strength} against a "
|
||||
f"{parity['prior_days']}d window's measured "
|
||||
f"{((prior.get('measurement') or {}).get('share_of_plugin') or 0):.2%} — "
|
||||
"too narrow to compare fairly")
|
||||
cand.setdefault("reasons", []).append(
|
||||
f"DEFERRED ({GATE}): the current window ({parity['current_days']}d) is "
|
||||
f"narrower than the {parity['required_days']}d parity floor set by the "
|
||||
f"{parity['prior_days']}d window that produced the prior `measured` "
|
||||
"reading, so this window's weak reading cannot fairly contradict it. "
|
||||
"Neither filed nor downgraded to a boundary question this run; carried "
|
||||
"forward for a comparison over an adequately wide window.")
|
||||
deferrals.append({
|
||||
"gate": GATE,
|
||||
"candidate_id": cand.get("candidate_id"),
|
||||
"target": target,
|
||||
"skill": cand.get("skill"),
|
||||
"matched_by": by,
|
||||
"prior": {
|
||||
"run_id": snap.get("run_id"),
|
||||
"window": snap.get("window"),
|
||||
"measurement_strength": prior.get("measurement_strength"),
|
||||
"share_of_audited_spend": (prior.get("measurement") or {}).get("share_of_plugin"),
|
||||
},
|
||||
"current": {
|
||||
"window": window,
|
||||
"measurement_strength": curr_strength,
|
||||
"share_of_audited_spend": curr_share,
|
||||
},
|
||||
# What WOULD have been reported as instability, kept so the
|
||||
# deferral is auditable rather than an unexplained absence.
|
||||
"suppressed_flips": moved,
|
||||
"window_parity": parity,
|
||||
"carry_forward": (
|
||||
f"re-audit over a window of at least {parity['required_days']}d and "
|
||||
f"compare against {snap.get('run_id')}, which remains the standing "
|
||||
"comparison point — this window contributes no reading that could "
|
||||
"overturn it"),
|
||||
})
|
||||
continue
|
||||
|
||||
cand["verdict"] = "boundary-question"
|
||||
cand["stability"] = f"unstable against {snap.get('run_id')} (matched by {by})"
|
||||
cand.setdefault("reasons", []).append(
|
||||
@@ -206,6 +398,10 @@ def gate(doc, history, scan, explicit, issue_map):
|
||||
"share_of_audited_spend": curr_share,
|
||||
},
|
||||
"flips": moved,
|
||||
# Recorded on downgrades too, so a reader can see the widths this
|
||||
# was decided on and that the narrow-window carve-out was
|
||||
# considered and did not apply.
|
||||
"window_parity": parity,
|
||||
"headline": (
|
||||
f"{cand.get('candidate_id')} is {prior.get('measurement_strength')} at "
|
||||
f"{((prior.get('measurement') or {}).get('share_of_plugin') or 0):.2%} of spend in "
|
||||
@@ -226,14 +422,21 @@ def gate(doc, history, scan, explicit, issue_map):
|
||||
summary["boundary_questions"] = sum(1 for c in rows if c.get("verdict") == "boundary-question")
|
||||
summary["stability_checked"] = checked
|
||||
summary["stability_downgraded"] = len(findings)
|
||||
summary["stability_insufficient_window"] = len(deferrals)
|
||||
|
||||
doc["stability_gate"] = {
|
||||
"gate": GATE,
|
||||
"snapshots_available": len(history),
|
||||
"checked": checked,
|
||||
"downgraded": len(findings),
|
||||
"insufficient_window": len(deferrals),
|
||||
"window_parity_ratio": WINDOW_PARITY_RATIO,
|
||||
}
|
||||
doc["stability_findings"] = findings
|
||||
# Kept out of `stability_findings` deliberately: those are FR-6.5
|
||||
# self-reports the caller files against this plugin's repo, and a deferral
|
||||
# is not a defect to report — it is a comparison that was not made.
|
||||
doc["stability_deferrals"] = deferrals
|
||||
return doc
|
||||
|
||||
|
||||
|
||||
+29
-1
@@ -340,7 +340,7 @@ Hence the same shape of rule as §5:
|
||||
> windows land on opposite sides of the filing threshold, the candidate is not
|
||||
> filed.
|
||||
|
||||
Two boundaries make this a usable rule rather than a blanket refusal.
|
||||
Three boundaries make this a usable rule rather than a blanket refusal.
|
||||
|
||||
**Movement is not instability.** A candidate at 12% one window and 8% the next
|
||||
is stable: the human is told the same thing, and the difference is a trend, which
|
||||
@@ -356,6 +356,34 @@ finding out of missing history would mean no plugin could ever be audited once,
|
||||
which contradicts the principle that a plugin with no measured history stays
|
||||
auditable on its definition alone.
|
||||
|
||||
**A narrow window is not instability either — and this one was learned the hard
|
||||
way.** The rule above, applied symmetrically, downgraded the very candidate it
|
||||
was written for. `anxious/agent-wip/release-policy-derivation` reads `thin, 0%`
|
||||
over 7 days and `measured, 11.9%` over 27 — which looks like the §5b failure
|
||||
until you measure two more windows: 0 → 2 → 12 → 15 invocations at 7, 14, 27 and
|
||||
60 days. It runs about once every two days. The short window did not catch an
|
||||
unstable candidate; it failed to observe a real one, and the wider window kept
|
||||
raising the reading rather than diluting it. Punishing the candidate for that
|
||||
gets the artifact backwards: the *window* was the artifact.
|
||||
|
||||
So a weak reading only overturns a `measured` one when it was sampled over a
|
||||
comparably wide window — at least 90% of the width that produced the `measured`
|
||||
reading. Below that, the answer is neither "file" nor "boundary question" but a
|
||||
third, honest one: **`insufficient-window` — cannot tell either way, come back
|
||||
with a wider window.** Nothing is filed, nothing is self-reported, and the wider
|
||||
snapshot stays the standing comparison point so the next adequate run compares
|
||||
against it.
|
||||
|
||||
The parity is deliberately relative, not "at least N days": the adequate width
|
||||
is a fact about the candidate's invocation rate, which the auditor learns from
|
||||
the measurement rather than knowing in advance, so an absolute floor would be a
|
||||
guess about frequency dressed up as a threshold. And the carve-out is
|
||||
one-directional. A `thin` *prior* against a `measured` current is still a
|
||||
downgrade — there the earlier reading is the weak one — and so is a threshold
|
||||
crossing where both windows read `measured`, because a window that reached
|
||||
`measured` did observe the candidate; what moved was its share of spend, which
|
||||
is signal, not sampling.
|
||||
|
||||
The instability, when it fires, is a finding **about the auditor**. The target
|
||||
did not change; the tool described it two different ways. So it is reported
|
||||
against this plugin's own repo, not the target's — the same self-application
|
||||
|
||||
+52
-2
@@ -237,6 +237,42 @@ fact about the target. A candidate with no prior window to compare against is
|
||||
marked `unchecked` and files normally; an absence of history is not evidence of
|
||||
instability, and manufacturing one from it would contradict FR-2.4.
|
||||
|
||||
**FR-4.5.1 — Minimum window width for a downgrade.** A weak reading may only
|
||||
overturn a `measured` one when it was taken over a comparably wide window. Let
|
||||
`width(w) = w.until - w.since` in days, read off the window each reading was
|
||||
taken over (the observed session span the scan records, not the requested
|
||||
range). A comparison is **not read as instability**, and takes a third verdict
|
||||
`insufficient-window`, when all of the following hold:
|
||||
|
||||
1. the prior reading's `measurement_strength` is `measured`;
|
||||
2. the current reading's `measurement_strength` is `thin` or `unmeasured`;
|
||||
3. `width(current) < 0.9 × width(prior)`.
|
||||
|
||||
An `insufficient-window` candidate is **neither filed nor downgraded to a
|
||||
boundary question**, emits **no FR-6.5 self-report**, and is carried forward: the
|
||||
wider prior snapshot remains the standing comparison point, so the next audit
|
||||
over an adequately wide window compares against it and not against this
|
||||
window's under-sample. The run records the widths it decided on, the parity
|
||||
floor, and the flips it suppressed.
|
||||
|
||||
Three cases remain downgrades, and must not be excused by narrowness: the
|
||||
reverse direction (`thin`/`unmeasured` prior, `measured` current — the earlier
|
||||
reading is then the weak one); any flip between windows that satisfy the 0.9
|
||||
parity; and a threshold crossing in which **both** readings are `measured`,
|
||||
since a window that reached `measured` observed the candidate often enough for
|
||||
the share to be a claim about relative spend rather than a sampling failure.
|
||||
|
||||
The rule is stated as a ratio against the prior window rather than an absolute
|
||||
number of days because the adequate width is a property of the candidate's
|
||||
invocation rate, which the auditor does not know in advance:
|
||||
`anxious/agent-wip/release-policy-derivation` runs roughly once every two days
|
||||
and needs ~27 days; a once-per-hour step is fully measured in one. The ratio is
|
||||
0.9 rather than 1.0 because window bounds are observed session spans, so two
|
||||
nominally identical windows differ by hours. When either window does not carry
|
||||
both bounds, width is not computable, the flip is **not** excused, and the run
|
||||
records that the check could not be made — an unreadable window must be visible,
|
||||
never a silent exemption.
|
||||
|
||||
### FR-5 — Snapshot accumulation and diffing
|
||||
|
||||
**FR-5.1** Every run appends one machine-readable snapshot record to an
|
||||
@@ -323,7 +359,10 @@ ranked candidate table with boundary classification and value; the diff against
|
||||
the previous snapshot; the boundary questions posed to the user — both the
|
||||
low-confidence candidates and any downgraded by the FR-4.5 stability gate, the
|
||||
latter marked as a defect in the auditor rather than a fact about the
|
||||
target; and the issues filed or updated, in full `owner/repo#num` form,
|
||||
target; any candidate the gate could not judge either way (`insufficient-window`,
|
||||
FR-4.5.1), stated as a window too narrow to compare rather than as a finding
|
||||
about the target or the auditor; and the issues filed or updated, in full
|
||||
`owner/repo#num` form,
|
||||
separating those on the target's tracker (FR-6.1) from those on this plugin's
|
||||
own (FR-6.5).
|
||||
|
||||
@@ -488,7 +527,7 @@ is verified by hand (TASKS 7.9.5).
|
||||
| S1 | Audits any plugin by path or name, with no target-specific code | Run against `token-budget`, `worktree-discipline`, `anxious`, `memory` |
|
||||
| S2 | Correctly returns "nothing to offload" for a well-cut plugin | Run against `token-budget` → no `high`-confidence candidates |
|
||||
| S3 | Rediscovers a known-correct historical cut | Run against `worktree-discipline` with `bin/worktree-audit` removed from the inventory → must flag the classification step as `high` confidence |
|
||||
| S4 | Mechanically refuses a candidate whose evidence is a window artifact | The two-window `anxious` fixture → the FR-4.5 gate downgrades `agent-wip/release-policy-derivation`, with no human in the loop |
|
||||
| S4 | Mechanically refuses a candidate whose evidence is a window artifact | The two-window `anxious` fixture → the FR-4.5 gate refuses `agent-wip/release-policy-derivation` (as `insufficient-window`, per FR-4.5.1, since the 7-day current window cannot fairly contradict the 26-day `measured` one), with no human in the loop; a flip between comparably wide windows is still downgraded to a boundary question |
|
||||
| S5 | Re-running does not duplicate issues | Run twice; second run comments, does not file |
|
||||
| S6 | Trend across ≥2 snapshots is computable and volume-normalized | Two runs on different windows; diff reports per-invocation deltas |
|
||||
| S7 | Passes its own audit | FR-8.2 |
|
||||
@@ -505,6 +544,17 @@ that a human review caught a candidate whose evidence flipped between windows
|
||||
so the criterion is now that gate, run mechanically (FR-4.5), against the real
|
||||
two-window evidence that produced the original finding.
|
||||
|
||||
The *reason* S4's candidate is refused changed in v0.7.0, and the change is
|
||||
itself instructive. Two further real windows (14 and 60 days, same script, same
|
||||
candidate) showed 0 → 2 → 12 → 15 invocations as the window widened: monotonic
|
||||
accumulation of a low-frequency step, not a label flipping about. The 7-day
|
||||
window had not caught an unstable candidate, it had failed to observe a real one,
|
||||
and calling that instability blamed the candidate for the auditor's window. So
|
||||
FR-4.5.1 makes the refusal honest — `insufficient-window`, cannot tell either
|
||||
way, compare again over a wider window — while keeping the property S4 exists to
|
||||
test: nothing is filed off a window artifact, with no human in the loop.
|
||||
(kotkan/claude-plugin-inference-arbitrage#15.)
|
||||
|
||||
---
|
||||
|
||||
## 5. Signal catalog
|
||||
|
||||
@@ -38,6 +38,26 @@ edited by hand on a filed issue. Removing it from an issue causes the next audit
|
||||
to file a duplicate — which the spec names as the single most likely way for
|
||||
this plugin to become hated.
|
||||
|
||||
**Known failure mode (kotkan/claude-plugin-inference-arbitrage#23).** Two
|
||||
issues filed on 2026-07-29 (oleks/claude-plugin-cluster#56, #57) reached Gitea
|
||||
with the `<!-- -->` wrapper stripped — the body opened with the bare text
|
||||
`ia-candidate: <id>` instead. `bin/filing-plan` always *renders* the wrapped
|
||||
form (verified: a direct `issue_write` probe with a literal `<!-- -->` in the
|
||||
body round-trips through the Gitea MCP write path unchanged, so the API/MCP
|
||||
layer is not the cause). The loss happens on the **relay hop** —
|
||||
`Agent(anxious:issuer-agent)` composing the actual Gitea call from the prompt
|
||||
it's handed — because an HTML comment is, by design, invisible when the prompt
|
||||
text is read as markdown; an agent relaying "what the issue should say" rather
|
||||
than pasting the string byte-for-byte can drop a line it never visually saw.
|
||||
That is why `skills/offload-audit/SKILL.md` §5d now fences the rendered body
|
||||
in the delegation prompt instead of inlining it as bare markdown — a fenced
|
||||
block reads as literal text, not as renderable markup, to the relaying agent.
|
||||
`bin/filing-plan`'s `find_existing()` also now matches the bare, unwrapped
|
||||
form as a backward-compatibility fallback, so issues already filed without the
|
||||
wrapper (like cluster#56/#57) are still found — but that fallback exists only
|
||||
to avoid duplicate-filing on *already-broken* issues, not as a second
|
||||
acceptable format going forward.
|
||||
|
||||
## issue-title
|
||||
|
||||
A proposal. `issuer` may rewrite it.
|
||||
|
||||
@@ -78,15 +78,36 @@ with `--note` so the wiki page records that this run is missing from memory.
|
||||
|
||||
## 3. Gather candidates
|
||||
|
||||
From `plugin-inventory`: high verb ratio **with no `bin/` script behind it**
|
||||
(a conjunction — never the ratio alone), duplicated command blocks, rule tables,
|
||||
low script coverage. From `offload-scan`: high MTR with real spend, recurring
|
||||
n-grams, read amplification, retry density, fan-out. `judgment_density` is a
|
||||
**brake** — high judgment density with high spend means the plugin is healthy
|
||||
and must be reported that way.
|
||||
**Never hand-reshape `inv.json` and `scan.json` into a candidate table.**
|
||||
That reshaping used to be done by inference on every single audit run and was
|
||||
itself 40% of this plugin's own audited spend
|
||||
(kotkan/claude-plugin-inference-arbitrage#24). Run the join instead:
|
||||
|
||||
`aggregate.hook_prose_drift` entries are **not offload candidates.** Pass them
|
||||
with `"category": "hook-prose-drift"` so they come back as `drift-note`.
|
||||
```bash
|
||||
$IA/bin/candidate-digest inv.json scan.json --json > digest.json
|
||||
```
|
||||
|
||||
This produces one `CandidateRow` per skill/agent (`row_kind`, `name`,
|
||||
`share_of_spend`, `mtr`, `judgment_density`, `read_amplification`,
|
||||
`retry_density`, `fanout_multiplier`, `dominant_ngram_recurrences`,
|
||||
`has_bin_script`, `flags`). It is a **digest, not a verdict** — position 3, the
|
||||
same as this whole plugin's own thesis. A `flags` entry is a shape the digest
|
||||
noticed crossing the same thresholds `boundary-classify` already applies
|
||||
(`MIN_INVOCATIONS`, `MIN_ATTRIBUTED_TURNS`, `MIN_SHARE_OF_SPEND`) plus the
|
||||
`judgment_density` brake; it is **never** grounds to skip step 4's judgment or
|
||||
to file directly. Read every flagged row's actual n-gram/tool-call shape before
|
||||
deciding candidate vs. this audit's own necessary two-pass discipline — the
|
||||
overrule case `bin/boundary-classify`'s hard gate (§4 below) exists for.
|
||||
|
||||
Also read `plugin-inventory`'s own fields the digest does not carry: duplicated
|
||||
command blocks, rule tables, low script coverage. `judgment_density` is a
|
||||
**brake, not an accelerator** — a flagged row with real judgment density
|
||||
(`>= 0.3`) comes back unflagged, and high judgment density with high spend
|
||||
means the plugin is healthy and must be reported that way.
|
||||
|
||||
`aggregate.hook_prose_drift` entries are **not offload candidates** and never
|
||||
appear in `digest.json`. Pass them with `"category": "hook-prose-drift"` so
|
||||
they come back as `drift-note`.
|
||||
|
||||
## 4. Judge, then grade mechanically
|
||||
|
||||
@@ -154,6 +175,19 @@ changed, or whose share crossed the 2% filing threshold, between the two
|
||||
windows. **Use `stable.json` for steps 5 and 6.** Never file from
|
||||
`classified.json` once this has run.
|
||||
|
||||
One comparison it deliberately refuses to make (FR-4.5.1): when the prior window
|
||||
read `measured`, this window reads `thin`/`unmeasured`, **and** this window is
|
||||
narrower than 90% of that prior window's width, the candidate comes back with
|
||||
verdict **`insufficient-window`** and lands in `stability_deferrals`, not
|
||||
`stability_findings`. A low-frequency-but-real candidate looks thin in any short
|
||||
window — that is what kotkan/claude-plugin-inference-arbitrage#15 turned out to
|
||||
be — so this is neither a filing nor a defect: **do not file it on the target's
|
||||
repo, and do not self-report it on this plugin's repo.** Mention it in the run
|
||||
summary as "window too narrow to compare" and, if you want the answer, re-audit
|
||||
over at least the `carry_forward` width the deferral names. Flips between
|
||||
comparably wide windows, and the reverse direction (`thin` prior → `measured`
|
||||
now), still downgrade and still self-report.
|
||||
|
||||
If the store has no history for this target even after the wiki fetch above —
|
||||
a genuine first audit — every candidate comes back `stability: unchecked` and
|
||||
files normally. That is the FR-2.4 outcome, not a failure.
|
||||
@@ -248,14 +282,31 @@ rendered. **Execute the plan verbatim — do not rewrite a body, and never turn
|
||||
|
||||
One `Agent` call per action (or one call carrying all of them):
|
||||
|
||||
```text
|
||||
**The body MUST be embedded inside its own fenced code block in the prompt, not
|
||||
inlined as bare markdown.** kotkan/claude-plugin-inference-arbitrage#23 found
|
||||
that two issues filed this way (oleks/claude-plugin-cluster#56, #57) reached
|
||||
Gitea with the leading `<!-- ia-candidate: ... -->` marker's HTML-comment
|
||||
wrapper stripped. The write path itself was verified clean (a literal `<!--
|
||||
-->` round-trips through the Gitea MCP write unchanged); the loss happens on
|
||||
the relay hop, because an HTML comment is invisible when its surrounding text
|
||||
is read as markdown — `issuer-agent` can only paste a line byte-for-byte if it
|
||||
was actually shown one, not markdown that silently ate it. A fenced block
|
||||
(` ``` `) forces the body to be read and relayed as literal text.
|
||||
|
||||
````text
|
||||
Agent(subagent_type="anxious:issuer-agent", prompt=...)
|
||||
# for action == "file"
|
||||
"File this on <owner/repo>. It is an inference-arbitrage audit finding.
|
||||
Proposed title: <proposed_title> (yours to rewrite per your taxonomy)
|
||||
Body — file VERBATIM, the first line is an idempotency marker that a later
|
||||
audit searches for and MUST NOT be altered or reformatted:
|
||||
Body — copy the fenced block below byte-for-byte as the issue body. Do not
|
||||
retype, paraphrase, reformat, or 'clean up' it — the first line is an
|
||||
idempotency marker (an HTML comment, `<!-- ia-candidate: ... -->`) that a
|
||||
later audit's marker search depends on, and it renders invisible on Gitea
|
||||
by design, so you will not visually see it in the rendered issue — that is
|
||||
expected, do not treat it as accidental syntax to strip:
|
||||
```
|
||||
<body>
|
||||
```
|
||||
Labels: 'token-offload' is required — create it on this repo if absent
|
||||
(colour #8b5cf6, 'A step paying for inference where a deterministic script
|
||||
would do'). Add the four-axis taxonomy labels and any milestone per your own
|
||||
@@ -265,11 +316,15 @@ Agent(subagent_type="anxious:issuer-agent", prompt=...)
|
||||
|
||||
# for action == "comment"
|
||||
"Add this comment to <issue>, an existing inference-arbitrage finding being
|
||||
re-measured. Do not open a new issue. Post the body VERBATIM (the first line
|
||||
is an idempotency marker). If the issue is closed, use your judgement about
|
||||
reopening; the audit does not require it.
|
||||
<body>"
|
||||
```
|
||||
re-measured. Do not open a new issue. Copy the fenced block below
|
||||
byte-for-byte as the comment body — do not retype or reformat it, the first
|
||||
line is an invisible-by-design HTML-comment idempotency marker (see the
|
||||
'file' case above for why). If the issue is closed, use your judgement
|
||||
about reopening; the audit does not require it.
|
||||
```
|
||||
<body>
|
||||
```"
|
||||
````
|
||||
|
||||
`issuer` owns repo, final title, labels beyond `token-offload`, and milestone.
|
||||
You own the body.
|
||||
|
||||
@@ -50,7 +50,7 @@ rather than in agent prose (FR-8.1 — the plugin obeying its own thesis).
|
||||
| --- | --- | --- |
|
||||
| 1 (S2) | `token-budget` | zero `high`-confidence candidates; conclusion "nothing to offload here" |
|
||||
| 2 (S3) | `worktree-discipline`, masked | `sweep-worktrees` worktree-safety classification at `high`, position `llm-over-script-digest` |
|
||||
| 3 (S4) | `anxious`, two windows | `release-policy-derivation` downgraded to a boundary question by the FR-4.5 stability gate, with no human in the loop |
|
||||
| 3 (S4) | `anxious`, two windows | `release-policy-derivation` refused by the FR-4.5 stability gate — as `insufficient-window` (FR-4.5.1), the 7-day window being too narrow to contradict the 26-day `measured` one — with no human in the loop |
|
||||
|
||||
## The two-window `anxious` fixture — gate 3
|
||||
|
||||
@@ -68,11 +68,32 @@ actually got filed, not a case constructed to be caught:
|
||||
- `anxious-7d.scan.json` — the 7-day scan, for signature identity. Masked
|
||||
shapes and counts only (FR-3.5); no transcript content.
|
||||
|
||||
- `anxious-60d.scan.json` — the real 60-day scan
|
||||
(`bin/offload-scan --plugin anxious --days 60`, 2026-06-26 → 2026-07-30, 176
|
||||
sessions), added while resolving
|
||||
kotkan/claude-plugin-inference-arbitrage#15. Same masking rules.
|
||||
- `anxious-60d.classified.json` — the same candidate's unchanged rubric grade
|
||||
with the 60-day window's real measurement: 15 invocations, `offload_value`
|
||||
4,953,520 of 35,722,973 weighted tokens = 13.87% of audited spend, `measured`.
|
||||
It carries only that one candidate, so no other row can be read as a 60-day
|
||||
number it was not measured over.
|
||||
|
||||
The test excludes the 7-day run from the store so it is the run being
|
||||
classified, exactly as at filing time, and asserts the gate downgrades it. The
|
||||
comparison is symmetric: classifying the 30-day window against the 7-day
|
||||
snapshot downgrades it too. Both windows cannot be right, which is the finding —
|
||||
recorded as kotkan/claude-plugin-inference-arbitrage#11.
|
||||
classified, exactly as at filing time, and asserts the gate **refuses** it. What
|
||||
it does *not* assert any more is that the refusal is a downgrade. The comparison
|
||||
is not symmetric, and treating it as symmetric was the bug behind
|
||||
kotkan/claude-plugin-inference-arbitrage#15: measuring two further real windows
|
||||
gave 0 → 2 → 12 → 15 invocations at 7, 14, 27 and 60 days, i.e. a candidate that
|
||||
runs about once every two days and that a 7-day window simply never sees. So the
|
||||
7-day-against-26-day pairing is now `insufficient-window` (FR-4.5.1) with no
|
||||
self-report, while the 60-day-against-26-day pairing — two adequately wide
|
||||
windows, `measured` both times, same side of the 2% threshold — is `stable` and
|
||||
files normally. Both directions are asserted, so a regression in either shows up
|
||||
as a test failure rather than as a wrong issue on someone's tracker.
|
||||
|
||||
The original two-window finding, and the fact that both windows cannot be right
|
||||
about a candidate they sample equally, is recorded as
|
||||
kotkan/claude-plugin-inference-arbitrage#11.
|
||||
|
||||
The candidate matches by **slug**, not signature: the 7-day scan produced no
|
||||
recurring n-gram for that skill, so its signature falls back to the slug and the
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
{
|
||||
"candidates": [
|
||||
{
|
||||
"boundary_confidence": "high",
|
||||
"candidate_id": "anxious/agent-wip/release-policy-derivation",
|
||||
"category": "usage",
|
||||
"determinism_tests": {
|
||||
"T1": true,
|
||||
"T2": true,
|
||||
"T3": true,
|
||||
"T4": true,
|
||||
"T5": true
|
||||
},
|
||||
"falsifiability_triple": {
|
||||
"examples": true,
|
||||
"overrule_case": true,
|
||||
"signature": true
|
||||
},
|
||||
"measurement_strength": "measured",
|
||||
"offload_value": 4953520,
|
||||
"position": "llm-over-script-digest",
|
||||
"reasons": [
|
||||
"all five determinism tests pass; triple complete; escalation path defined",
|
||||
"measured over the 60-day window: 15 invocations, offload_value 4,953,520 weighted tokens, 13.87% of audited spend"
|
||||
],
|
||||
"share_of_audited_spend": 0.1387,
|
||||
"skill": "anxious:agent-wip",
|
||||
"verdict": "file"
|
||||
}
|
||||
],
|
||||
"rubric_version": "1.0.0",
|
||||
"summary": {
|
||||
"boundary_questions": 0,
|
||||
"candidates": 1,
|
||||
"conclusion": "1 high-confidence candidate(s)",
|
||||
"drift_notes": 0,
|
||||
"high_confidence": 1,
|
||||
"no_offload": 0,
|
||||
"to_file": 1
|
||||
},
|
||||
"target": "anxious (60-day window)"
|
||||
}
|
||||
@@ -0,0 +1,335 @@
|
||||
{
|
||||
"window": {
|
||||
"since": "2026-06-26T17:41:40.988000+00:00",
|
||||
"until": "2026-07-30T02:30:59.111000+00:00",
|
||||
"requested_since": "2026-05-31T02:30:03.118644+00:00",
|
||||
"requested_until": null,
|
||||
"sessions": 176
|
||||
},
|
||||
"plugin": "anxious",
|
||||
"tool_versions": {
|
||||
"cc_tokens_path": "/home/oleks/projects/claude-plugins/token-budget/bin/cc-tokens"
|
||||
},
|
||||
"coverage": {
|
||||
"attributed_turns": 5197,
|
||||
"candidate_turns": 249648,
|
||||
"ratio": 0.021,
|
||||
"caveat": "hook-driven and unattributed agent turns excluded"
|
||||
},
|
||||
"totals": {
|
||||
"weighted_tokens": 35722973,
|
||||
"cost_estimate_usd": 178.61,
|
||||
"invocations": 700,
|
||||
"mechanical_share": 0.106
|
||||
},
|
||||
"skills": [
|
||||
{
|
||||
"skill": "anxious:agent-wip",
|
||||
"invocations": 15,
|
||||
"turns": 375,
|
||||
"sidechain_turns": 0,
|
||||
"weighted_tokens": 14007053,
|
||||
"cost_estimate_usd": 70.04,
|
||||
"share_of_plugin": 0.392,
|
||||
"mtr": 0.144,
|
||||
"judgment_density": 0.027,
|
||||
"retry_density": 0.069,
|
||||
"offload_waste": 1532536,
|
||||
"read_amplification": 383.83,
|
||||
"fanout_multiplier": 1.0,
|
||||
"repetition_factor": 3.322,
|
||||
"ngrams": [
|
||||
{
|
||||
"sig": [
|
||||
"label_write(color=<str>, method=<str>, name=<str>, owner=<str>, repo=<str>)",
|
||||
"label_write(color=<str>, method=<str>, name=<str>, owner=<str>, repo=<str>)",
|
||||
"issue_write(index=<n>, labels=<list>, method=<str>, owner=<str>, repo=<str>)"
|
||||
],
|
||||
"recurrences": 5,
|
||||
"invocations": 4,
|
||||
"waste": 345101
|
||||
},
|
||||
{
|
||||
"sig": [
|
||||
"Edit(file_path=<path>, new_string=<str>, old_string=<str>, replace_all=<bool>)",
|
||||
"Read(file_path=<path>)",
|
||||
"Edit(file_path=<path>, new_string=<str>, old_string=<str>, replace_all=<bool>)"
|
||||
],
|
||||
"recurrences": 3,
|
||||
"invocations": 3,
|
||||
"waste": 342125
|
||||
},
|
||||
{
|
||||
"sig": [
|
||||
"Read(file_path=<path>)",
|
||||
"Read(file_path=<path>)",
|
||||
"Bash(command=grep <str>, description=<str>)"
|
||||
],
|
||||
"recurrences": 3,
|
||||
"invocations": 3,
|
||||
"waste": 302043
|
||||
},
|
||||
{
|
||||
"sig": [
|
||||
"Bash(command=echo <str>, description=<str>)",
|
||||
"issue_read(index=<n>, method=<str>, owner=<str>, repo=<str>)",
|
||||
"issue_read(index=<n>, method=<str>, owner=<str>, repo=<str>)"
|
||||
],
|
||||
"recurrences": 3,
|
||||
"invocations": 3,
|
||||
"waste": 299691
|
||||
},
|
||||
{
|
||||
"sig": [
|
||||
"ToolSearch(max_results=<n>, query=<str>)",
|
||||
"issue_read(index=<n>, method=<str>, owner=<str>, repo=<str>)",
|
||||
"issue_read(index=<n>, method=<str>, owner=<str>, repo=<str>)"
|
||||
],
|
||||
"recurrences": 4,
|
||||
"invocations": 4,
|
||||
"waste": 258683
|
||||
}
|
||||
],
|
||||
"offload_value": 4953520
|
||||
},
|
||||
{
|
||||
"skill": "anxious:delivery-flow",
|
||||
"invocations": 1,
|
||||
"turns": 1,
|
||||
"sidechain_turns": 0,
|
||||
"weighted_tokens": 48152,
|
||||
"cost_estimate_usd": 0.24,
|
||||
"share_of_plugin": 0.001,
|
||||
"mtr": 0.0,
|
||||
"judgment_density": 0.0,
|
||||
"retry_density": 0.0,
|
||||
"offload_waste": 0,
|
||||
"read_amplification": 378.0,
|
||||
"fanout_multiplier": 1.0,
|
||||
"repetition_factor": 1.0,
|
||||
"ngrams": [],
|
||||
"offload_value": 0
|
||||
}
|
||||
],
|
||||
"agents": [
|
||||
{
|
||||
"agent": "anxious:issuer-agent",
|
||||
"invocations": 608,
|
||||
"turns": 4207,
|
||||
"sidechain_turns": 4207,
|
||||
"weighted_tokens": 13682748,
|
||||
"cost_estimate_usd": 68.41,
|
||||
"share_of_plugin": 0.383,
|
||||
"mtr": 0.097,
|
||||
"judgment_density": 0.051,
|
||||
"retry_density": 0.082,
|
||||
"offload_waste": 778919,
|
||||
"read_amplification": 545.98,
|
||||
"fanout_multiplier": 0.2,
|
||||
"repetition_factor": 7.741,
|
||||
"ngrams": [
|
||||
{
|
||||
"sig": [
|
||||
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)",
|
||||
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)",
|
||||
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)"
|
||||
],
|
||||
"recurrences": 107,
|
||||
"invocations": 47,
|
||||
"waste": 874165
|
||||
},
|
||||
{
|
||||
"sig": [
|
||||
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)",
|
||||
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)",
|
||||
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)",
|
||||
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)"
|
||||
],
|
||||
"recurrences": 56,
|
||||
"invocations": 25,
|
||||
"waste": 496902
|
||||
},
|
||||
{
|
||||
"sig": [
|
||||
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)",
|
||||
"issue_read(index=<n>, method=<str>, owner=<str>, repo=<str>)",
|
||||
"issue_read(index=<n>, method=<str>, owner=<str>, repo=<str>)"
|
||||
],
|
||||
"recurrences": 28,
|
||||
"invocations": 26,
|
||||
"waste": 318874
|
||||
},
|
||||
{
|
||||
"sig": [
|
||||
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)",
|
||||
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)",
|
||||
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)",
|
||||
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)",
|
||||
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)"
|
||||
],
|
||||
"recurrences": 31,
|
||||
"invocations": 10,
|
||||
"waste": 207299
|
||||
},
|
||||
{
|
||||
"sig": [
|
||||
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)",
|
||||
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)",
|
||||
"issue_read(index=<n>, method=<str>, owner=<str>, repo=<str>)"
|
||||
],
|
||||
"recurrences": 13,
|
||||
"invocations": 13,
|
||||
"waste": 184334
|
||||
}
|
||||
],
|
||||
"offload_value": 5722452
|
||||
},
|
||||
{
|
||||
"agent": "repo-worker",
|
||||
"invocations": 2,
|
||||
"turns": 268,
|
||||
"sidechain_turns": 268,
|
||||
"weighted_tokens": 1816949,
|
||||
"cost_estimate_usd": 9.08,
|
||||
"share_of_plugin": 0.051,
|
||||
"mtr": 0.19,
|
||||
"judgment_density": 0.004,
|
||||
"retry_density": 0.075,
|
||||
"offload_waste": 303294,
|
||||
"read_amplification": 650.6,
|
||||
"fanout_multiplier": 0.66,
|
||||
"repetition_factor": 1.0,
|
||||
"ngrams": [],
|
||||
"offload_value": 302081
|
||||
},
|
||||
{
|
||||
"agent": "anxious:steward-agent",
|
||||
"invocations": 56,
|
||||
"turns": 225,
|
||||
"sidechain_turns": 225,
|
||||
"weighted_tokens": 5597218,
|
||||
"cost_estimate_usd": 27.99,
|
||||
"share_of_plugin": 0.157,
|
||||
"mtr": 0.009,
|
||||
"judgment_density": 0.08,
|
||||
"retry_density": 0.076,
|
||||
"offload_waste": 17835,
|
||||
"read_amplification": 677.33,
|
||||
"fanout_multiplier": 0.34,
|
||||
"repetition_factor": 7.728,
|
||||
"ngrams": [
|
||||
{
|
||||
"sig": [
|
||||
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)",
|
||||
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)",
|
||||
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)"
|
||||
],
|
||||
"recurrences": 106,
|
||||
"invocations": 29,
|
||||
"waste": 2987563
|
||||
},
|
||||
{
|
||||
"sig": [
|
||||
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)",
|
||||
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)",
|
||||
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)",
|
||||
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)"
|
||||
],
|
||||
"recurrences": 77,
|
||||
"invocations": 22,
|
||||
"waste": 2434027
|
||||
},
|
||||
{
|
||||
"sig": [
|
||||
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)",
|
||||
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)",
|
||||
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)",
|
||||
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)",
|
||||
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)"
|
||||
],
|
||||
"recurrences": 55,
|
||||
"invocations": 15,
|
||||
"waste": 1816278
|
||||
},
|
||||
{
|
||||
"sig": [
|
||||
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)",
|
||||
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)",
|
||||
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)",
|
||||
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)",
|
||||
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)",
|
||||
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)"
|
||||
],
|
||||
"recurrences": 40,
|
||||
"invocations": 8,
|
||||
"waste": 1196514
|
||||
},
|
||||
{
|
||||
"sig": [
|
||||
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)",
|
||||
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)",
|
||||
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)"
|
||||
],
|
||||
"recurrences": 5,
|
||||
"invocations": 3,
|
||||
"waste": 205780
|
||||
}
|
||||
],
|
||||
"offload_value": 126805
|
||||
},
|
||||
{
|
||||
"agent": "anxious:docs-agent",
|
||||
"invocations": 10,
|
||||
"turns": 98,
|
||||
"sidechain_turns": 98,
|
||||
"weighted_tokens": 502645,
|
||||
"cost_estimate_usd": 2.51,
|
||||
"share_of_plugin": 0.014,
|
||||
"mtr": 0.276,
|
||||
"judgment_density": 0.061,
|
||||
"retry_density": 0.051,
|
||||
"offload_waste": 107413,
|
||||
"read_amplification": 164.08,
|
||||
"fanout_multiplier": 0.34,
|
||||
"repetition_factor": 1.0,
|
||||
"ngrams": [],
|
||||
"offload_value": 100861
|
||||
},
|
||||
{
|
||||
"agent": "feature-dev:code-reviewer",
|
||||
"invocations": 3,
|
||||
"turns": 4,
|
||||
"sidechain_turns": 4,
|
||||
"weighted_tokens": 26969,
|
||||
"cost_estimate_usd": 0.13,
|
||||
"share_of_plugin": 0.001,
|
||||
"mtr": 0.75,
|
||||
"judgment_density": 0.25,
|
||||
"retry_density": 0.0,
|
||||
"offload_waste": 13850,
|
||||
"read_amplification": 797.8,
|
||||
"fanout_multiplier": 0.09,
|
||||
"repetition_factor": 1.0,
|
||||
"ngrams": [],
|
||||
"offload_value": 10388
|
||||
},
|
||||
{
|
||||
"agent": "cluster:gitea-agent",
|
||||
"invocations": 5,
|
||||
"turns": 19,
|
||||
"sidechain_turns": 19,
|
||||
"weighted_tokens": 41235,
|
||||
"cost_estimate_usd": 0.21,
|
||||
"share_of_plugin": 0.001,
|
||||
"mtr": 0.105,
|
||||
"judgment_density": 0.158,
|
||||
"retry_density": 0.0,
|
||||
"offload_waste": 4163,
|
||||
"read_amplification": 299.0,
|
||||
"fanout_multiplier": 0.24,
|
||||
"repetition_factor": 1.0,
|
||||
"ngrams": [],
|
||||
"offload_value": 3505
|
||||
}
|
||||
]
|
||||
}
|
||||
Executable
+163
@@ -0,0 +1,163 @@
|
||||
#!/usr/bin/env bash
|
||||
# bin/candidate-digest: the inv.json/scan.json join
|
||||
# (kotkan/claude-plugin-inference-arbitrage#24). Synthetic fixtures for the
|
||||
# three worked examples from the issue body, plus the has_bin_script suppression.
|
||||
set -euo pipefail
|
||||
cd "$(dirname "${BASH_SOURCE[0]}")"
|
||||
|
||||
BIN=../bin/candidate-digest
|
||||
tmp=$(mktemp -d)
|
||||
trap 'rm -rf "$tmp"' EXIT
|
||||
fail=0
|
||||
|
||||
field() {
|
||||
python3 -c "import json,sys;print(json.load(open(sys.argv[1]))[int(sys.argv[2])][sys.argv[3]])" "$@"
|
||||
}
|
||||
|
||||
check() {
|
||||
local what=$1 got=$2 want=$3
|
||||
if [ "$got" = "$want" ]; then
|
||||
echo " ok $what = $want"
|
||||
else
|
||||
echo " FAIL $what = $got (want $want)" >&2
|
||||
fail=1
|
||||
fi
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Example 1: verb_ratio=0.71 skill backed by a bin/ script -> has_bin_script
|
||||
# True, flags=[] regardless of its dynamic numbers (conjunction fails).
|
||||
# Example 2: agent row, mtr=0.17 share=0.40 judgment_density=0.034 with a
|
||||
# dominant recurring 8-gram -> flags=['high-share-low-judgment-recurring-ngram'].
|
||||
# Example 3 (EDGE): same mtr/share/ngram but judgment_density=0.55 -> flags=[]
|
||||
# (the judgment_density brake suppresses the false positive).
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
cat >"$tmp/inv.json" <<'EOF'
|
||||
{
|
||||
"plugin": {"name": "synthetic"},
|
||||
"skills": [
|
||||
{
|
||||
"name": "syn:scripted-skill",
|
||||
"verb_ratio": 0.71,
|
||||
"command_blocks": [{"sig": "${CLAUDE_PLUGIN_ROOT}/bin/do-the-thing --json", "count": 1}]
|
||||
}
|
||||
]
|
||||
}
|
||||
EOF
|
||||
|
||||
cat >"$tmp/scan.json" <<'EOF'
|
||||
{
|
||||
"skills": [
|
||||
{
|
||||
"skill": "syn:scripted-skill",
|
||||
"invocations": 12,
|
||||
"turns": 90,
|
||||
"share_of_plugin": 0.5,
|
||||
"mtr": 0.8,
|
||||
"judgment_density": 0.05,
|
||||
"read_amplification": 10.0,
|
||||
"retry_density": 0.0,
|
||||
"fanout_multiplier": 1.0,
|
||||
"ngrams": [{"sig": ["a", "b", "c"], "recurrences": 9, "invocations": 5, "waste": 1000}]
|
||||
}
|
||||
],
|
||||
"agents": [
|
||||
{
|
||||
"agent": "syn:flagged-agent",
|
||||
"invocations": 26,
|
||||
"turns": 208,
|
||||
"share_of_plugin": 0.40,
|
||||
"mtr": 0.17,
|
||||
"judgment_density": 0.034,
|
||||
"read_amplification": 4.0,
|
||||
"retry_density": 0.02,
|
||||
"fanout_multiplier": 1.0,
|
||||
"ngrams": [{"sig": ["x", "y"], "recurrences": 8, "invocations": 6, "waste": 800000}]
|
||||
},
|
||||
{
|
||||
"agent": "syn:healthy-agent",
|
||||
"invocations": 26,
|
||||
"turns": 208,
|
||||
"share_of_plugin": 0.40,
|
||||
"mtr": 0.17,
|
||||
"judgment_density": 0.55,
|
||||
"read_amplification": 4.0,
|
||||
"retry_density": 0.02,
|
||||
"fanout_multiplier": 1.0,
|
||||
"ngrams": [{"sig": ["x", "y"], "recurrences": 8, "invocations": 6, "waste": 800000}]
|
||||
}
|
||||
]
|
||||
}
|
||||
EOF
|
||||
|
||||
$BIN "$tmp/inv.json" "$tmp/scan.json" >"$tmp/out.json"
|
||||
|
||||
echo "== example 1: scripted skill, high mtr/share, has_bin_script suppresses everything =="
|
||||
row=$(python3 -c "
|
||||
import json
|
||||
rows = json.load(open('$tmp/out.json'))
|
||||
print([i for i, r in enumerate(rows) if r['name'] == 'syn:scripted-skill'][0])
|
||||
")
|
||||
check "row_kind" "$(field "$tmp/out.json" "$row" row_kind)" "skill"
|
||||
check "has_bin_script" "$(field "$tmp/out.json" "$row" has_bin_script)" "True"
|
||||
check "flags" "$(field "$tmp/out.json" "$row" flags)" "[]"
|
||||
|
||||
echo "== example 2: agent, judgment_density=0.034 -> flagged =="
|
||||
row=$(python3 -c "
|
||||
import json
|
||||
rows = json.load(open('$tmp/out.json'))
|
||||
print([i for i, r in enumerate(rows) if r['name'] == 'syn:flagged-agent'][0])
|
||||
")
|
||||
check "row_kind" "$(field "$tmp/out.json" "$row" row_kind)" "agent"
|
||||
check "has_bin_script" "$(field "$tmp/out.json" "$row" has_bin_script)" "False"
|
||||
check "dominant_ngram_recurrences" "$(field "$tmp/out.json" "$row" dominant_ngram_recurrences)" "8"
|
||||
check "flags" "$(field "$tmp/out.json" "$row" flags)" "['high-share-low-judgment-recurring-ngram']"
|
||||
|
||||
echo "== example 3 (EDGE): same mtr/share/ngram, judgment_density=0.55 -> brake suppresses =="
|
||||
row=$(python3 -c "
|
||||
import json
|
||||
rows = json.load(open('$tmp/out.json'))
|
||||
print([i for i, r in enumerate(rows) if r['name'] == 'syn:healthy-agent'][0])
|
||||
")
|
||||
check "flags" "$(field "$tmp/out.json" "$row" flags)" "[]"
|
||||
|
||||
echo "== below MIN_SHARE_OF_SPEND -> never flagged even with a low judgment_density =="
|
||||
cat >"$tmp/scan-thin.json" <<'EOF'
|
||||
{
|
||||
"skills": [],
|
||||
"agents": [
|
||||
{
|
||||
"agent": "syn:thin-agent",
|
||||
"invocations": 12,
|
||||
"turns": 90,
|
||||
"share_of_plugin": 0.005,
|
||||
"mtr": 0.9,
|
||||
"judgment_density": 0.01,
|
||||
"read_amplification": 3.0,
|
||||
"retry_density": 0.0,
|
||||
"fanout_multiplier": 1.0,
|
||||
"ngrams": [{"sig": ["p", "q"], "recurrences": 5, "invocations": 4, "waste": 100}]
|
||||
}
|
||||
]
|
||||
}
|
||||
EOF
|
||||
cat >"$tmp/inv-empty.json" <<'EOF'
|
||||
{"plugin": {"name": "synthetic"}, "skills": []}
|
||||
EOF
|
||||
$BIN "$tmp/inv-empty.json" "$tmp/scan-thin.json" >"$tmp/out-thin.json"
|
||||
check "flags" "$(field "$tmp/out-thin.json" 0 flags)" "[]"
|
||||
|
||||
echo "== schema: every CandidateRow key present =="
|
||||
python3 -c "
|
||||
import json
|
||||
rows = json.load(open('$tmp/out.json'))
|
||||
want = {'row_kind', 'name', 'share_of_spend', 'mtr', 'judgment_density',
|
||||
'read_amplification', 'retry_density', 'fanout_multiplier',
|
||||
'dominant_ngram_recurrences', 'has_bin_script', 'flags'}
|
||||
for r in rows:
|
||||
assert set(r.keys()) == want, (r.keys(), want)
|
||||
print('ok')
|
||||
"
|
||||
|
||||
exit "$fail"
|
||||
@@ -112,6 +112,43 @@ check "found via the comment" "$(q "$tmp/plan3.json" "d['actions'][0]['action']"
|
||||
check "closed state surfaced for issuer" \
|
||||
"$(q "$tmp/plan3.json" "d['actions'][0]['issue_state']")" "closed"
|
||||
|
||||
echo "== kotkan/claude-plugin-inference-arbitrage#23: a marker that lost its"
|
||||
echo " HTML-comment wrapper on write (oleks/claude-plugin-cluster#56, #57)"
|
||||
echo " is still found — backward compat, not a second accepted format =="
|
||||
python3 - "$tmp/plan1.json" "$tmp/bare.json" <<'PY'
|
||||
import json, sys
|
||||
plan = json.load(open(sys.argv[1]))
|
||||
filed = next(a for a in plan["actions"] if a["action"] == "file")
|
||||
# Same body, but the first line has lost its "<!-- " / " -->" wrapper — the
|
||||
# exact shape observed on the live tracker.
|
||||
bare_body = filed["body"].replace(
|
||||
"<!-- ia-candidate: fixture-plugin/skill-a/label-derivation -->",
|
||||
"ia-candidate: fixture-plugin/skill-a/label-derivation",
|
||||
)
|
||||
json.dump([{"number": 56, "state": "closed", "title": filed["proposed_title"],
|
||||
"body": bare_body, "comments": []}], open(sys.argv[2], "w"))
|
||||
PY
|
||||
plan "$tmp/bare.json" "$tmp/plan-bare.json"
|
||||
check "bare marker still found, no duplicate filed" \
|
||||
"$(q "$tmp/plan-bare.json" "d['summary']['to_file']")" "0"
|
||||
check "recognized as a comment on the existing issue" \
|
||||
"$(q "$tmp/plan-bare.json" "d['actions'][0]['action']")" "comment"
|
||||
check "on issue 56, matching the observed failure" \
|
||||
"$(q "$tmp/plan-bare.json" "d['actions'][0]['issue_number']")" "56"
|
||||
|
||||
echo "== a bare marker for a DIFFERENT id still does not false-match =="
|
||||
python3 - "$tmp/bare.json" "$tmp/bare-other.json" <<'PY'
|
||||
import json, sys
|
||||
issues = json.load(open(sys.argv[1]))
|
||||
issues[0]["body"] = issues[0]["body"].replace(
|
||||
"fixture-plugin/skill-a/label-derivation", "fixture-plugin/skill-a/label-derivation-longer",
|
||||
)
|
||||
json.dump(issues, open(sys.argv[2], "w"))
|
||||
PY
|
||||
plan "$tmp/bare-other.json" "$tmp/plan-bare-other.json"
|
||||
check "different id (even as a prefix) still files fresh" \
|
||||
"$(q "$tmp/plan-bare-other.json" "d['summary']['to_file']")" "1"
|
||||
|
||||
echo "== real Gitea API shapes parse, and a DIFFERENT id does not false-match =="
|
||||
plan tests/fixtures/gitea-issues.json "$tmp/plan4.json"
|
||||
check "unrelated markers -> files" "$(q "$tmp/plan4.json" "d['summary']['to_file']")" "1"
|
||||
|
||||
@@ -43,6 +43,16 @@ check "agent attributed via attributionAgent" "d['agents'][0]['agent'] == 'fixtu
|
||||
# --- 3.2 invocation reconstruction ----------------------------------------
|
||||
check "idle gap splits mine-me into 3 invocations" "$(skill mine-me)['invocations'] == 3"
|
||||
check "think-hard is one contiguous invocation" "$(skill think-hard)['invocations'] == 1"
|
||||
# kotkan/claude-plugin-inference-arbitrage#12: reconstruct_invocations() only
|
||||
# sees already-attributed turns and splits on wall-clock idle gap alone, so
|
||||
# it cannot tell two genuinely separate same-skill calls apart from one
|
||||
# contiguous call if they land inside the gap. The emitted totals must say so
|
||||
# rather than let the headline invocation/cost-per-invocation figures imply
|
||||
# more confidence than the reconstruction can support.
|
||||
check "totals carry a lower-bound caveat naming the idle gap threshold" "
|
||||
'lower bound' in d['totals']['invocation_caveat']
|
||||
and '10 minutes' in d['totals']['invocation_caveat']
|
||||
"
|
||||
|
||||
# --- 3.3 turn classification ----------------------------------------------
|
||||
check "mine-me is wholly mechanical" "$(skill mine-me)['mtr'] == 1.0"
|
||||
|
||||
@@ -317,6 +317,32 @@ else
|
||||
fail=1
|
||||
fi
|
||||
|
||||
echo "== kotkan/claude-plugin-inference-arbitrage#12: scan's invocation_caveat reaches the summary page =="
|
||||
store6=$tmp/store6
|
||||
inventory "$tmp/inv6.json" 1.0.0 steady
|
||||
scan "$tmp/scan6.json" 10000000 "steady:20:2000000:list_issues,issue_read"
|
||||
python3 -c "
|
||||
import json
|
||||
d = json.load(open('$tmp/scan6.json'))
|
||||
d['totals']['invocation_caveat'] = ('invocation counts are a lower bound: two same-skill '
|
||||
'calls within 10 minutes of each other on the wall clock are merged into one reported '
|
||||
'invocation')
|
||||
json.dump(d, open('$tmp/scan6.json', 'w'))
|
||||
"
|
||||
classified "$tmp/cls6.json" "synth/steady/a:steady:2000000:0.20:file"
|
||||
$BIN --store "$store6" write --target synth --run-id 2026-07-01T00:00:00Z \
|
||||
--from "$tmp/inv6.json" "$tmp/scan6.json" --classified "$tmp/cls6.json" >/dev/null
|
||||
check "invocation_caveat stored in totals" "$(python3 -c "
|
||||
import json
|
||||
s=[json.loads(l) for l in open('$store6/Data/synth/snapshots.jsonl')][0]
|
||||
print('lower bound' in s['totals']['invocation_caveat'])")" "True"
|
||||
if grep -qF "lower bound" "$store6/Audits/synth/2026-07-01.md"; then
|
||||
echo " ok invocation caveat rendered in the audit page"
|
||||
else
|
||||
echo " FAIL invocation caveat missing from the audit page" >&2
|
||||
fail=1
|
||||
fi
|
||||
|
||||
echo "== a re-run of the same run_id is refused =="
|
||||
if $BIN --store "$store" write --target synth --run-id 2026-07-08T00:00:00Z \
|
||||
--from "$tmp/inv2.json" "$tmp/scan2.json" --classified "$tmp/cls2.json" >/dev/null 2>&1; then
|
||||
|
||||
+185
-12
@@ -33,6 +33,16 @@ gatefield() {
|
||||
python3 -c "import json,sys;print(json.load(open(sys.argv[1]))['stability_gate'][sys.argv[2]])" "$@"
|
||||
}
|
||||
|
||||
# scanwin <file> <since> <until>
|
||||
# The only field the gate reads out of a scan for the FR-4.5 parity rule is the
|
||||
# window's observed span, so the fixture carries exactly that.
|
||||
scanwin() {
|
||||
python3 -c "
|
||||
import json, sys
|
||||
json.dump({'window': {'since': sys.argv[2], 'until': sys.argv[3], 'sessions': 10}},
|
||||
open(sys.argv[1], 'w'))" "$@"
|
||||
}
|
||||
|
||||
# classified <file> <strength> <share>
|
||||
# One position-3 candidate the boundary gate already graded `file`.
|
||||
classified() {
|
||||
@@ -59,17 +69,18 @@ json.dump({
|
||||
EOF
|
||||
}
|
||||
|
||||
# snapshot-store <store> <strength> <share>
|
||||
# A prior snapshot recording the same candidate by slug identity.
|
||||
# snapshot-store <store> <strength> <share> [since] [until]
|
||||
# A prior snapshot recording the same candidate by slug identity. The window
|
||||
# defaults to 30 days; the width matters for the FR-4.5 parity rule (cases j-m).
|
||||
prior() {
|
||||
local store=$1
|
||||
mkdir -p "$store/Data/synth"
|
||||
python3 - "$store/Data/synth/snapshots.jsonl" "$2" "$3" <<'EOF'
|
||||
python3 - "$store/Data/synth/snapshots.jsonl" "$2" "$3" "${4:-2026-06-01}" "${5:-2026-07-01}" <<'EOF'
|
||||
import json, sys
|
||||
out, strength, share = sys.argv[1:]
|
||||
out, strength, share, since, until = sys.argv[1:]
|
||||
json.dump({
|
||||
"run_id": "2026-07-01T00-00-00Z",
|
||||
"window": {"since": "2026-06-01", "until": "2026-07-01"},
|
||||
"window": {"since": since, "until": until},
|
||||
"target": {"name": "synth", "version": "1.0.0", "skills": ["synth:skill"]},
|
||||
"coverage": {"ratio": 0.1, "attributed_turns": 500},
|
||||
"totals": {"weighted_tokens": 5000000},
|
||||
@@ -184,12 +195,20 @@ check "not counted as checked" "$(gatefield "$tmp/e.out.json" checked)" "0"
|
||||
check "no finding" "$(gatefield "$tmp/e.out.json" downgraded)" "0"
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
echo "== (f) S4 — the real anxious candidate is caught with no human in the loop =="
|
||||
echo "== (f) S4 — the real anxious 7-day window is refused, as insufficient-window =="
|
||||
# The exact evidence from kotkan/claude-plugin-inference-arbitrage#11: the
|
||||
# 30-day window graded `anxious/agent-wip/release-policy-derivation` measured at
|
||||
# ~26-day window graded `anxious/agent-wip/release-policy-derivation` measured at
|
||||
# 11.94% of audited spend, the 7-day window graded the same candidate thin at
|
||||
# 0.00%. Both windows' snapshots are in the store; --exclude-run makes the 7-day
|
||||
# run the one being classified, exactly as it is at filing time.
|
||||
#
|
||||
# S4 still holds — the candidate is mechanically refused with no human in the
|
||||
# loop — but per kotkan/claude-plugin-inference-arbitrage#15 the honest reason is
|
||||
# not "unstable". Re-measuring the same candidate over 14/27/60 days gave 0 -> 2
|
||||
# -> 12 -> 15 invocations: a monotonic accumulation of a ~once-every-two-days
|
||||
# step, so the 7-day window under-sampled it rather than contradicting it. A
|
||||
# 7.0-day window cannot overturn a 26.1-day `measured` reading, so this is a
|
||||
# deferral, not a downgrade, and nothing is self-reported against this plugin.
|
||||
store=$tmp/store-anxious
|
||||
mkdir -p "$store/Data/anxious"
|
||||
cp calibration/anxious-windows.snapshots.jsonl "$store/Data/anxious/snapshots.jsonl"
|
||||
@@ -198,25 +217,67 @@ $BIN --classified calibration/anxious-7d.classified.json \
|
||||
--scan calibration/anxious-7d.scan.json \
|
||||
--exclude-run 2026-07-29T01-00-00Z >"$tmp/f.out.json"
|
||||
check "verdict" "$(field "$tmp/f.out.json" anxious/agent-wip/release-policy-derivation verdict)" \
|
||||
"boundary-question"
|
||||
check "downgraded" "$(gatefield "$tmp/f.out.json" downgraded)" "1"
|
||||
"insufficient-window"
|
||||
check "downgraded" "$(gatefield "$tmp/f.out.json" downgraded)" "0"
|
||||
check "insufficient_window" "$(gatefield "$tmp/f.out.json" insufficient_window)" "1"
|
||||
check "findings" "$(python3 -c "import json;print(len(json.load(open('$tmp/f.out.json'))['stability_findings']))")" "0"
|
||||
python3 -c "
|
||||
import json
|
||||
d=json.load(open('$tmp/f.out.json'))
|
||||
f=[x for x in d['stability_findings']
|
||||
f=[x for x in d['stability_deferrals']
|
||||
if x['candidate_id']=='anxious/agent-wip/release-policy-derivation'][0]
|
||||
assert f['prior']['measurement_strength']=='measured', f['prior']
|
||||
assert abs(f['prior']['share_of_audited_spend']-0.1194) < 1e-9, f['prior']
|
||||
assert f['current']['measurement_strength']=='thin', f['current']
|
||||
assert f['current']['share_of_audited_spend']==0.0, f['current']
|
||||
assert f['prior']['run_id']=='2026-07-29T00-00-00Z', f['prior']
|
||||
print(' ok finding carries the real numbers: measured 11.94% vs thin 0.00%')
|
||||
p=f['window_parity']
|
||||
assert p['comparable'] is False, p
|
||||
assert abs(p['current_days']-7.0) < 0.05, p
|
||||
assert 25.9 < p['prior_days'] < 26.2, p
|
||||
assert {x['field'] for x in f['suppressed_flips']} == {
|
||||
'measurement_strength','share_of_audited_spend'}, f['suppressed_flips']
|
||||
assert 'self_report' not in f, f
|
||||
print(' ok deferral carries the real widths (7.0d vs 26.1d) and the suppressed flips')
|
||||
" || fail=1
|
||||
python3 -c "
|
||||
import json
|
||||
d=json.load(open('$tmp/f.out.json'))
|
||||
assert d['summary']['to_file']==0, d['summary']
|
||||
print(' ok nothing is left to file from that window')
|
||||
assert d['summary']['boundary_questions']==2, d['summary']
|
||||
r=[c for c in d['candidates']
|
||||
if c['candidate_id']=='anxious/agent-wip/release-policy-derivation'][0]['reasons'][-1]
|
||||
assert 'DEFERRED (FR-4.5)' in r and 'carried' in r, r
|
||||
print(' ok not filed, not counted as a boundary question, reason cites the deferral')
|
||||
" || fail=1
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
echo "== (f2) real 60-day window against the real 27-day prior -> reports normally =="
|
||||
# The other half of kotkan/claude-plugin-inference-arbitrage#15's evidence: the
|
||||
# 60-day window (15 invocations, offload_value 4,953,520 of 35,722,973 weighted
|
||||
# tokens = 13.87%) against the ~27-day `measured` 11.94% prior. Two adequately
|
||||
# wide windows, same label, same side of the 2% threshold -> stable, files
|
||||
# normally. The parity rule must not suppress this: the fix is about a narrow
|
||||
# CURRENT window, not about refusing to ever conclude anything.
|
||||
#
|
||||
# --exclude-run drops the 7-day snapshot so the pairing under test really is
|
||||
# 27d-vs-60d; the 60d-against-7d pairing is the reverse direction, and it still
|
||||
# downgrades — see (k).
|
||||
$BIN --classified calibration/anxious-60d.classified.json \
|
||||
--target anxious --store "$store" \
|
||||
--scan calibration/anxious-60d.scan.json \
|
||||
--exclude-run 2026-07-29T01-00-00Z >"$tmp/f2.out.json"
|
||||
check "verdict" "$(field "$tmp/f2.out.json" anxious/agent-wip/release-policy-derivation verdict)" \
|
||||
"file"
|
||||
check "downgraded" "$(gatefield "$tmp/f2.out.json" downgraded)" "0"
|
||||
check "insufficient_window" "$(gatefield "$tmp/f2.out.json" insufficient_window)" "0"
|
||||
python3 -c "
|
||||
import json
|
||||
d=json.load(open('$tmp/f2.out.json'))
|
||||
c=d['candidates'][0]
|
||||
assert c['stability'].startswith('stable against 2026-07-29T00-00-00Z'), c['stability']
|
||||
assert abs(c['share_of_audited_spend']-0.1387) < 1e-9, c['share_of_audited_spend']
|
||||
print(' ok the 60-day reading (13.87%) is stable against the 27-day one (11.94%)')
|
||||
" || fail=1
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
@@ -266,4 +327,116 @@ $BIN --classified "$tmp/i.json" --target synth --store "$store" \
|
||||
check "verdict" "$(field "$tmp/i.out.json" synth/skill/step verdict)" "boundary-question"
|
||||
check "downgraded" "$(gatefield "$tmp/i.out.json" downgraded)" "1"
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
echo "== (j) narrow current window, measured wide prior -> insufficient-window =="
|
||||
# The kotkan/claude-plugin-inference-arbitrage#15 shape, synthetically: a 30-day
|
||||
# window measured the candidate at 12%, a 7-day window sees nothing. 7d is below
|
||||
# the 27.0d parity floor, so the weak reading does not get to overturn the
|
||||
# measured one — neither filed nor downgraded, and no self-report.
|
||||
classified "$tmp/j.json" thin 0.0
|
||||
prior "$tmp/store-j" measured 0.12
|
||||
scanwin "$tmp/j.scan.json" 2026-07-23 2026-07-30
|
||||
$BIN --classified "$tmp/j.json" --target synth --store "$tmp/store-j" \
|
||||
--scan "$tmp/j.scan.json" >"$tmp/j.out.json"
|
||||
check "verdict" "$(field "$tmp/j.out.json" synth/skill/step verdict)" "insufficient-window"
|
||||
check "downgraded" "$(gatefield "$tmp/j.out.json" downgraded)" "0"
|
||||
check "insufficient_window" "$(gatefield "$tmp/j.out.json" insufficient_window)" "1"
|
||||
check "self-reports" "$(python3 -c "import json;print(len(json.load(open('$tmp/j.out.json'))['stability_findings']))")" "0"
|
||||
python3 -c "
|
||||
import json
|
||||
d=json.load(open('$tmp/j.out.json'))
|
||||
f=d['stability_deferrals'][0]
|
||||
p=f['window_parity']
|
||||
assert p['prior_days']==30.0 and p['current_days']==7.0 and p['required_days']==27.0, p
|
||||
assert p['comparable'] is False and p['parity_ratio']==0.9, p
|
||||
assert 'at least 27.0d' in f['carry_forward'], f['carry_forward']
|
||||
assert f['prior']['run_id']=='2026-07-01T00-00-00Z', f['prior']
|
||||
s=d['candidates'][0]['stability']
|
||||
assert s.startswith('insufficient-window against 2026-07-01T00-00-00Z'), s
|
||||
assert d['summary']['to_file']==0 and d['summary']['boundary_questions']==0, d['summary']
|
||||
assert d['summary']['stability_insufficient_window']==1, d['summary']
|
||||
print(' ok deferral names both widths, the floor, and the window it defers against')
|
||||
" || fail=1
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
echo "== (k) reverse direction — thin wide prior, measured narrow current -> downgraded =="
|
||||
# The narrowness carve-out is one-directional. Here the WIDE window is the one
|
||||
# that saw nothing and the narrow one measures 12%: the weak reading is the
|
||||
# earlier one, the contradiction is genuine, and the human would still be told
|
||||
# two different things. Downgrade, exactly as before the fix.
|
||||
classified "$tmp/k.json" measured 0.12
|
||||
prior "$tmp/store-k" thin 0.0
|
||||
scanwin "$tmp/k.scan.json" 2026-07-23 2026-07-30
|
||||
$BIN --classified "$tmp/k.json" --target synth --store "$tmp/store-k" \
|
||||
--scan "$tmp/k.scan.json" >"$tmp/k.out.json"
|
||||
check "verdict" "$(field "$tmp/k.out.json" synth/skill/step verdict)" "boundary-question"
|
||||
check "downgraded" "$(gatefield "$tmp/k.out.json" downgraded)" "1"
|
||||
check "insufficient_window" "$(gatefield "$tmp/k.out.json" insufficient_window)" "0"
|
||||
python3 -c "
|
||||
import json
|
||||
d=json.load(open('$tmp/k.out.json'))
|
||||
f=d['stability_findings'][0]
|
||||
assert f['self_report']['repo']=='kotkan/claude-plugin-inference-arbitrage', f['self_report']
|
||||
p=f['window_parity']
|
||||
assert p['comparable'] is False, p
|
||||
print(' ok still self-reported, and the parity block records the widths anyway')
|
||||
" || fail=1
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
echo "== (l) two comparably wide windows, genuine flip -> downgraded =="
|
||||
# 28 days against 30 is at the parity floor (27.0d), so the windows are
|
||||
# comparably sampled and a measured -> thin flip is real instability.
|
||||
classified "$tmp/l.json" thin 0.0
|
||||
prior "$tmp/store-l" measured 0.12
|
||||
scanwin "$tmp/l.scan.json" 2026-07-02 2026-07-30
|
||||
$BIN --classified "$tmp/l.json" --target synth --store "$tmp/store-l" \
|
||||
--scan "$tmp/l.scan.json" >"$tmp/l.out.json"
|
||||
check "verdict" "$(field "$tmp/l.out.json" synth/skill/step verdict)" "boundary-question"
|
||||
check "downgraded" "$(gatefield "$tmp/l.out.json" downgraded)" "1"
|
||||
check "insufficient_window" "$(gatefield "$tmp/l.out.json" insufficient_window)" "0"
|
||||
python3 -c "
|
||||
import json
|
||||
p=json.load(open('$tmp/l.out.json'))['stability_findings'][0]['window_parity']
|
||||
assert p['current_days']==28.0 and p['required_days']==27.0 and p['comparable'] is True, p
|
||||
print(' ok 28.0d clears the 27.0d floor, so the flip stands')
|
||||
" || fail=1
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
echo "== (m) measured both windows, threshold crossed on a narrow window -> downgraded =="
|
||||
# A window that still reached `measured` observed the candidate often enough; a
|
||||
# share that then crosses the 2% filing threshold is a claim about relative
|
||||
# spend, not a sampling failure, so narrowness does not excuse it.
|
||||
classified "$tmp/m.json" measured 0.01
|
||||
prior "$tmp/store-m" measured 0.12
|
||||
scanwin "$tmp/m.scan.json" 2026-07-23 2026-07-30
|
||||
$BIN --classified "$tmp/m.json" --target synth --store "$tmp/store-m" \
|
||||
--scan "$tmp/m.scan.json" >"$tmp/m.out.json"
|
||||
check "verdict" "$(field "$tmp/m.out.json" synth/skill/step verdict)" "boundary-question"
|
||||
check "downgraded" "$(gatefield "$tmp/m.out.json" downgraded)" "1"
|
||||
check "insufficient_window" "$(gatefield "$tmp/m.out.json" insufficient_window)" "0"
|
||||
python3 -c "
|
||||
import json
|
||||
f=json.load(open('$tmp/m.out.json'))['stability_findings'][0]
|
||||
assert {x['field'] for x in f['flips']} == {'share_of_audited_spend'}, f['flips']
|
||||
print(' ok only the threshold flip is reported, and it is not excused')
|
||||
" || fail=1
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
echo "== (n) unreadable window widths -> downgraded, and said so explicitly =="
|
||||
# No --scan means no current window bounds. Narrowness cannot be established, so
|
||||
# the flip is NOT quietly excused — the finding records that the check could not
|
||||
# run, rather than defaulting either way.
|
||||
classified "$tmp/n.json" thin 0.0
|
||||
prior "$tmp/store-n" measured 0.12
|
||||
$BIN --classified "$tmp/n.json" --target synth --store "$tmp/store-n" >"$tmp/n.out.json"
|
||||
check "verdict" "$(field "$tmp/n.out.json" synth/skill/step verdict)" "boundary-question"
|
||||
check "downgraded" "$(gatefield "$tmp/n.out.json" downgraded)" "1"
|
||||
python3 -c "
|
||||
import json
|
||||
p=json.load(open('$tmp/n.out.json'))['stability_findings'][0]['window_parity']
|
||||
assert p['comparable'] is None and p['current_days'] is None, p
|
||||
assert 'unreadable' in p['why'], p
|
||||
print(' ok comparability is null with an explicit reason, not a silent exemption')
|
||||
" || fail=1
|
||||
|
||||
exit "$fail"
|
||||
|
||||
Reference in New Issue
Block a user