10 Commits

Author SHA1 Message Date
Oleks 5f5462571c Use 4-backtick outer fence (markdownlint MD048 requires backtick, not tilde)
ci/woodpecker/push/test Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
The tilde fence from the previous commit avoided the nesting collision but
violated this repo's markdownlint style (backtick-only), which the pre-push
hook enforces and rejected. A 4-backtick outer fence with the real 3-backtick
example nested inside satisfies both: CommonMark only closes a fence on a
matching-or-longer run of the same character, and MD048 sees backticks
throughout.
2026-07-30 19:25:27 +03:00
Oleks 64d6c90284 Fix nested code-fence collision the pre-push formatter introduced
The previous commit nested a triple-backtick example fence inside a
triple-backtick pseudocode block; the repo's markdown auto-formatter treated
the inner ``` as closing the outer block and reindented/moved the "for
action == comment" case out of the Agent(...) call. Use ~~~ for the outer
fence so the real triple-backtick example nests without collision.
2026-07-30 19:24:54 +03:00
Oleks b77fbdddbb Fix marker-wrapper loss and add backward-compat match (kotkan/claude-plugin-inference-arbitrage#23)
Root cause: bin/filing-plan and references/issue-template.md always render
the ia-candidate marker HTML-comment-wrapped. A direct issue_write probe
confirmed the Gitea MCP write path preserves a literal <!-- --> unchanged, so
the loss on oleks/claude-plugin-cluster#56 and #57 happened on the
Agent(anxious:issuer-agent) relay hop: an HTML comment is invisible when its
surrounding text is read as markdown, so an agent relaying "what the body
should say" instead of pasting it verbatim can silently drop it.

Two fixes:
- bin/filing-plan: find_existing()'s marker_re() now also matches the bare,
  unwrapped "ia-candidate: <id>" line (end-of-line anchored, not \b, since
  candidate ids contain "-" and a word-boundary check would false-match a
  longer id sharing a prefix) so already-filed issues like cluster#56/#57
  are still recognized and not duplicate-filed.
- skills/offload-audit/SKILL.md: the issuer delegation prompt now fences the
  rendered body in its own code block and explicitly warns that the leading
  marker is an invisible-by-design HTML comment, not stray syntax to clean
  up, instead of inlining it as bare markdown the relaying agent could read
  right through.

references/issue-template.md documents both, and tests/filing.test.sh adds
coverage for the bare-marker match plus a same-prefix-different-id
non-match case.

Bumps plugin.json to 0.7.1.
2026-07-30 19:24:13 +03:00
Oleks 88fc58b2fd FR-4.5.1: a narrow window may not overturn a wide measured reading
ci/woodpecker/push/test Pipeline was successful
The stability gate treated every evidence flip as symmetric, and that
produced a false downgrade on the candidate it was written for
(kotkan/claude-plugin-inference-arbitrage#15). Re-measuring
anxious/agent-wip/release-policy-derivation over four real windows with
the same script gave 0 -> 2 -> 12 -> 15 invocations at 7/14/27/60 days:
monotonic accumulation of a step that runs about once every two days, not
a label flipping about. The 7-day window had not caught an unstable
candidate, it had failed to observe a real one.

So a thin/unmeasured reading now only overturns a `measured` one when its
window is at least 90% as wide (observed since->until span). Below that
the comparison takes a third verdict, `insufficient-window`: not filed,
not downgraded to a boundary question, no FR-6.5 self-report, and carried
forward with the wider snapshot left as the standing comparison point.
The parity is relative rather than an absolute day count because the
adequate width is a property of the candidate's invocation rate, which
the auditor does not know in advance.

Unchanged, and tested: the reverse direction (thin prior -> measured now)
still downgrades, a flip between comparably wide windows still downgrades,
and a threshold crossing with `measured` on both sides is never excused by
narrowness. An unreadable window is not an exemption either — the flip
stands and window_parity records that the check could not run.

- bin/stability-classify: window_days/window_parity/deferrable, the
  insufficient-window verdict, stability_deferrals, window_parity on
  findings too, insufficient_window + window_parity_ratio in the gate block
- bin/audit-snapshot: record deferrals in the snapshot notes so the row is
  visible on the wiki page instead of silently dropped
- design/spec.md: FR-4.5.1 as a testable requirement; S4 restated
- design/rubric.md: the third boundary in §5b
- skills/offload-audit/SKILL.md: do not file and do not self-report a deferral
- tests: real 60-day anxious fixture; cases (f)/(f2) on real data, (j)-(n)
  synthetic — 7 test files pass

Verified against the live store (~/.cache/inference-arbitrage/wiki): the
real 7.0d-vs-26.49d comparison now reports insufficient-window with zero
findings, while the real 33.4d-vs-26.49d one reports stable at 13.87% and
files normally.
2026-07-30 15:57:02 +03:00
oleks 4230782aaa Merge pull request 'ci: fetch cc-tokens from token-budget for the suites step' (#22) from fix-20-ci-cc-tokens into main
ci/woodpecker/push/test Pipeline is pending
2026-07-30 15:48:44 +03:00
oleks eba3449e60 ci: fetch cc-tokens from token-budget for the suites step
ci/woodpecker/push/test Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
kotkan/claude-plugin-inference-arbitrage#20 -- the bare alpine:3 CI
container has neither the Claude Code plugin cache nor `claude plugin
install`, so bin/offload-scan's cc-tokens PATH lookup always failed.
Per design decision 0.1 this plugin never vendors token-budget's
pricing/token-accounting logic, so fetch the single cc-tokens file
straight from kotkan/claude-plugin-token-budget's raw main branch and
put it on PATH before running tests/run-all.sh, rather than mocking
its behavior.
2026-07-30 15:48:27 +03:00
oleks 1a14456220 Merge pull request 'Make the wiki history check mandatory before stability-classify' (#19) from fix-17-stability-gate-wiki-check into main
ci/woodpecker/push/test Pipeline is pending
2026-07-30 05:41:34 +03:00
Oleks 713469d0d9 Fix markdownlint line-length in the step 4b wiki-fetch snippet
ci/woodpecker/push/test Pipeline failed
ci/woodpecker/pr/test Pipeline failed
Follow-up to the kotkan/claude-plugin-inference-arbitrage#17 commit.
2026-07-30 05:40:41 +03:00
Oleks 69ff443bd2 Make the wiki history check mandatory before stability-classify (fix kotkan/claude-plugin-inference-arbitrage#17)
bin/stability-classify only ever reads the local filesystem snapshot store
(by design, no bin/ script holds wiki credentials), but SKILL.md step 4b
documented seeding it from the wiki as a conditional "if the store is cold"
aside. That made a cold-but-not-actually-empty local cache indistinguishable
from a genuine first audit: both silently emit stability: unchecked and file
unstable candidates, as observed on the 2026-07-30 anxious run.

- bin/stability-classify: add a required --wiki-checked {empty,imported}
  flag; hard-error when the store is empty and the flag is omitted, instead
  of silently degrading to unchecked.
- tests/stability.test.sh: (g) no flag on an empty store errors, (h)
  --wiki-checked=empty behaves as before, (i) --wiki-checked=imported after a
  real `audit-snapshot import` runs the gate normally against the imported
  history.
- skills/offload-audit/SKILL.md: step 4b now fetches the wiki's
  Data/<target>/snapshots.jsonl and runs `audit-snapshot import`
  unconditionally (it dedupes by run_id, so this is safe every run) before
  invoking the gate.
- Bump plugin.json to 0.6.3.
2026-07-30 05:40:08 +03:00
oleks 58b14cbde1 Merge pull request 'offload-scan: caveat reconstructed invocation counts as a lower bound' (#18) from fix/issue-12-invocation-caveat into main
ci/woodpecker/push/test Pipeline is pending
2026-07-30 05:40:01 +03:00
14 changed files with 1143 additions and 47 deletions
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "inference-arbitrage",
"version": "0.6.3",
"version": "0.7.1",
"description": "Audits a Claude Code plugin's definitions and usage transcripts to find steps that should be a deterministic script instead of raw LLM inference, and files the well-evidenced ones as issues on the target repo.",
"author": {
"name": "oleks"
+15 -2
View File
@@ -14,13 +14,26 @@ steps:
- name: suites
image: alpine:3
commands:
- apk add --no-cache bash python3 >/dev/null
- apk add --no-cache bash python3 wget >/dev/null
# bin/offload-scan (design decision 0.1) deliberately never vendors
# token-budget's pricing/token-accounting logic -- it imports the real
# cc-tokens module and fails loudly if it's missing. This bare alpine
# container has neither the Claude Code plugin cache nor `claude
# plugin install`, so fetch the one file it needs straight from
# token-budget's own repo and put it on PATH. See
# kotkan/claude-plugin-inference-arbitrage#20.
- mkdir -p /tmp/ci-tools
- >-
wget -q -O /tmp/ci-tools/cc-tokens
https://git.oleks.space/kotkan/claude-plugin-token-budget/raw/branch/main/bin/cc-tokens
- chmod +x /tmp/ci-tools/cc-tokens
- export PATH="/tmp/ci-tools:$PATH"
- timeout 600 bash tests/run-all.sh
- name: shellcheck
image: koalaman/shellcheck-alpine:stable
commands:
# bin/ mixes python and bash (plugin-inventory is python) filter by
# bin/ mixes python and bash (plugin-inventory is python) -- filter by
# actual shebang, not just the executable bit or a *.sh glob, or
# shellcheck chokes on the python entries (SC1071).
- >-
+6 -1
View File
@@ -180,9 +180,14 @@ def build_snapshot(target, inventory, scan, classified, issue_map, explicit, run
# the filing pass — `filing: unavailable` and why (FR-6.4) — which this
# script cannot see and which have to reach the wiki page, not just the
# user's terminal.
# `insufficient-window` is in this list, not in `candidates` or
# `boundary_questions`: an FR-4.5 deferral is neither a filed candidate nor
# a question put to a human, and the wider snapshot it was deferred against
# remains the standing comparison point. Recording it as a note keeps the
# deferral visible on the wiki page instead of silently dropping the row.
notes = [
f"{c['candidate_id']}: {c['verdict']} — {c['reasons'][-1] if c.get('reasons') else ''}"
for c in rows if c.get("verdict") in ("no-offload", "drift-note")
for c in rows if c.get("verdict") in ("no-offload", "drift-note", "insufficient-window")
] + list(extra_notes)
return {
+23 -4
View File
@@ -85,10 +85,29 @@ def marker(candidate_id):
def marker_re(candidate_id):
"""Tolerates whitespace drift inside the comment, nothing else. The id
itself must match exactly — a fuzzy id match would let two genuinely
different candidates collapse onto one issue."""
return re.compile(r"<!--\s*ia-candidate:\s*" + re.escape(candidate_id) + r"\s*-->")
"""Tolerates whitespace drift inside the comment, and tolerates the
HTML-comment wrapper (`<!-- -->`) being absent entirely, nothing else. The
id itself must match exactly — a fuzzy id match would let two genuinely
different candidates collapse onto one issue.
The bare form (no `<!-- -->`) is a backward-compatibility case, not the
contract: kotkan/claude-plugin-inference-arbitrage#23 found that issues
filed on 2026-07-29 (oleks/claude-plugin-cluster#56, #57) lost the wrapper
somewhere between `filing-plan`'s render and the Gitea write — the
template and this script always render the wrapped form (see `marker()`
below and references/issue-template.md). Matching the bare line here is
purely so a *later* audit still recognizes those already-filed issues and
doesn't duplicate-file them; it does not license filing new issues without
the wrapper.
"""
wrapped = r"<!--\s*ia-candidate:\s*" + re.escape(candidate_id) + r"\s*-->"
# End-of-line anchored (not \b): candidate_id contains "/" and "-", so a
# word-boundary check would treat "…label-derivation" as a match inside
# "…label-derivation-longer" (the "n"/"-" transition IS a \b). Anchoring
# to end-of-line (re.MULTILINE) instead requires an exact id, matching
# what marker() actually renders as the line's sole content.
bare = r"^ia-candidate:\s*" + re.escape(candidate_id) + r"[ \t]*$"
return re.compile(wrapped + "|" + bare, re.MULTILINE)
def find_existing(candidate_id, issues):
+241
View File
@@ -40,6 +40,66 @@ window and 5% the next is stable — same label, same side of the threshold —
that variation is what `audit-snapshot diff` exists to report as a trend. This
gate fires only when the *description a human receives* changes.
A NARROW WINDOW IS NOT EVIDENCE OF INSTABILITY EITHER
-----------------------------------------------------
The gate as first written treated every flip as symmetric, and that produced a
false downgrade on the very candidate that motivated it
(kotkan/claude-plugin-inference-arbitrage#15). Re-measuring
`anxious/agent-wip/release-policy-derivation` across four real windows with the
same script showed a monotonic accumulation, not a flip:
7 days -> 0 invocations, 0.00% of spend -> thin
14 days -> 2 invocations, 0.72% -> thin
27 days -> 12 invocations, 11.73% -> measured
60 days -> 15 invocations, 13.87% -> measured
The candidate runs about once every two days. A 7-day window did not observe an
unstable candidate; it failed to observe a real one. The wider the window, the
higher the reading — the 60-day pass had every opportunity to dilute the share
back down and instead raised it. Calling that "the confidence label moves with
the calendar" gets the direction of the artifact backwards: the *narrow* window
was the artifact, and the gate punished the candidate for it.
So a weak reading only gets to overturn a `measured` one when it was taken over
a comparably wide window. Concretely, all three must hold:
* the prior reading is `measured`,
* the current reading is `thin` or `unmeasured`,
* the current window is narrower than WINDOW_PARITY_RATIO of the prior
window's observed width (`until - since`),
and then the comparison is not read as instability at all. The candidate takes
a third verdict, `insufficient-window`: it is neither filed nor downgraded, no
self-report is emitted, and it is carried forward — the wider prior snapshot
stays the standing comparison point, so the next audit run over an adequate
window compares against it rather than against this window's under-sample.
The parity is relative, not an absolute day count, on purpose. "At least 21
days" would be a guess about candidate frequency dressed up as a threshold —
this candidate needs ~27 days, a once-a-week step would need more, and a
once-an-hour step is fully measured in one. What the gate can know without
guessing is that a window materially narrower than the one that produced the
`measured` reading cannot fairly contradict it. WINDOW_PARITY_RATIO is 0.9
rather than 1.0 because window bounds are observed session spans, not requested
ranges, so two nominally identical windows differ by hours.
Three things stay downgrades, unchanged:
1. the reverse direction (`thin` prior, `measured` current) — a wider or
equally wide current window that now sees the candidate genuinely
contradicts the earlier reading, and the earlier reading is the weak one;
2. any flip between comparably wide windows;
3. a threshold-side flip in which BOTH windows read `measured`. If the
current window still reached `measured`, it observed the candidate often
enough; a share that then crosses the filing threshold is a claim about
relative spend, not a sampling failure, and narrowness does not excuse it.
Width is read off the snapshot's own `window.since`/`window.until`. When either
window does not carry both bounds the widths are not comparable, and the gate
does NOT quietly excuse the flip: it downgrades as before and says so in the
finding's `window_parity` block, so an unreadable window is visible rather than
a silent exemption.
NO PRIOR WINDOW MEANS UNCHECKED, NOT UNSTABLE
---------------------------------------------
A first-ever audit of a target, or a candidate seen for the first time, has
@@ -48,6 +108,26 @@ nothing to compare against. It files normally, marked
of history would contradict FR-2.4 — a target with no history stays auditable on
its own terms — and would make the first audit of every plugin file nothing.
AN EMPTY LOCAL STORE IS NOT THE SAME CLAIM AS AN EMPTY WIKI
-------------------------------------------------------------
This script only ever reads the local filesystem mirror of the store — by
design, no `bin/` script holds wiki credentials (see `audit-snapshot`'s
docstring). That means an empty `--store` is ambiguous: it might be a genuine
first audit (FR-2.4), or it might just be a cold local cache sitting in front
of real wiki history nobody imported yet (kotkan/claude-plugin-inference-
arbitrage#17). The two look identical to `find_prior` but are not the same
fact about the target, and only the caller — who can read the wiki — knows
which one is true.
So when the store is empty, `--wiki-checked` is mandatory: `empty` asserts the
caller checked the wiki page `Data/<target>/snapshots.jsonl` and it genuinely
has no history (or the target truly is new), so `unchecked` is correct;
`imported` asserts the caller ran `audit-snapshot import` first — in which case
the store would no longer be empty and this path would not be taken at all.
Omitting the flag on an empty store is a hard error: it turns the SKILL.md
step 4b instruction to check the wiki first from a doc-only aside into
something the gate itself refuses to skip past.
The prior window is the most recent snapshot that contains this candidate by
FR-5.2 identity (issue > signature hash > slug), not merely the most recent
snapshot: a candidate that sat out one window is still comparable against the
@@ -60,6 +140,7 @@ of `classified.json`.
"""
import argparse
import datetime
import json
import sys
@@ -72,6 +153,21 @@ GATE = "FR-4.5"
UNCHECKED_NO_HISTORY = "unchecked — no prior window to compare"
UNCHECKED_NO_MATCH = "unchecked — no prior window recorded this candidate"
# The verdict a comparison takes when it is too lopsided to read either way.
# Distinct from `boundary-question` on purpose: a boundary question is something
# a human is asked to judge, and this is the auditor declining to make a claim.
INSUFFICIENT_WINDOW = "insufficient-window"
# Readings that mean "this window did not observe the candidate enough", as
# opposed to `measured`, which means it did.
WEAK_STRENGTHS = ("thin", "unmeasured")
# How wide the current window must be, as a fraction of the prior window that
# produced the `measured` reading, before a weak reading may overturn it. Not
# 1.0: window bounds are observed session spans, so two nominally equal windows
# differ by hours and an exact-parity rule would fire on clock noise.
WINDOW_PARITY_RATIO = 0.9
def keyed(cand, scan, explicit, issue_map):
"""The identity fields FR-5.2 matches on, computed for a candidate that is
@@ -129,10 +225,78 @@ def flips(prior, curr_strength, curr_share):
return out
def window_days(window):
"""Observed width of a window in days, or None when it cannot be read.
None is a real answer, not a default: `window_parity` reports it as
"not comparable" and lets the flip stand, rather than treating an
unreadable window as narrow (which would excuse every flip) or as wide
(which would hide that the check could not run)."""
window = window or {}
bounds = []
for key in ("since", "until"):
raw = window.get(key)
if not isinstance(raw, str):
return None
try:
bounds.append(datetime.datetime.fromisoformat(raw))
except ValueError:
return None
since, until = bounds
# One bound tz-aware and the other naive cannot be subtracted, and guessing
# a zone for the naive one would invent a width.
if (since.tzinfo is None) != (until.tzinfo is None):
return None
return (until - since).total_seconds() / 86400.0
def window_parity(prior_window, curr_window):
"""Whether the current window is wide enough to contradict the prior one.
Always returned, whether or not it changes the outcome, so the finding
records the widths it was decided on."""
prior_days = window_days(prior_window)
curr_days = window_days(curr_window)
out = {
"parity_ratio": WINDOW_PARITY_RATIO,
"prior_days": round(prior_days, 2) if prior_days is not None else None,
"current_days": round(curr_days, 2) if curr_days is not None else None,
"required_days": (round(prior_days * WINDOW_PARITY_RATIO, 2)
if prior_days is not None else None),
}
if prior_days is None or curr_days is None:
out["comparable"] = None
out["why"] = ("window width is unreadable — one of the two windows does not carry "
"both `since` and `until` — so narrowness cannot be established and "
"the flip is not excused; this is recorded rather than defaulted")
return out
out["comparable"] = curr_days >= out["required_days"]
out["why"] = (
f"this window spans {out['current_days']}d against the prior window's "
f"{out['prior_days']}d; a weak reading needs at least "
f"{out['required_days']}d ({WINDOW_PARITY_RATIO:.0%} of the prior width) to "
"contradict a `measured` one"
if not out["comparable"] else
f"this window spans {out['current_days']}d against the prior window's "
f"{out['prior_days']}d, at or above the {out['required_days']}d parity floor, so "
"the two readings are comparably sampled and the flip is real")
return out
def deferrable(prior_strength, curr_strength):
"""The only direction narrowness can excuse: a wide `measured` reading that a
narrow window failed to observe. The reverse (`thin` prior, `measured` now)
is a genuine contradiction in which the *earlier* reading is the weak one,
and a `measured` -> `measured` threshold crossing means both windows saw the
candidate, so neither is excused here."""
return prior_strength == "measured" and curr_strength in WEAK_STRENGTHS
def gate(doc, history, scan, explicit, issue_map):
target = doc.get("target")
window = scan.get("window") or {}
findings = []
deferrals = []
checked = 0
for cand in doc.get("candidates", []):
@@ -160,6 +324,54 @@ def gate(doc, history, scan, explicit, issue_map):
f"{curr_strength}, {curr_share:.2%} of audited spend either side")
continue
parity = window_parity(snap.get("window"), window)
if deferrable(prior.get("measurement_strength"), curr_strength) \
and parity["comparable"] is False:
# Not filed, not downgraded, not self-reported: this comparison
# cannot be made fairly, and saying so is the honest third answer.
cand["verdict"] = INSUFFICIENT_WINDOW
cand["stability"] = (
f"insufficient-window against {snap.get('run_id')} (matched by {by}): "
f"{parity['current_days']}d window read {curr_strength} against a "
f"{parity['prior_days']}d window's measured "
f"{((prior.get('measurement') or {}).get('share_of_plugin') or 0):.2%} — "
"too narrow to compare fairly")
cand.setdefault("reasons", []).append(
f"DEFERRED ({GATE}): the current window ({parity['current_days']}d) is "
f"narrower than the {parity['required_days']}d parity floor set by the "
f"{parity['prior_days']}d window that produced the prior `measured` "
"reading, so this window's weak reading cannot fairly contradict it. "
"Neither filed nor downgraded to a boundary question this run; carried "
"forward for a comparison over an adequately wide window.")
deferrals.append({
"gate": GATE,
"candidate_id": cand.get("candidate_id"),
"target": target,
"skill": cand.get("skill"),
"matched_by": by,
"prior": {
"run_id": snap.get("run_id"),
"window": snap.get("window"),
"measurement_strength": prior.get("measurement_strength"),
"share_of_audited_spend": (prior.get("measurement") or {}).get("share_of_plugin"),
},
"current": {
"window": window,
"measurement_strength": curr_strength,
"share_of_audited_spend": curr_share,
},
# What WOULD have been reported as instability, kept so the
# deferral is auditable rather than an unexplained absence.
"suppressed_flips": moved,
"window_parity": parity,
"carry_forward": (
f"re-audit over a window of at least {parity['required_days']}d and "
f"compare against {snap.get('run_id')}, which remains the standing "
"comparison point — this window contributes no reading that could "
"overturn it"),
})
continue
cand["verdict"] = "boundary-question"
cand["stability"] = f"unstable against {snap.get('run_id')} (matched by {by})"
cand.setdefault("reasons", []).append(
@@ -186,6 +398,10 @@ def gate(doc, history, scan, explicit, issue_map):
"share_of_audited_spend": curr_share,
},
"flips": moved,
# Recorded on downgrades too, so a reader can see the widths this
# was decided on and that the narrow-window carve-out was
# considered and did not apply.
"window_parity": parity,
"headline": (
f"{cand.get('candidate_id')} is {prior.get('measurement_strength')} at "
f"{((prior.get('measurement') or {}).get('share_of_plugin') or 0):.2%} of spend in "
@@ -206,14 +422,21 @@ def gate(doc, history, scan, explicit, issue_map):
summary["boundary_questions"] = sum(1 for c in rows if c.get("verdict") == "boundary-question")
summary["stability_checked"] = checked
summary["stability_downgraded"] = len(findings)
summary["stability_insufficient_window"] = len(deferrals)
doc["stability_gate"] = {
"gate": GATE,
"snapshots_available": len(history),
"checked": checked,
"downgraded": len(findings),
"insufficient_window": len(deferrals),
"window_parity_ratio": WINDOW_PARITY_RATIO,
}
doc["stability_findings"] = findings
# Kept out of `stability_findings` deliberately: those are FR-6.5
# self-reports the caller files against this plugin's repo, and a deferral
# is not a defect to report — it is a comparison that was not made.
doc["stability_deferrals"] = deferrals
return doc
@@ -232,6 +455,13 @@ def main():
ap.add_argument("--signatures", help="JSON {candidate_id: [tool signature, ...]}")
ap.add_argument("--issue-map", help='JSON {candidate_id: "owner/repo#N"}')
ap.add_argument("--exclude-run", help="run_id to treat as not-yet-recorded (tests, re-runs)")
ap.add_argument("--wiki-checked", choices=["empty", "imported"],
help="required when the local store is empty for this target: 'empty' "
"asserts the wiki was checked and genuinely has no history; "
"'imported' asserts `audit-snapshot import` already ran. Without "
"this, an empty store is ambiguous between a genuine first audit "
"(FR-2.4) and a merely cold local cache in front of real wiki "
"history (kotkan/claude-plugin-inference-arbitrage#17).")
ap.add_argument("--json", action="store_true", help="compact single-line JSON")
args = ap.parse_args()
@@ -239,6 +469,17 @@ def main():
history = [s for s in load_snapshots(args.store, args.target)
if s.get("run_id") != args.exclude_run]
if not history and args.wiki_checked is None:
sys.exit(
"error: the store has no history for this target and --wiki-checked was not "
"given. This is ambiguous: it may be a genuine first audit (FR-2.4), or it may "
"just be a cold local cache in front of real wiki history "
"(kotkan/claude-plugin-inference-arbitrage#17). Check the wiki page "
f"Data/{args.target}/snapshots.jsonl first, then either pass "
"--wiki-checked=empty (the wiki genuinely has no history for this target), or "
"run `audit-snapshot import` to seed the store and re-run with "
"--wiki-checked=imported.")
result = gate(doc, history,
load_json(args.scan) if args.scan else {},
load_json(args.signatures) if args.signatures else {},
+29 -1
View File
@@ -340,7 +340,7 @@ Hence the same shape of rule as §5:
> windows land on opposite sides of the filing threshold, the candidate is not
> filed.
Two boundaries make this a usable rule rather than a blanket refusal.
Three boundaries make this a usable rule rather than a blanket refusal.
**Movement is not instability.** A candidate at 12% one window and 8% the next
is stable: the human is told the same thing, and the difference is a trend, which
@@ -356,6 +356,34 @@ finding out of missing history would mean no plugin could ever be audited once,
which contradicts the principle that a plugin with no measured history stays
auditable on its definition alone.
**A narrow window is not instability either — and this one was learned the hard
way.** The rule above, applied symmetrically, downgraded the very candidate it
was written for. `anxious/agent-wip/release-policy-derivation` reads `thin, 0%`
over 7 days and `measured, 11.9%` over 27 — which looks like the §5b failure
until you measure two more windows: 0 → 2 → 12 → 15 invocations at 7, 14, 27 and
60 days. It runs about once every two days. The short window did not catch an
unstable candidate; it failed to observe a real one, and the wider window kept
raising the reading rather than diluting it. Punishing the candidate for that
gets the artifact backwards: the *window* was the artifact.
So a weak reading only overturns a `measured` one when it was sampled over a
comparably wide window — at least 90% of the width that produced the `measured`
reading. Below that, the answer is neither "file" nor "boundary question" but a
third, honest one: **`insufficient-window` — cannot tell either way, come back
with a wider window.** Nothing is filed, nothing is self-reported, and the wider
snapshot stays the standing comparison point so the next adequate run compares
against it.
The parity is deliberately relative, not "at least N days": the adequate width
is a fact about the candidate's invocation rate, which the auditor learns from
the measurement rather than knowing in advance, so an absolute floor would be a
guess about frequency dressed up as a threshold. And the carve-out is
one-directional. A `thin` *prior* against a `measured` current is still a
downgrade — there the earlier reading is the weak one — and so is a threshold
crossing where both windows read `measured`, because a window that reached
`measured` did observe the candidate; what moved was its share of spend, which
is signal, not sampling.
The instability, when it fires, is a finding **about the auditor**. The target
did not change; the tool described it two different ways. So it is reported
against this plugin's own repo, not the target's — the same self-application
+52 -2
View File
@@ -237,6 +237,42 @@ fact about the target. A candidate with no prior window to compare against is
marked `unchecked` and files normally; an absence of history is not evidence of
instability, and manufacturing one from it would contradict FR-2.4.
**FR-4.5.1 — Minimum window width for a downgrade.** A weak reading may only
overturn a `measured` one when it was taken over a comparably wide window. Let
`width(w) = w.until - w.since` in days, read off the window each reading was
taken over (the observed session span the scan records, not the requested
range). A comparison is **not read as instability**, and takes a third verdict
`insufficient-window`, when all of the following hold:
1. the prior reading's `measurement_strength` is `measured`;
2. the current reading's `measurement_strength` is `thin` or `unmeasured`;
3. `width(current) < 0.9 × width(prior)`.
An `insufficient-window` candidate is **neither filed nor downgraded to a
boundary question**, emits **no FR-6.5 self-report**, and is carried forward: the
wider prior snapshot remains the standing comparison point, so the next audit
over an adequately wide window compares against it and not against this
window's under-sample. The run records the widths it decided on, the parity
floor, and the flips it suppressed.
Three cases remain downgrades, and must not be excused by narrowness: the
reverse direction (`thin`/`unmeasured` prior, `measured` current — the earlier
reading is then the weak one); any flip between windows that satisfy the 0.9
parity; and a threshold crossing in which **both** readings are `measured`,
since a window that reached `measured` observed the candidate often enough for
the share to be a claim about relative spend rather than a sampling failure.
The rule is stated as a ratio against the prior window rather than an absolute
number of days because the adequate width is a property of the candidate's
invocation rate, which the auditor does not know in advance:
`anxious/agent-wip/release-policy-derivation` runs roughly once every two days
and needs ~27 days; a once-per-hour step is fully measured in one. The ratio is
0.9 rather than 1.0 because window bounds are observed session spans, so two
nominally identical windows differ by hours. When either window does not carry
both bounds, width is not computable, the flip is **not** excused, and the run
records that the check could not be made — an unreadable window must be visible,
never a silent exemption.
### FR-5 — Snapshot accumulation and diffing
**FR-5.1** Every run appends one machine-readable snapshot record to an
@@ -323,7 +359,10 @@ ranked candidate table with boundary classification and value; the diff against
the previous snapshot; the boundary questions posed to the user — both the
low-confidence candidates and any downgraded by the FR-4.5 stability gate, the
latter marked as a defect in the auditor rather than a fact about the
target; and the issues filed or updated, in full `owner/repo#num` form,
target; any candidate the gate could not judge either way (`insufficient-window`,
FR-4.5.1), stated as a window too narrow to compare rather than as a finding
about the target or the auditor; and the issues filed or updated, in full
`owner/repo#num` form,
separating those on the target's tracker (FR-6.1) from those on this plugin's
own (FR-6.5).
@@ -488,7 +527,7 @@ is verified by hand (TASKS 7.9.5).
| S1 | Audits any plugin by path or name, with no target-specific code | Run against `token-budget`, `worktree-discipline`, `anxious`, `memory` |
| S2 | Correctly returns "nothing to offload" for a well-cut plugin | Run against `token-budget` → no `high`-confidence candidates |
| S3 | Rediscovers a known-correct historical cut | Run against `worktree-discipline` with `bin/worktree-audit` removed from the inventory → must flag the classification step as `high` confidence |
| S4 | Mechanically refuses a candidate whose evidence is a window artifact | The two-window `anxious` fixture → the FR-4.5 gate downgrades `agent-wip/release-policy-derivation`, with no human in the loop |
| S4 | Mechanically refuses a candidate whose evidence is a window artifact | The two-window `anxious` fixture → the FR-4.5 gate refuses `agent-wip/release-policy-derivation` (as `insufficient-window`, per FR-4.5.1, since the 7-day current window cannot fairly contradict the 26-day `measured` one), with no human in the loop; a flip between comparably wide windows is still downgraded to a boundary question |
| S5 | Re-running does not duplicate issues | Run twice; second run comments, does not file |
| S6 | Trend across ≥2 snapshots is computable and volume-normalized | Two runs on different windows; diff reports per-invocation deltas |
| S7 | Passes its own audit | FR-8.2 |
@@ -505,6 +544,17 @@ that a human review caught a candidate whose evidence flipped between windows
so the criterion is now that gate, run mechanically (FR-4.5), against the real
two-window evidence that produced the original finding.
The *reason* S4's candidate is refused changed in v0.7.0, and the change is
itself instructive. Two further real windows (14 and 60 days, same script, same
candidate) showed 0 → 2 → 12 → 15 invocations as the window widened: monotonic
accumulation of a low-frequency step, not a label flipping about. The 7-day
window had not caught an unstable candidate, it had failed to observe a real one,
and calling that instability blamed the candidate for the auditor's window. So
FR-4.5.1 makes the refusal honest — `insufficient-window`, cannot tell either
way, compare again over a wider window — while keeping the property S4 exists to
test: nothing is filed off a window artifact, with no human in the loop.
(kotkan/claude-plugin-inference-arbitrage#15.)
---
## 5. Signal catalog
+20
View File
@@ -38,6 +38,26 @@ edited by hand on a filed issue. Removing it from an issue causes the next audit
to file a duplicate — which the spec names as the single most likely way for
this plugin to become hated.
**Known failure mode (kotkan/claude-plugin-inference-arbitrage#23).** Two
issues filed on 2026-07-29 (oleks/claude-plugin-cluster#56, #57) reached Gitea
with the `<!-- -->` wrapper stripped — the body opened with the bare text
`ia-candidate: <id>` instead. `bin/filing-plan` always *renders* the wrapped
form (verified: a direct `issue_write` probe with a literal `<!-- -->` in the
body round-trips through the Gitea MCP write path unchanged, so the API/MCP
layer is not the cause). The loss happens on the **relay hop**
`Agent(anxious:issuer-agent)` composing the actual Gitea call from the prompt
it's handed — because an HTML comment is, by design, invisible when the prompt
text is read as markdown; an agent relaying "what the issue should say" rather
than pasting the string byte-for-byte can drop a line it never visually saw.
That is why `skills/offload-audit/SKILL.md` §5d now fences the rendered body
in the delegation prompt instead of inlining it as bare markdown — a fenced
block reads as literal text, not as renderable markup, to the relaying agent.
`bin/filing-plan`'s `find_existing()` also now matches the bare, unwrapped
form as a backward-compatibility fallback, so issues already filed without the
wrapper (like cluster#56/#57) are still found — but that fallback exists only
to avoid duplicate-filing on *already-broken* issues, not as a second
acceptable format going forward.
## issue-title
A proposal. `issuer` may rewrite it.
+78 -17
View File
@@ -109,27 +109,67 @@ complete including the overrule case) lives in that script.
## 4b. Check the evidence against the previous window
The second hard gate (FR-4.5, rubric §5b). It runs on every audit, before
anything is filed, and its output **replaces** `classified.json` downstream:
anything is filed, and its output **replaces** `classified.json` downstream.
**First, unconditionally — not just when the store looks cold — fetch the
wiki's copy of this target's history and seed the local store from it.** The
local store this gate reads is a filesystem-only mirror (no `bin/` script
holds wiki credentials); it never talks to the wiki itself. That means a store
that is empty *locally* is not evidence the target has no history — it may
just be a cache nobody has seeded on this machine yet, and treating the two as
the same thing is exactly the bug behind
kotkan/claude-plugin-inference-arbitrage#17: a candidate silently filed as
`unchecked` when real prior-window history existed on the wiki all along.
`import` dedupes by `run_id`, so running this every time is safe and
idempotent — there is no "if cold" branch to reason about:
```bash
Agent(subagent_type="cluster:gitea-agent", prompt=
"wiki_read Data/<name>/snapshots.jsonl on
kotkan/claude-plugin-inference-arbitrage (or the target's own repo, per
how this target's store is organized). If the page exists, save its
content to a local file and report the path. If it does not exist, say
so plainly.")
# if the page existed:
$IA/bin/audit-snapshot import --target <name> --snapshots <fetched snapshots.jsonl>
```
Then run the gate:
```bash
$IA/bin/stability-classify --classified classified.json --target <name> \
--scan scan.json > stable.json
--scan scan.json --wiki-checked <empty|imported> > stable.json
```
`--wiki-checked` is mandatory whenever the local store turns out empty for
this target: pass `empty` if the wiki fetch above genuinely found no page (or
no history), or `imported` if it did and you ran `audit-snapshot import`. The
script hard-errors if the store is empty and this flag is omitted — the wiki
check is no longer a documentation aside the gate quietly trusts you did.
It matches each `file` candidate against the most recent prior snapshot that
recorded it (FR-5.2 identity) and downgrades any whose `measurement_strength`
changed, or whose share crossed the 2% filing threshold, between the two
windows. **Use `stable.json` for steps 5 and 6.** Never file from
`classified.json` once this has run.
If the store has no history for this target — a first audit — every candidate
comes back `stability: unchecked` and files normally. That is the FR-2.4
outcome, not a failure. If the store is cold but the wiki has history, seed it
first, or the gate has nothing to check against:
One comparison it deliberately refuses to make (FR-4.5.1): when the prior window
read `measured`, this window reads `thin`/`unmeasured`, **and** this window is
narrower than 90% of that prior window's width, the candidate comes back with
verdict **`insufficient-window`** and lands in `stability_deferrals`, not
`stability_findings`. A low-frequency-but-real candidate looks thin in any short
window — that is what kotkan/claude-plugin-inference-arbitrage#15 turned out to
be — so this is neither a filing nor a defect: **do not file it on the target's
repo, and do not self-report it on this plugin's repo.** Mention it in the run
summary as "window too narrow to compare" and, if you want the answer, re-audit
over at least the `carry_forward` width the deferral names. Flips between
comparably wide windows, and the reverse direction (`thin` prior → `measured`
now), still downgrade and still self-report.
```bash
$IA/bin/audit-snapshot import --target <name> --snapshots <fetched snapshots.jsonl>
```
If the store has no history for this target even after the wiki fetch above —
a genuine first audit — every candidate comes back `stability: unchecked` and
files normally. That is the FR-2.4 outcome, not a failure.
### 4b-i. A downgrade is a finding against *this* plugin
@@ -221,14 +261,31 @@ rendered. **Execute the plan verbatim — do not rewrite a body, and never turn
One `Agent` call per action (or one call carrying all of them):
```text
**The body MUST be embedded inside its own fenced code block in the prompt, not
inlined as bare markdown.** kotkan/claude-plugin-inference-arbitrage#23 found
that two issues filed this way (oleks/claude-plugin-cluster#56, #57) reached
Gitea with the leading `<!-- ia-candidate: ... -->` marker's HTML-comment
wrapper stripped. The write path itself was verified clean (a literal `<!--
-->` round-trips through the Gitea MCP write unchanged); the loss happens on
the relay hop, because an HTML comment is invisible when its surrounding text
is read as markdown — `issuer-agent` can only paste a line byte-for-byte if it
was actually shown one, not markdown that silently ate it. A fenced block
(` ``` `) forces the body to be read and relayed as literal text.
````text
Agent(subagent_type="anxious:issuer-agent", prompt=...)
# for action == "file"
"File this on <owner/repo>. It is an inference-arbitrage audit finding.
Proposed title: <proposed_title> (yours to rewrite per your taxonomy)
Body — file VERBATIM, the first line is an idempotency marker that a later
audit searches for and MUST NOT be altered or reformatted:
Body — copy the fenced block below byte-for-byte as the issue body. Do not
retype, paraphrase, reformat, or 'clean up' it — the first line is an
idempotency marker (an HTML comment, `<!-- ia-candidate: ... -->`) that a
later audit's marker search depends on, and it renders invisible on Gitea
by design, so you will not visually see it in the rendered issue — that is
expected, do not treat it as accidental syntax to strip:
```
<body>
```
Labels: 'token-offload' is required — create it on this repo if absent
(colour #8b5cf6, 'A step paying for inference where a deterministic script
would do'). Add the four-axis taxonomy labels and any milestone per your own
@@ -238,11 +295,15 @@ Agent(subagent_type="anxious:issuer-agent", prompt=...)
# for action == "comment"
"Add this comment to <issue>, an existing inference-arbitrage finding being
re-measured. Do not open a new issue. Post the body VERBATIM (the first line
is an idempotency marker). If the issue is closed, use your judgement about
reopening; the audit does not require it.
<body>"
```
re-measured. Do not open a new issue. Copy the fenced block below
byte-for-byte as the comment body — do not retype or reformat it, the first
line is an invisible-by-design HTML-comment idempotency marker (see the
'file' case above for why). If the issue is closed, use your judgement
about reopening; the audit does not require it.
```
<body>
```"
````
`issuer` owns repo, final title, labels beyond `token-offload`, and milestone.
You own the body.
+26 -5
View File
@@ -50,7 +50,7 @@ rather than in agent prose (FR-8.1 — the plugin obeying its own thesis).
| --- | --- | --- |
| 1 (S2) | `token-budget` | zero `high`-confidence candidates; conclusion "nothing to offload here" |
| 2 (S3) | `worktree-discipline`, masked | `sweep-worktrees` worktree-safety classification at `high`, position `llm-over-script-digest` |
| 3 (S4) | `anxious`, two windows | `release-policy-derivation` downgraded to a boundary question by the FR-4.5 stability gate, with no human in the loop |
| 3 (S4) | `anxious`, two windows | `release-policy-derivation` refused by the FR-4.5 stability gate — as `insufficient-window` (FR-4.5.1), the 7-day window being too narrow to contradict the 26-day `measured` one — with no human in the loop |
## The two-window `anxious` fixture — gate 3
@@ -68,11 +68,32 @@ actually got filed, not a case constructed to be caught:
- `anxious-7d.scan.json` — the 7-day scan, for signature identity. Masked
shapes and counts only (FR-3.5); no transcript content.
- `anxious-60d.scan.json` — the real 60-day scan
(`bin/offload-scan --plugin anxious --days 60`, 2026-06-26 → 2026-07-30, 176
sessions), added while resolving
kotkan/claude-plugin-inference-arbitrage#15. Same masking rules.
- `anxious-60d.classified.json` — the same candidate's unchanged rubric grade
with the 60-day window's real measurement: 15 invocations, `offload_value`
4,953,520 of 35,722,973 weighted tokens = 13.87% of audited spend, `measured`.
It carries only that one candidate, so no other row can be read as a 60-day
number it was not measured over.
The test excludes the 7-day run from the store so it is the run being
classified, exactly as at filing time, and asserts the gate downgrades it. The
comparison is symmetric: classifying the 30-day window against the 7-day
snapshot downgrades it too. Both windows cannot be right, which is the finding —
recorded as kotkan/claude-plugin-inference-arbitrage#11.
classified, exactly as at filing time, and asserts the gate **refuses** it. What
it does *not* assert any more is that the refusal is a downgrade. The comparison
is not symmetric, and treating it as symmetric was the bug behind
kotkan/claude-plugin-inference-arbitrage#15: measuring two further real windows
gave 0 → 2 → 12 → 15 invocations at 7, 14, 27 and 60 days, i.e. a candidate that
runs about once every two days and that a 7-day window simply never sees. So the
7-day-against-26-day pairing is now `insufficient-window` (FR-4.5.1) with no
self-report, while the 60-day-against-26-day pairing — two adequately wide
windows, `measured` both times, same side of the 2% threshold — is `stable` and
files normally. Both directions are asserted, so a regression in either shows up
as a test failure rather than as a wrong issue on someone's tracker.
The original two-window finding, and the fact that both windows cannot be right
about a candidate they sample equally, is recorded as
kotkan/claude-plugin-inference-arbitrage#11.
The candidate matches by **slug**, not signature: the 7-day scan produced no
recurring n-gram for that skill, so its signature falls back to the slug and the
@@ -0,0 +1,42 @@
{
"candidates": [
{
"boundary_confidence": "high",
"candidate_id": "anxious/agent-wip/release-policy-derivation",
"category": "usage",
"determinism_tests": {
"T1": true,
"T2": true,
"T3": true,
"T4": true,
"T5": true
},
"falsifiability_triple": {
"examples": true,
"overrule_case": true,
"signature": true
},
"measurement_strength": "measured",
"offload_value": 4953520,
"position": "llm-over-script-digest",
"reasons": [
"all five determinism tests pass; triple complete; escalation path defined",
"measured over the 60-day window: 15 invocations, offload_value 4,953,520 weighted tokens, 13.87% of audited spend"
],
"share_of_audited_spend": 0.1387,
"skill": "anxious:agent-wip",
"verdict": "file"
}
],
"rubric_version": "1.0.0",
"summary": {
"boundary_questions": 0,
"candidates": 1,
"conclusion": "1 high-confidence candidate(s)",
"drift_notes": 0,
"high_confidence": 1,
"no_offload": 0,
"to_file": 1
},
"target": "anxious (60-day window)"
}
+335
View File
@@ -0,0 +1,335 @@
{
"window": {
"since": "2026-06-26T17:41:40.988000+00:00",
"until": "2026-07-30T02:30:59.111000+00:00",
"requested_since": "2026-05-31T02:30:03.118644+00:00",
"requested_until": null,
"sessions": 176
},
"plugin": "anxious",
"tool_versions": {
"cc_tokens_path": "/home/oleks/projects/claude-plugins/token-budget/bin/cc-tokens"
},
"coverage": {
"attributed_turns": 5197,
"candidate_turns": 249648,
"ratio": 0.021,
"caveat": "hook-driven and unattributed agent turns excluded"
},
"totals": {
"weighted_tokens": 35722973,
"cost_estimate_usd": 178.61,
"invocations": 700,
"mechanical_share": 0.106
},
"skills": [
{
"skill": "anxious:agent-wip",
"invocations": 15,
"turns": 375,
"sidechain_turns": 0,
"weighted_tokens": 14007053,
"cost_estimate_usd": 70.04,
"share_of_plugin": 0.392,
"mtr": 0.144,
"judgment_density": 0.027,
"retry_density": 0.069,
"offload_waste": 1532536,
"read_amplification": 383.83,
"fanout_multiplier": 1.0,
"repetition_factor": 3.322,
"ngrams": [
{
"sig": [
"label_write(color=<str>, method=<str>, name=<str>, owner=<str>, repo=<str>)",
"label_write(color=<str>, method=<str>, name=<str>, owner=<str>, repo=<str>)",
"issue_write(index=<n>, labels=<list>, method=<str>, owner=<str>, repo=<str>)"
],
"recurrences": 5,
"invocations": 4,
"waste": 345101
},
{
"sig": [
"Edit(file_path=<path>, new_string=<str>, old_string=<str>, replace_all=<bool>)",
"Read(file_path=<path>)",
"Edit(file_path=<path>, new_string=<str>, old_string=<str>, replace_all=<bool>)"
],
"recurrences": 3,
"invocations": 3,
"waste": 342125
},
{
"sig": [
"Read(file_path=<path>)",
"Read(file_path=<path>)",
"Bash(command=grep <str>, description=<str>)"
],
"recurrences": 3,
"invocations": 3,
"waste": 302043
},
{
"sig": [
"Bash(command=echo <str>, description=<str>)",
"issue_read(index=<n>, method=<str>, owner=<str>, repo=<str>)",
"issue_read(index=<n>, method=<str>, owner=<str>, repo=<str>)"
],
"recurrences": 3,
"invocations": 3,
"waste": 299691
},
{
"sig": [
"ToolSearch(max_results=<n>, query=<str>)",
"issue_read(index=<n>, method=<str>, owner=<str>, repo=<str>)",
"issue_read(index=<n>, method=<str>, owner=<str>, repo=<str>)"
],
"recurrences": 4,
"invocations": 4,
"waste": 258683
}
],
"offload_value": 4953520
},
{
"skill": "anxious:delivery-flow",
"invocations": 1,
"turns": 1,
"sidechain_turns": 0,
"weighted_tokens": 48152,
"cost_estimate_usd": 0.24,
"share_of_plugin": 0.001,
"mtr": 0.0,
"judgment_density": 0.0,
"retry_density": 0.0,
"offload_waste": 0,
"read_amplification": 378.0,
"fanout_multiplier": 1.0,
"repetition_factor": 1.0,
"ngrams": [],
"offload_value": 0
}
],
"agents": [
{
"agent": "anxious:issuer-agent",
"invocations": 608,
"turns": 4207,
"sidechain_turns": 4207,
"weighted_tokens": 13682748,
"cost_estimate_usd": 68.41,
"share_of_plugin": 0.383,
"mtr": 0.097,
"judgment_density": 0.051,
"retry_density": 0.082,
"offload_waste": 778919,
"read_amplification": 545.98,
"fanout_multiplier": 0.2,
"repetition_factor": 7.741,
"ngrams": [
{
"sig": [
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)",
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)",
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)"
],
"recurrences": 107,
"invocations": 47,
"waste": 874165
},
{
"sig": [
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)",
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)",
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)",
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)"
],
"recurrences": 56,
"invocations": 25,
"waste": 496902
},
{
"sig": [
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)",
"issue_read(index=<n>, method=<str>, owner=<str>, repo=<str>)",
"issue_read(index=<n>, method=<str>, owner=<str>, repo=<str>)"
],
"recurrences": 28,
"invocations": 26,
"waste": 318874
},
{
"sig": [
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)",
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)",
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)",
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)",
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)"
],
"recurrences": 31,
"invocations": 10,
"waste": 207299
},
{
"sig": [
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)",
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)",
"issue_read(index=<n>, method=<str>, owner=<str>, repo=<str>)"
],
"recurrences": 13,
"invocations": 13,
"waste": 184334
}
],
"offload_value": 5722452
},
{
"agent": "repo-worker",
"invocations": 2,
"turns": 268,
"sidechain_turns": 268,
"weighted_tokens": 1816949,
"cost_estimate_usd": 9.08,
"share_of_plugin": 0.051,
"mtr": 0.19,
"judgment_density": 0.004,
"retry_density": 0.075,
"offload_waste": 303294,
"read_amplification": 650.6,
"fanout_multiplier": 0.66,
"repetition_factor": 1.0,
"ngrams": [],
"offload_value": 302081
},
{
"agent": "anxious:steward-agent",
"invocations": 56,
"turns": 225,
"sidechain_turns": 225,
"weighted_tokens": 5597218,
"cost_estimate_usd": 27.99,
"share_of_plugin": 0.157,
"mtr": 0.009,
"judgment_density": 0.08,
"retry_density": 0.076,
"offload_waste": 17835,
"read_amplification": 677.33,
"fanout_multiplier": 0.34,
"repetition_factor": 7.728,
"ngrams": [
{
"sig": [
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)",
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)",
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)"
],
"recurrences": 106,
"invocations": 29,
"waste": 2987563
},
{
"sig": [
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)",
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)",
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)",
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)"
],
"recurrences": 77,
"invocations": 22,
"waste": 2434027
},
{
"sig": [
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)",
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)",
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)",
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)",
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)"
],
"recurrences": 55,
"invocations": 15,
"waste": 1816278
},
{
"sig": [
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)",
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)",
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)",
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)",
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)",
"Agent(description=<str>, prompt=<str>, run_in_background=<bool>, subagent_type=<str>)"
],
"recurrences": 40,
"invocations": 8,
"waste": 1196514
},
{
"sig": [
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)",
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)",
"Agent(description=<str>, prompt=<str>, subagent_type=<str>)"
],
"recurrences": 5,
"invocations": 3,
"waste": 205780
}
],
"offload_value": 126805
},
{
"agent": "anxious:docs-agent",
"invocations": 10,
"turns": 98,
"sidechain_turns": 98,
"weighted_tokens": 502645,
"cost_estimate_usd": 2.51,
"share_of_plugin": 0.014,
"mtr": 0.276,
"judgment_density": 0.061,
"retry_density": 0.051,
"offload_waste": 107413,
"read_amplification": 164.08,
"fanout_multiplier": 0.34,
"repetition_factor": 1.0,
"ngrams": [],
"offload_value": 100861
},
{
"agent": "feature-dev:code-reviewer",
"invocations": 3,
"turns": 4,
"sidechain_turns": 4,
"weighted_tokens": 26969,
"cost_estimate_usd": 0.13,
"share_of_plugin": 0.001,
"mtr": 0.75,
"judgment_density": 0.25,
"retry_density": 0.0,
"offload_waste": 13850,
"read_amplification": 797.8,
"fanout_multiplier": 0.09,
"repetition_factor": 1.0,
"ngrams": [],
"offload_value": 10388
},
{
"agent": "cluster:gitea-agent",
"invocations": 5,
"turns": 19,
"sidechain_turns": 19,
"weighted_tokens": 41235,
"cost_estimate_usd": 0.21,
"share_of_plugin": 0.001,
"mtr": 0.105,
"judgment_density": 0.158,
"retry_density": 0.0,
"offload_waste": 4163,
"read_amplification": 299.0,
"fanout_multiplier": 0.24,
"repetition_factor": 1.0,
"ngrams": [],
"offload_value": 3505
}
]
}
+37
View File
@@ -112,6 +112,43 @@ check "found via the comment" "$(q "$tmp/plan3.json" "d['actions'][0]['action']"
check "closed state surfaced for issuer" \
"$(q "$tmp/plan3.json" "d['actions'][0]['issue_state']")" "closed"
echo "== kotkan/claude-plugin-inference-arbitrage#23: a marker that lost its"
echo " HTML-comment wrapper on write (oleks/claude-plugin-cluster#56, #57)"
echo " is still found — backward compat, not a second accepted format =="
python3 - "$tmp/plan1.json" "$tmp/bare.json" <<'PY'
import json, sys
plan = json.load(open(sys.argv[1]))
filed = next(a for a in plan["actions"] if a["action"] == "file")
# Same body, but the first line has lost its "<!-- " / " -->" wrapper — the
# exact shape observed on the live tracker.
bare_body = filed["body"].replace(
"<!-- ia-candidate: fixture-plugin/skill-a/label-derivation -->",
"ia-candidate: fixture-plugin/skill-a/label-derivation",
)
json.dump([{"number": 56, "state": "closed", "title": filed["proposed_title"],
"body": bare_body, "comments": []}], open(sys.argv[2], "w"))
PY
plan "$tmp/bare.json" "$tmp/plan-bare.json"
check "bare marker still found, no duplicate filed" \
"$(q "$tmp/plan-bare.json" "d['summary']['to_file']")" "0"
check "recognized as a comment on the existing issue" \
"$(q "$tmp/plan-bare.json" "d['actions'][0]['action']")" "comment"
check "on issue 56, matching the observed failure" \
"$(q "$tmp/plan-bare.json" "d['actions'][0]['issue_number']")" "56"
echo "== a bare marker for a DIFFERENT id still does not false-match =="
python3 - "$tmp/bare.json" "$tmp/bare-other.json" <<'PY'
import json, sys
issues = json.load(open(sys.argv[1]))
issues[0]["body"] = issues[0]["body"].replace(
"fixture-plugin/skill-a/label-derivation", "fixture-plugin/skill-a/label-derivation-longer",
)
json.dump(issues, open(sys.argv[2], "w"))
PY
plan "$tmp/bare-other.json" "$tmp/plan-bare-other.json"
check "different id (even as a prefix) still files fresh" \
"$(q "$tmp/plan-bare-other.json" "d['summary']['to_file']")" "1"
echo "== real Gitea API shapes parse, and a DIFFERENT id does not false-match =="
plan tests/fixtures/gitea-issues.json "$tmp/plan4.json"
check "unrelated markers -> files" "$(q "$tmp/plan4.json" "d['summary']['to_file']")" "1"
+238 -14
View File
@@ -33,6 +33,16 @@ gatefield() {
python3 -c "import json,sys;print(json.load(open(sys.argv[1]))['stability_gate'][sys.argv[2]])" "$@"
}
# scanwin <file> <since> <until>
# The only field the gate reads out of a scan for the FR-4.5 parity rule is the
# window's observed span, so the fixture carries exactly that.
scanwin() {
python3 -c "
import json, sys
json.dump({'window': {'since': sys.argv[2], 'until': sys.argv[3], 'sessions': 10}},
open(sys.argv[1], 'w'))" "$@"
}
# classified <file> <strength> <share>
# One position-3 candidate the boundary gate already graded `file`.
classified() {
@@ -59,23 +69,26 @@ json.dump({
EOF
}
# snapshot-store <store> <strength> <share>
# A prior snapshot recording the same candidate by slug identity.
# snapshot-store <store> <strength> <share> [since] [until]
# A prior snapshot recording the same candidate by slug identity. The window
# defaults to 30 days; the width matters for the FR-4.5 parity rule (cases j-m).
prior() {
local store=$1
mkdir -p "$store/Data/synth"
python3 - "$store/Data/synth/snapshots.jsonl" "$2" "$3" <<'EOF'
python3 - "$store/Data/synth/snapshots.jsonl" "$2" "$3" "${4:-2026-06-01}" "${5:-2026-07-01}" <<'EOF'
import json, sys
out, strength, share = sys.argv[1:]
out, strength, share, since, until = sys.argv[1:]
json.dump({
"run_id": "2026-07-01T00-00-00Z",
"window": {"since": "2026-06-01", "until": "2026-07-01"},
"window": {"since": since, "until": until},
"target": {"name": "synth", "version": "1.0.0", "skills": ["synth:skill"]},
"coverage": {"ratio": 0.1, "attributed_turns": 500},
"totals": {"weighted_tokens": 5000000},
"candidates": [{
"candidate_id": "synth/skill/step",
"skill": "synth:skill",
"position": "llm-over-script-digest",
"boundary_confidence": "high",
"signature_hash": "slugonly",
"measurement_strength": strength,
"measurement": {"invocations": 10, "offload_value": 500000,
@@ -141,9 +154,11 @@ print(' ok stability names the window it was checked against')
# --------------------------------------------------------------------------
echo "== (c) no prior snapshot -> files normally, unchecked (FR-2.4) =="
# An absence of history is not evidence of instability. A first audit of a
# target must still be able to file.
# target must still be able to file. The caller must still assert the wiki
# was checked (kotkan/claude-plugin-inference-arbitrage#17) -- see (g)/(h).
classified "$tmp/c.json" measured 0.12
$BIN --classified "$tmp/c.json" --target synth --store "$tmp/store-empty" >"$tmp/c.out.json"
$BIN --classified "$tmp/c.json" --target synth --store "$tmp/store-empty" \
--wiki-checked=empty >"$tmp/c.out.json"
check "verdict" "$(field "$tmp/c.out.json" synth/skill/step verdict)" "file"
check "stability" "$(field "$tmp/c.out.json" synth/skill/step stability)" \
"unchecked — no prior window to compare"
@@ -180,12 +195,20 @@ check "not counted as checked" "$(gatefield "$tmp/e.out.json" checked)" "0"
check "no finding" "$(gatefield "$tmp/e.out.json" downgraded)" "0"
# --------------------------------------------------------------------------
echo "== (f) S4 — the real anxious candidate is caught with no human in the loop =="
echo "== (f) S4 — the real anxious 7-day window is refused, as insufficient-window =="
# The exact evidence from kotkan/claude-plugin-inference-arbitrage#11: the
# 30-day window graded `anxious/agent-wip/release-policy-derivation` measured at
# ~26-day window graded `anxious/agent-wip/release-policy-derivation` measured at
# 11.94% of audited spend, the 7-day window graded the same candidate thin at
# 0.00%. Both windows' snapshots are in the store; --exclude-run makes the 7-day
# run the one being classified, exactly as it is at filing time.
#
# S4 still holds — the candidate is mechanically refused with no human in the
# loop — but per kotkan/claude-plugin-inference-arbitrage#15 the honest reason is
# not "unstable". Re-measuring the same candidate over 14/27/60 days gave 0 -> 2
# -> 12 -> 15 invocations: a monotonic accumulation of a ~once-every-two-days
# step, so the 7-day window under-sampled it rather than contradicting it. A
# 7.0-day window cannot overturn a 26.1-day `measured` reading, so this is a
# deferral, not a downgrade, and nothing is self-reported against this plugin.
store=$tmp/store-anxious
mkdir -p "$store/Data/anxious"
cp calibration/anxious-windows.snapshots.jsonl "$store/Data/anxious/snapshots.jsonl"
@@ -194,25 +217,226 @@ $BIN --classified calibration/anxious-7d.classified.json \
--scan calibration/anxious-7d.scan.json \
--exclude-run 2026-07-29T01-00-00Z >"$tmp/f.out.json"
check "verdict" "$(field "$tmp/f.out.json" anxious/agent-wip/release-policy-derivation verdict)" \
"boundary-question"
check "downgraded" "$(gatefield "$tmp/f.out.json" downgraded)" "1"
"insufficient-window"
check "downgraded" "$(gatefield "$tmp/f.out.json" downgraded)" "0"
check "insufficient_window" "$(gatefield "$tmp/f.out.json" insufficient_window)" "1"
check "findings" "$(python3 -c "import json;print(len(json.load(open('$tmp/f.out.json'))['stability_findings']))")" "0"
python3 -c "
import json
d=json.load(open('$tmp/f.out.json'))
f=[x for x in d['stability_findings']
f=[x for x in d['stability_deferrals']
if x['candidate_id']=='anxious/agent-wip/release-policy-derivation'][0]
assert f['prior']['measurement_strength']=='measured', f['prior']
assert abs(f['prior']['share_of_audited_spend']-0.1194) < 1e-9, f['prior']
assert f['current']['measurement_strength']=='thin', f['current']
assert f['current']['share_of_audited_spend']==0.0, f['current']
assert f['prior']['run_id']=='2026-07-29T00-00-00Z', f['prior']
print(' ok finding carries the real numbers: measured 11.94% vs thin 0.00%')
p=f['window_parity']
assert p['comparable'] is False, p
assert abs(p['current_days']-7.0) < 0.05, p
assert 25.9 < p['prior_days'] < 26.2, p
assert {x['field'] for x in f['suppressed_flips']} == {
'measurement_strength','share_of_audited_spend'}, f['suppressed_flips']
assert 'self_report' not in f, f
print(' ok deferral carries the real widths (7.0d vs 26.1d) and the suppressed flips')
" || fail=1
python3 -c "
import json
d=json.load(open('$tmp/f.out.json'))
assert d['summary']['to_file']==0, d['summary']
print(' ok nothing is left to file from that window')
assert d['summary']['boundary_questions']==2, d['summary']
r=[c for c in d['candidates']
if c['candidate_id']=='anxious/agent-wip/release-policy-derivation'][0]['reasons'][-1]
assert 'DEFERRED (FR-4.5)' in r and 'carried' in r, r
print(' ok not filed, not counted as a boundary question, reason cites the deferral')
" || fail=1
# --------------------------------------------------------------------------
echo "== (f2) real 60-day window against the real 27-day prior -> reports normally =="
# The other half of kotkan/claude-plugin-inference-arbitrage#15's evidence: the
# 60-day window (15 invocations, offload_value 4,953,520 of 35,722,973 weighted
# tokens = 13.87%) against the ~27-day `measured` 11.94% prior. Two adequately
# wide windows, same label, same side of the 2% threshold -> stable, files
# normally. The parity rule must not suppress this: the fix is about a narrow
# CURRENT window, not about refusing to ever conclude anything.
#
# --exclude-run drops the 7-day snapshot so the pairing under test really is
# 27d-vs-60d; the 60d-against-7d pairing is the reverse direction, and it still
# downgrades — see (k).
$BIN --classified calibration/anxious-60d.classified.json \
--target anxious --store "$store" \
--scan calibration/anxious-60d.scan.json \
--exclude-run 2026-07-29T01-00-00Z >"$tmp/f2.out.json"
check "verdict" "$(field "$tmp/f2.out.json" anxious/agent-wip/release-policy-derivation verdict)" \
"file"
check "downgraded" "$(gatefield "$tmp/f2.out.json" downgraded)" "0"
check "insufficient_window" "$(gatefield "$tmp/f2.out.json" insufficient_window)" "0"
python3 -c "
import json
d=json.load(open('$tmp/f2.out.json'))
c=d['candidates'][0]
assert c['stability'].startswith('stable against 2026-07-29T00-00-00Z'), c['stability']
assert abs(c['share_of_audited_spend']-0.1387) < 1e-9, c['share_of_audited_spend']
print(' ok the 60-day reading (13.87%) is stable against the 27-day one (11.94%)')
" || fail=1
# --------------------------------------------------------------------------
echo "== (g) empty store, no --wiki-checked -> hard error (kotkan/claude-plugin-inference-arbitrage#17) =="
# The bug: a cold local cache and a genuinely history-free wiki look identical
# to this script, which only ever reads the filesystem. Without an explicit
# assertion from the caller, refuse rather than silently degrade to unchecked.
classified "$tmp/g.json" measured 0.12
if $BIN --classified "$tmp/g.json" --target synth --store "$tmp/store-empty-g" \
>"$tmp/g.out.json" 2>"$tmp/g.err"; then
echo " FAIL expected non-zero exit, got success" >&2
fail=1
else
echo " ok exits non-zero"
fi
if grep -q "wiki-checked" "$tmp/g.err"; then
echo " ok error names --wiki-checked"
else
echo " FAIL error does not mention --wiki-checked: $(cat "$tmp/g.err")" >&2
fail=1
fi
# --------------------------------------------------------------------------
echo "== (h) empty store, --wiki-checked=empty -> unchecked, as today =="
# The caller checked the wiki and it really is history-free; behavior matches (c).
classified "$tmp/h.json" measured 0.12
$BIN --classified "$tmp/h.json" --target synth --store "$tmp/store-empty-h" \
--wiki-checked=empty >"$tmp/h.out.json"
check "verdict" "$(field "$tmp/h.out.json" synth/skill/step verdict)" "file"
check "stability" "$(field "$tmp/h.out.json" synth/skill/step stability)" \
"unchecked — no prior window to compare"
# --------------------------------------------------------------------------
echo "== (i) empty store seeded via a real import, --wiki-checked=imported -> gate runs normally =="
# After `audit-snapshot import` seeds the store from a wiki-fetched
# snapshots.jsonl, the store is no longer empty, so the gate compares against
# the imported history exactly as if it had been local all along.
store=$tmp/store-import-i
mkdir -p "$store"
prior "$tmp/seed-i" thin 0.0
IMPORT_BIN=../bin/audit-snapshot
$IMPORT_BIN --store "$store" import --target synth \
--snapshots "$tmp/seed-i/Data/synth/snapshots.jsonl" >/dev/null
classified "$tmp/i.json" measured 0.12
$BIN --classified "$tmp/i.json" --target synth --store "$store" \
--wiki-checked=imported >"$tmp/i.out.json"
check "verdict" "$(field "$tmp/i.out.json" synth/skill/step verdict)" "boundary-question"
check "downgraded" "$(gatefield "$tmp/i.out.json" downgraded)" "1"
# --------------------------------------------------------------------------
echo "== (j) narrow current window, measured wide prior -> insufficient-window =="
# The kotkan/claude-plugin-inference-arbitrage#15 shape, synthetically: a 30-day
# window measured the candidate at 12%, a 7-day window sees nothing. 7d is below
# the 27.0d parity floor, so the weak reading does not get to overturn the
# measured one — neither filed nor downgraded, and no self-report.
classified "$tmp/j.json" thin 0.0
prior "$tmp/store-j" measured 0.12
scanwin "$tmp/j.scan.json" 2026-07-23 2026-07-30
$BIN --classified "$tmp/j.json" --target synth --store "$tmp/store-j" \
--scan "$tmp/j.scan.json" >"$tmp/j.out.json"
check "verdict" "$(field "$tmp/j.out.json" synth/skill/step verdict)" "insufficient-window"
check "downgraded" "$(gatefield "$tmp/j.out.json" downgraded)" "0"
check "insufficient_window" "$(gatefield "$tmp/j.out.json" insufficient_window)" "1"
check "self-reports" "$(python3 -c "import json;print(len(json.load(open('$tmp/j.out.json'))['stability_findings']))")" "0"
python3 -c "
import json
d=json.load(open('$tmp/j.out.json'))
f=d['stability_deferrals'][0]
p=f['window_parity']
assert p['prior_days']==30.0 and p['current_days']==7.0 and p['required_days']==27.0, p
assert p['comparable'] is False and p['parity_ratio']==0.9, p
assert 'at least 27.0d' in f['carry_forward'], f['carry_forward']
assert f['prior']['run_id']=='2026-07-01T00-00-00Z', f['prior']
s=d['candidates'][0]['stability']
assert s.startswith('insufficient-window against 2026-07-01T00-00-00Z'), s
assert d['summary']['to_file']==0 and d['summary']['boundary_questions']==0, d['summary']
assert d['summary']['stability_insufficient_window']==1, d['summary']
print(' ok deferral names both widths, the floor, and the window it defers against')
" || fail=1
# --------------------------------------------------------------------------
echo "== (k) reverse direction — thin wide prior, measured narrow current -> downgraded =="
# The narrowness carve-out is one-directional. Here the WIDE window is the one
# that saw nothing and the narrow one measures 12%: the weak reading is the
# earlier one, the contradiction is genuine, and the human would still be told
# two different things. Downgrade, exactly as before the fix.
classified "$tmp/k.json" measured 0.12
prior "$tmp/store-k" thin 0.0
scanwin "$tmp/k.scan.json" 2026-07-23 2026-07-30
$BIN --classified "$tmp/k.json" --target synth --store "$tmp/store-k" \
--scan "$tmp/k.scan.json" >"$tmp/k.out.json"
check "verdict" "$(field "$tmp/k.out.json" synth/skill/step verdict)" "boundary-question"
check "downgraded" "$(gatefield "$tmp/k.out.json" downgraded)" "1"
check "insufficient_window" "$(gatefield "$tmp/k.out.json" insufficient_window)" "0"
python3 -c "
import json
d=json.load(open('$tmp/k.out.json'))
f=d['stability_findings'][0]
assert f['self_report']['repo']=='kotkan/claude-plugin-inference-arbitrage', f['self_report']
p=f['window_parity']
assert p['comparable'] is False, p
print(' ok still self-reported, and the parity block records the widths anyway')
" || fail=1
# --------------------------------------------------------------------------
echo "== (l) two comparably wide windows, genuine flip -> downgraded =="
# 28 days against 30 is at the parity floor (27.0d), so the windows are
# comparably sampled and a measured -> thin flip is real instability.
classified "$tmp/l.json" thin 0.0
prior "$tmp/store-l" measured 0.12
scanwin "$tmp/l.scan.json" 2026-07-02 2026-07-30
$BIN --classified "$tmp/l.json" --target synth --store "$tmp/store-l" \
--scan "$tmp/l.scan.json" >"$tmp/l.out.json"
check "verdict" "$(field "$tmp/l.out.json" synth/skill/step verdict)" "boundary-question"
check "downgraded" "$(gatefield "$tmp/l.out.json" downgraded)" "1"
check "insufficient_window" "$(gatefield "$tmp/l.out.json" insufficient_window)" "0"
python3 -c "
import json
p=json.load(open('$tmp/l.out.json'))['stability_findings'][0]['window_parity']
assert p['current_days']==28.0 and p['required_days']==27.0 and p['comparable'] is True, p
print(' ok 28.0d clears the 27.0d floor, so the flip stands')
" || fail=1
# --------------------------------------------------------------------------
echo "== (m) measured both windows, threshold crossed on a narrow window -> downgraded =="
# A window that still reached `measured` observed the candidate often enough; a
# share that then crosses the 2% filing threshold is a claim about relative
# spend, not a sampling failure, so narrowness does not excuse it.
classified "$tmp/m.json" measured 0.01
prior "$tmp/store-m" measured 0.12
scanwin "$tmp/m.scan.json" 2026-07-23 2026-07-30
$BIN --classified "$tmp/m.json" --target synth --store "$tmp/store-m" \
--scan "$tmp/m.scan.json" >"$tmp/m.out.json"
check "verdict" "$(field "$tmp/m.out.json" synth/skill/step verdict)" "boundary-question"
check "downgraded" "$(gatefield "$tmp/m.out.json" downgraded)" "1"
check "insufficient_window" "$(gatefield "$tmp/m.out.json" insufficient_window)" "0"
python3 -c "
import json
f=json.load(open('$tmp/m.out.json'))['stability_findings'][0]
assert {x['field'] for x in f['flips']} == {'share_of_audited_spend'}, f['flips']
print(' ok only the threshold flip is reported, and it is not excused')
" || fail=1
# --------------------------------------------------------------------------
echo "== (n) unreadable window widths -> downgraded, and said so explicitly =="
# No --scan means no current window bounds. Narrowness cannot be established, so
# the flip is NOT quietly excused — the finding records that the check could not
# run, rather than defaulting either way.
classified "$tmp/n.json" thin 0.0
prior "$tmp/store-n" measured 0.12
$BIN --classified "$tmp/n.json" --target synth --store "$tmp/store-n" >"$tmp/n.out.json"
check "verdict" "$(field "$tmp/n.out.json" synth/skill/step verdict)" "boundary-question"
check "downgraded" "$(gatefield "$tmp/n.out.json" downgraded)" "1"
python3 -c "
import json
p=json.load(open('$tmp/n.out.json'))['stability_findings'][0]['window_parity']
assert p['comparable'] is None and p['current_days'] is None, p
assert 'unreadable' in p['why'], p
print(' ok comparability is null with an explicit reason, not a silent exemption')
" || fail=1
exit "$fail"