The offload-analyst agent was hand-reshaping bin/plugin-inventory's
PluginInventory and bin/offload-scan's OffloadScan into a candidate
table on every audit run — 40.09% of this plugin's own audited spend
over 26 invocations (kotkan/claude-plugin-inference-arbitrage#24).
candidate_digest(inv, scan) -> list[CandidateRow] joins the two JSON
documents and computes shape: share_of_spend, mtr, judgment_density,
read_amplification, retry_density, fanout_multiplier,
dominant_ngram_recurrences, has_bin_script, and an advisory `flags`
list. It reuses boundary-classify's MIN_INVOCATIONS/
MIN_ATTRIBUTED_TURNS and ia_store's MIN_SHARE_OF_SPEND rather than
reinventing thresholds, and applies the judgment_density brake
(new JUDGMENT_DENSITY_BRAKE=0.3) so a row with real judgment density
never gets flagged even if every mechanical threshold is crossed.
This is a digest, not an auto-filing script (position 3, never
position 1): flags are advisory input to the analyst's own reading of
the actual n-gram/tool-call shape, never a verdict on their own.
bin/boundary-classify remains the only place a verdict is computed.
Wired into skills/offload-audit/SKILL.md step 3 and
agents/offload-analyst.md section 2 so future audit runs call the
script instead of reshaping by hand.
Tests: tests/candidate-digest.test.sh covers the issue's three worked
examples (scripted skill suppressed via has_bin_script, a flagged
agent row, and the judgment_density-brake-suppressed edge case) plus
a below-MIN_SHARE_OF_SPEND case and a full CandidateRow schema check.
Also verified against a real matched inv.json/scan.json pair from a
prior anxious audit in /tmp.
The stability gate treated every evidence flip as symmetric, and that
produced a false downgrade on the candidate it was written for
(kotkan/claude-plugin-inference-arbitrage#15). Re-measuring
anxious/agent-wip/release-policy-derivation over four real windows with
the same script gave 0 -> 2 -> 12 -> 15 invocations at 7/14/27/60 days:
monotonic accumulation of a step that runs about once every two days, not
a label flipping about. The 7-day window had not caught an unstable
candidate, it had failed to observe a real one.
So a thin/unmeasured reading now only overturns a `measured` one when its
window is at least 90% as wide (observed since->until span). Below that
the comparison takes a third verdict, `insufficient-window`: not filed,
not downgraded to a boundary question, no FR-6.5 self-report, and carried
forward with the wider snapshot left as the standing comparison point.
The parity is relative rather than an absolute day count because the
adequate width is a property of the candidate's invocation rate, which
the auditor does not know in advance.
Unchanged, and tested: the reverse direction (thin prior -> measured now)
still downgrades, a flip between comparably wide windows still downgrades,
and a threshold crossing with `measured` on both sides is never excused by
narrowness. An unreadable window is not an exemption either — the flip
stands and window_parity records that the check could not run.
- bin/stability-classify: window_days/window_parity/deferrable, the
insufficient-window verdict, stability_deferrals, window_parity on
findings too, insufficient_window + window_parity_ratio in the gate block
- bin/audit-snapshot: record deferrals in the snapshot notes so the row is
visible on the wiki page instead of silently dropped
- design/spec.md: FR-4.5.1 as a testable requirement; S4 restated
- design/rubric.md: the third boundary in §5b
- skills/offload-audit/SKILL.md: do not file and do not self-report a deferral
- tests: real 60-day anxious fixture; cases (f)/(f2) on real data, (j)-(n)
synthetic — 7 test files pass
Verified against the live store (~/.cache/inference-arbitrage/wiki): the
real 7.0d-vs-26.49d comparison now reports insufficient-window with zero
findings, while the real 33.4d-vs-26.49d one reports stable at 13.87% and
files normally.
kotkan/claude-plugin-inference-arbitrage#20 -- the bare alpine:3 CI
container has neither the Claude Code plugin cache nor `claude plugin
install`, so bin/offload-scan's cc-tokens PATH lookup always failed.
Per design decision 0.1 this plugin never vendors token-budget's
pricing/token-accounting logic, so fetch the single cc-tokens file
straight from kotkan/claude-plugin-token-budget's raw main branch and
put it on PATH before running tests/run-all.sh, rather than mocking
its behavior.
bin/stability-classify only ever reads the local filesystem snapshot store
(by design, no bin/ script holds wiki credentials), but SKILL.md step 4b
documented seeding it from the wiki as a conditional "if the store is cold"
aside. That made a cold-but-not-actually-empty local cache indistinguishable
from a genuine first audit: both silently emit stability: unchecked and file
unstable candidates, as observed on the 2026-07-30 anxious run.
- bin/stability-classify: add a required --wiki-checked {empty,imported}
flag; hard-error when the store is empty and the flag is omitted, instead
of silently degrading to unchecked.
- tests/stability.test.sh: (g) no flag on an empty store errors, (h)
--wiki-checked=empty behaves as before, (i) --wiki-checked=imported after a
real `audit-snapshot import` runs the gate normally against the imported
history.
- skills/offload-audit/SKILL.md: step 4b now fetches the wiki's
Data/<target>/snapshots.jsonl and runs `audit-snapshot import`
unconditionally (it dedupes by run_id, so this is safe every run) before
invoking the gate.
- Bump plugin.json to 0.6.3.
reconstruct_invocations() only sees already-attributed turns and splits on a
wall-clock idle gap (10 min default); it has no signal for unrelated
activity between two genuinely separate same-skill calls, so back-to-back
invocations closer together than the gap collapse into one reported
invocation. This inflates the headline per-invocation figures quoted in
filed issue bodies without affecting group totals (weighted_tokens,
offload_value, share_of_audited_spend).
Sidechain-leakage was investigated as the suspected root cause and ruled
out: turn attribution (the `mine` check) is correct, nothing session-wide
enters `turns`. The real, narrower defect is invocation-count inflation
from idle-gap merging.
Adds `totals.invocation_caveat` to offload-scan's JSON output, threads it
through audit-snapshot's snapshot totals, and renders it as a note in the
issue-body template so downstream issue bodies don't overstate confidence
in the per-invocation figure.
Fixeskotkan/claude-plugin-inference-arbitrage#12
kotkan/claude-plugin-inference-arbitrage#10 — a hook file is only counted
if it's referenced by .claude-plugin/hooks.json, executable, or has a
shebang. hooks/hooks.json itself is always excluded regardless, since it
is configuration rather than logic. mask-worktree-audit.py needed no
change: it derives script_loc from the already-filtered hooks[] output.
Bumps plugin.json to 0.6.2 (fix).
bin/boundary-classify defined its own copy of the spec §5.2.7 filing
threshold that only happened to match ia_store.py's copy, which
bin/stability-classify's FR-4.5 stability gate actually reads. Tuning
either constant independently would silently desynchronize the gate
from the classifier it's meant to gate. ia_store.py is now the single
definition; boundary-classify imports it.
MIN_INVOCATIONS/MIN_ATTRIBUTED_TURNS are left as-is: stability-classify
does not import or use them, so their duplication doesn't create the
same fail-silently risk. Centralizing all thresholds via
references/signals-catalog.md remains a separate, larger follow-up per
the issue's "Related / proper fix" note.
Fixeskotkan/claude-plugin-inference-arbitrage#14
README replaced (was still 'scaffolding, pre-implementation' at v0.5.1 with
two real audits already filed): usage, the six pipeline steps, the rubric's
four positions in one table, and three worked examples from real runs
(oleks/claude-plugin-cluster#56, oleks/claude-plugin-anxious#29,
kotkan/claude-plugin-inference-arbitrage#15).
Publish scoped to local use per explicit user decision: already registered
in oleks-local (1.2c, since Phase 1); deliberately not added to the public
claude-plugin-index or verified on claude-plugins.oleks.space. tasks.md 8.2
records that as a decision, not an oversight, so a later session doesn't
'fix' it by publishing publicly without asking.
8.3 (optional cc-tokens upstream PR) stays open, not required for this
release. 8.4 (memory checkpoint) already satisfied — FR-9 writes on every
run plus this session's manual checkpoints.
decline_reason() only ever read a candidate's own grade, so a candidate
boundary-classify graded high/complete-triple -- meaning it cleared FR-4.2
fine -- fell through to the generic 'did not clear the filing threshold'
line when stability-classify downgraded it afterward. That's the opposite of
what happened, and it directly contradicted the correct FR-4.5 section
rendered a few lines below it in the same judgment-calls drawer, poisoning
the next audit's step-2b recall with a false reason.
Found live: real audit of anxious (2026-07-30) downgraded
agent-wip/release-policy-derivation at 11.85% of audited spend -- the one
gate it DID clear -- and the drawer said it didn't. Filed and now fixed as
kotkan/claude-plugin-inference-arbitrage#16.
Fix keeps FR-4.2's own check order first (a candidate can coincidentally be
named in an unrelated stability finding without having actually cleared
those checks) and only replaces the final fallback, since a candidate that
reaches it cleared every earlier check on its own grade -- it cannot have
also failed the filing threshold on its own merits.
New regression case (h) in tests/memory.test.sh. Suite: all tests pass.
commands/offload-audit.md cited 'spec §7' for the one-agent-per-audit rule;
that constraint now has a real home at FR-8.3 (added alongside the recursion
fix, commit 06cb5b8) and the file was out of spec-review's scope to touch.
tasks.md 8.4 predates FR-9 and still pointed a manual memory checkpoint at
wing_claude_memory/working-practice, while FR-9.3's shipped wing/room split
routes the same kind of cross-target judgment to claude-plugins/
inference-arbitrage-lessons via bin/audit-snapshot memory-notes. Two
destinations for one category of content defeats the point of a single
accumulating room, so 8.4 now points at the one the automated path already
writes to.
Suite: unchanged, exit 0 on all tests.
An independent spec-quality review (spec-review) found real drift and one real
gap, but also overstated FR-9.2 as an unenforced 'discipline' when three of its
four prohibitions are already mechanical (filing-plan only files verdict=='file',
requires --existing so a failed marker search can't silently duplicate, and
stability-classify rewrites rather than re-asks). Only 'what the analyst writes
into the judgments JSON' is genuinely unchecked — narrowed the claim to that.
The review also asserted an obligation — 'a run that leans on a recalled
conclusion must say so in the summary' — that didn't exist in any procedure.
Rather than strip a good idea, implemented it: skills/offload-audit/SKILL.md §7
and agents/offload-analyst.md §5 now require exactly that disclosure.
Same class of error, caught independently: FR-7.3's new stability/tracker-
separation clause describes what the analyst's spoken report can do, not what
the rendered wiki page does — snapshot_candidate carries no stability field, and
findings from stability-classify never enter the snapshot. Documented the gap
and its mechanical cause (widening the snapshot schema, not the renderer) rather
than leaving the spec's claim wrong.
FR-6.5, the FR-9.3 wing/room table, and the Q1/Q3 resolutions the review added
were checked against the shipped code and are accurate; left unchanged.
Suite: 159 assertions, exit 0.
FR-9's read and write steps must not turn a missing memory plugin or an
unreachable palace into a blocked audit. Path B already always runs; path C
degrades the same way, with the failure recorded via --note so the wiki page
shows the run is missing from memory.
An audit that forgets every run makes the analyst re-derive the same judgment
calls from scratch each time, and loses the declined-candidate reasoning that is
the most expensive thing a run produces.
Two new audit-snapshot subcommands, both with `pages`' contract — filesystem in,
JSON out, no MCP call ever made here:
memory-queries what to search before grading
memory-notes the drawers and kg facts to write after the run
Placement: target-scoped findings go to the audited plugin's own wing (room
`inference-arbitrage-audits`), mirroring FR-6.1's "the finding belongs on the
target's surface"; cross-target calibration goes to `claude-plugins` (room
`inference-arbitrage-lessons`) on every run regardless of outcome. Rooms are
namespaced by this plugin's name because plugin-named wings really do collide
with unrelated ones (`cluster`, `imagex`), and a namespaced room makes that a
shelving overlap rather than a content collision.
Reading is advisory and never a gate: nothing recalled may override
boundary-classify (FR-4.2) or stability-classify (FR-4.5), excuse skipping the
marker search, or re-grade a returned verdict.
The memory renderers deliberately do NOT reuse render_run/render_latest.
`snapshot.notes` is caller-supplied free text that legitimately reaches a wiki
page in this plugin's own repo, and must never reach a drawer in a wing shared
with other projects. They read from a fixed allowlist of structured fields —
allowlist, never blocklist, since a blocklist is one new snapshot field away
from leaking.
The MCP calls are not covered by tests/memory.test.sh and are not faked there.
That is the point of putting all the judgment in the script: what remains
untestable is one verbatim tool call with nothing in it to get wrong. The live
round trip was verified by hand against the real anxious snapshot.
commands/offload-audit.md tells a heavy-target caller to hand the work to
Agent(offload-analyst). Nothing told the analyst it IS that delegation target,
so on a large plugin it re-judged the target heavy by the same criteria and
spawned another offload-analyst instance, which did the same again — observed
live against the cluster plugin (11 agents, ~27 skills), 3-4 levels deep, each
restarting the same early measurement steps with no forward progress.
Fixed by making the analyst explicit that its only Agent calls are the fixed
filing chain, and scoping the command's delegation guidance to the top level
only.
Phase 7 filed anxious/agent-wip/release-policy-derivation as `measured` at
11.94% of audited spend on a 30-day window. The same candidate over 7 days came
back `thin` at 0.00%. Only the date range differed. A human reading either issue
body alone cannot know the other exists, which is rubric.md §4a's failure mode
reached through the plugin's own headline number.
That was caught by a human running a second skeptical review — luck, not a
control. It is now mechanical, in the same register as FR-4.2's overrule-case
gate:
- bin/stability-classify matches each `file` candidate against the most recent
prior snapshot recording it (FR-5.2 identity) and downgrades it to a boundary
question if `measurement_strength` changed or the share crossed the 2% filing
threshold between windows. Movement within a label is a trend, not an
instability; no prior window is `unchecked`, not unstable (FR-2.4).
- The downgrade is a finding against THIS repo, not the target's — the target
did not change, the auditor described it two ways. Carries an `ia-stability`
marker so a re-run comments rather than duplicating.
- bin/ia_store.py factors the store and identity resolution out of
audit-snapshot so both scripts hash signatures identically. A divergence there
would make the gate match nothing and fail open.
Documented as spec FR-4.5 and rubric §5b (shipped verbatim in references/).
Wired into skills/offload-audit as step 4b, before filing-plan; steps 5-6 now
consume its output rather than classified.json.
S4 is retired in its old form — a human triage catching a bad candidate is not a
repeatable test — and re-passed as: the gate catches the real
release-policy-derivation case with no human in the loop. Asserted in
tests/stability.test.sh case (f) against the verbatim two-window Phase 7 output.
Closeskotkan/claude-plugin-inference-arbitrage#11
Suite: 178 assertions, exit 0.
S1-S3 and S5-S8 pass; S4 is prepared but deliberately unfiled pending user
sign-off (tasks.md 7.4 stays unchecked).
Two bugs surfaced by running the composed pipeline against real data rather
than synthetic fixtures, both in load-bearing places:
audit-snapshot: cost_per_invocation folded to 0 when a candidate had zero
invocations in a window, so a step that simply did not run reported -100% and
direction() called it `shrunk`. That is the quiet-window artefact FR-5.4 exists
to prevent, one level up from absolute tokens, and it contradicted direction()'s
own docstring. It is now None, metric_delta marks the pair `unmeasured`,
direction returns a new `unmeasured` status, and the pages render "not run".
tests/calibration/mask-worktree-audit.py: recomputed script_loc from scripts
alone, silently dropping all 1051 LOC of worktree-discipline's hooks. Masking
one 272-LOC script appeared to remove 1323, taking script_coverage 0.353 -> 0.083
instead of 0.297, which let S3 pass partly on an artefact of the mask. The mask
now matches build_aggregate. S3 still flags the known cut `high` /
llm-over-script-digest on the corrected, harder input.
Both fixes carry regression tests; suite is 156 assertions, exit 0.
Closes FR-6.4's wiki half — the reason reaches Audits/<target>/<date> instead
of living only in filing-plan's output. Also records the wiki_write
delete-by-sub_url gotcha.
FR-6.4 requires a target with no writable repo to record `filing: unavailable`
with its reason in the output. The filing pass knows that; audit-snapshot does
not, and `notes` was derived purely from no-offload/drift-note verdicts, so the
fact could reach the user's terminal but never the wiki page FR-6.4 points at.
Requested by the Phase 6 output-paths work.
bin/filing-plan decides file/comment/skip per candidate and renders both
bodies from references/issue-template.md. The ia-candidate marker match
(FR-6.3) is a literal string comparison over bodies AND comments, done
locally: verified against live Gitea that neither list_issues nor
search_issues returns a body, so the search is list-by-label + per-issue
issue_read, and --existing is mandatory so a failed search can never
become a duplicate-filing spree.
Repo-less targets degrade to summary-only (FR-6.4), tested for real
against third-party/baoyu-design's github.com remote.
skills/offload-audit ties the procedure together and carries the fixed
delegation chain (issuer -> gitea-agent), the implicit-filing rule, and
the coverage-as-dilution reporting contract.
bin/audit-snapshot write|list|diff, plus two subcommands that carry the wiki
bridge: `pages` emits what to publish and `import` seeds a cold cache from a
wiki-fetched snapshots.jsonl. The script is offline and filesystem-only — it
mirrors the wiki tree into a local store and never opens a socket, so the
plugin still holds no Gitea credentials and the push stays an explicit caller
action. Resolves the tension in plan §5, where "cloned and pushed" and "never
writes Gitea directly" cannot both be true of one process.
Identity resolves issue -> signature hash -> slug (FR-5.2); a hash change under
a stable identity is reported as signature-drifted rather than a silent
disappearance. Diff verdicts come from cost-per-invocation and share-of-spend
with a 10% noise band (FR-5.4) so a quiet window cannot read as progress.
`resolved` is the strict FR-5.5 conjunction — issue closed AND remeasured under
the same 2%/3-invocation bar boundary-classify uses to file; everything else
that merely stopped being reported is claimed-fixed-unconfirmed, naming the
missing leg.
Two additions to the plan §6 schema, both because a requirement is otherwise
uncomputable: target.skills (stale needs to tell "deleted" from "got cheap")
and measurements_by_signature (resolution needs a later window's cost for a
candidate that is gone).
references/boundary-rubric.md is design/rubric.md verbatim (4.1).
skills/boundary-rubric is usable standalone on a single step before any code
exists — the rubric is more valuable applied before the fact than after (4.2).
agents/offload-analyst classifies pre-computed inventory/scan JSON, leads with
coverage, treats hook/prose drift as its own category, and enforces the hard
gate: no overrule case -> boundary question, never filed (4.3).
bin/boundary-classify applies rubric §6's confidence table and the FR-4.2 filing
gate. The determinism-test outcomes and the falsifiability triple are inference;
grading them is a lookup, so it is a script rather than agent prose (FR-8.1).
Calibration gate (4.4) passes on real runs:
- token-budget -> 0 high-confidence, "nothing to offload here"
- worktree-discipline with bin/worktree-audit masked out
-> worktree-safety classification at high,
position llm-over-script-digest
tests/classify.test.sh asserts both gates plus synthetic table cases; 67
assertions green across the suite.
Streams ~/.claude/projects/**/*.jsonl once, filters to turns attributed to the
target plugin (attributionPlugin / attributionSkill / attributionAgent),
de-duplicates by (message.id, requestId) taking max on every usage field, and
computes the spec.md §5.2 dynamic signals: MTR, offload_waste, recurring
tool-signature n-grams, read amplification, retry density, judgment density,
fan-out multiplier, and the composite offload_value — plus the FR-3.4
attribution-coverage headline.
Token and cost arithmetic is taken from token-budget's cc-tokens, never
reimplemented. It is imported as a module rather than shelled out with --json,
because cc-tokens' subcommands aggregate by day/session/project/model and none
can answer "weighted tokens for this set of turns" — the attribution fields are
not on its CLI surface. Its Rec.cost / Rec.weighted still compute every number
here, so the pricing table and cache multipliers keep exactly one home.
Two deviations from design/plan.md §4.1, both because the transcripts say
otherwise: agents attribute via attributionAgent on the subagent sidechain
files, not agentName (which is a separate `type: "agent-name"` line mapping a
session to a display label); and attributionSkill is sometimes bare rather than
plugin-qualified, so it is normalized.
Tool arguments are masked to <path>/<n>/<sha>/<url>/<str> before entering any
emitted structure, and Bash commands are reduced to bare executable words —
FR-3.5, asserted in the suite. Fixtures are synthetic by construction; no real
transcript content is in this repo.
Real-history check: 30-day window over a 1.7 GB history, 94s wall, 29 MB peak
RSS, one process.
Oleks
oleks