An independent spec-quality review (spec-review) found real drift and one real
gap, but also overstated FR-9.2 as an unenforced 'discipline' when three of its
four prohibitions are already mechanical (filing-plan only files verdict=='file',
requires --existing so a failed marker search can't silently duplicate, and
stability-classify rewrites rather than re-asks). Only 'what the analyst writes
into the judgments JSON' is genuinely unchecked — narrowed the claim to that.
The review also asserted an obligation — 'a run that leans on a recalled
conclusion must say so in the summary' — that didn't exist in any procedure.
Rather than strip a good idea, implemented it: skills/offload-audit/SKILL.md §7
and agents/offload-analyst.md §5 now require exactly that disclosure.
Same class of error, caught independently: FR-7.3's new stability/tracker-
separation clause describes what the analyst's spoken report can do, not what
the rendered wiki page does — snapshot_candidate carries no stability field, and
findings from stability-classify never enter the snapshot. Documented the gap
and its mechanical cause (widening the snapshot schema, not the renderer) rather
than leaving the spec's claim wrong.
FR-6.5, the FR-9.3 wing/room table, and the Q1/Q3 resolutions the review added
were checked against the shipped code and are accurate; left unchanged.
Suite: 159 assertions, exit 0.
An audit that forgets every run makes the analyst re-derive the same judgment
calls from scratch each time, and loses the declined-candidate reasoning that is
the most expensive thing a run produces.
Two new audit-snapshot subcommands, both with `pages`' contract — filesystem in,
JSON out, no MCP call ever made here:
memory-queries what to search before grading
memory-notes the drawers and kg facts to write after the run
Placement: target-scoped findings go to the audited plugin's own wing (room
`inference-arbitrage-audits`), mirroring FR-6.1's "the finding belongs on the
target's surface"; cross-target calibration goes to `claude-plugins` (room
`inference-arbitrage-lessons`) on every run regardless of outcome. Rooms are
namespaced by this plugin's name because plugin-named wings really do collide
with unrelated ones (`cluster`, `imagex`), and a namespaced room makes that a
shelving overlap rather than a content collision.
Reading is advisory and never a gate: nothing recalled may override
boundary-classify (FR-4.2) or stability-classify (FR-4.5), excuse skipping the
marker search, or re-grade a returned verdict.
The memory renderers deliberately do NOT reuse render_run/render_latest.
`snapshot.notes` is caller-supplied free text that legitimately reaches a wiki
page in this plugin's own repo, and must never reach a drawer in a wing shared
with other projects. They read from a fixed allowlist of structured fields —
allowlist, never blocklist, since a blocklist is one new snapshot field away
from leaking.
The MCP calls are not covered by tests/memory.test.sh and are not faked there.
That is the point of putting all the judgment in the script: what remains
untestable is one verbatim tool call with nothing in it to get wrong. The live
round trip was verified by hand against the real anxious snapshot.
commands/offload-audit.md tells a heavy-target caller to hand the work to
Agent(offload-analyst). Nothing told the analyst it IS that delegation target,
so on a large plugin it re-judged the target heavy by the same criteria and
spawned another offload-analyst instance, which did the same again — observed
live against the cluster plugin (11 agents, ~27 skills), 3-4 levels deep, each
restarting the same early measurement steps with no forward progress.
Fixed by making the analyst explicit that its only Agent calls are the fixed
filing chain, and scoping the command's delegation guidance to the top level
only.
Phase 7 filed anxious/agent-wip/release-policy-derivation as `measured` at
11.94% of audited spend on a 30-day window. The same candidate over 7 days came
back `thin` at 0.00%. Only the date range differed. A human reading either issue
body alone cannot know the other exists, which is rubric.md §4a's failure mode
reached through the plugin's own headline number.
That was caught by a human running a second skeptical review — luck, not a
control. It is now mechanical, in the same register as FR-4.2's overrule-case
gate:
- bin/stability-classify matches each `file` candidate against the most recent
prior snapshot recording it (FR-5.2 identity) and downgrades it to a boundary
question if `measurement_strength` changed or the share crossed the 2% filing
threshold between windows. Movement within a label is a trend, not an
instability; no prior window is `unchecked`, not unstable (FR-2.4).
- The downgrade is a finding against THIS repo, not the target's — the target
did not change, the auditor described it two ways. Carries an `ia-stability`
marker so a re-run comments rather than duplicating.
- bin/ia_store.py factors the store and identity resolution out of
audit-snapshot so both scripts hash signatures identically. A divergence there
would make the gate match nothing and fail open.
Documented as spec FR-4.5 and rubric §5b (shipped verbatim in references/).
Wired into skills/offload-audit as step 4b, before filing-plan; steps 5-6 now
consume its output rather than classified.json.
S4 is retired in its old form — a human triage catching a bad candidate is not a
repeatable test — and re-passed as: the gate catches the real
release-policy-derivation case with no human in the loop. Asserted in
tests/stability.test.sh case (f) against the verbatim two-window Phase 7 output.
Closeskotkan/claude-plugin-inference-arbitrage#11
Suite: 178 assertions, exit 0.
references/boundary-rubric.md is design/rubric.md verbatim (4.1).
skills/boundary-rubric is usable standalone on a single step before any code
exists — the rubric is more valuable applied before the fact than after (4.2).
agents/offload-analyst classifies pre-computed inventory/scan JSON, leads with
coverage, treats hook/prose drift as its own category, and enforces the hard
gate: no overrule case -> boundary question, never filed (4.3).
bin/boundary-classify applies rubric §6's confidence table and the FR-4.2 filing
gate. The determinism-test outcomes and the falsifiability triple are inference;
grading them is a lookup, so it is a script rather than agent prose (FR-8.1).
Calibration gate (4.4) passes on real runs:
- token-budget -> 0 high-confidence, "nothing to offload here"
- worktree-discipline with bin/worktree-audit masked out
-> worktree-safety classification at high,
position llm-over-script-digest
tests/classify.test.sh asserts both gates plus synthetic table cases; 67
assertions green across the suite.
Oleks