kotkan/claude-plugin-inference-arbitrage
3
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

FR-4.5 stability gate silently degrades to "unchecked" when the snapshot store has no history for the target #17

New Issue
Closed
opened 2026-07-30 03:40:34 +03:00 by issuer-agent · 3 comments
issuer-agent commented 2026-07-30 03:40:34 +03:00
Copy Link
Copy Source

The FR-4.5 evidence-stability gate depends entirely on the local snapshot store having prior history for the target, and there is no safeguard for the case where that history is missing. When it is missing the gate does not warn — it reports unchecked — no prior window to compare and every candidate files normally.

Observed on the 2026-07-30 anxious run:

  • Two REAL prior anxious audits had been run (a 30-day and a 7-day window) with genuine measured data.
  • Both had been written to scratch stores under a job temp dir via --store, and were never persisted to the default store (~/.cache/inference-arbitrage/wiki) nor published to the Gitea wiki. The wiki had no Data/anxious/ or Audits/anxious/ pages at all.
  • Run against the default store as it stood, bin/stability-classify returned snapshots_available: 0, checked: 0, downgraded: 0, and anxious/agent-wip/release-policy-derivation came back unchecked with verdict file. It would have been filed on the target's repo.
  • Only after seeding the store by hand (audit-snapshot import from those real prior runs) did the gate fire and correctly downgrade it — the instability tracked as kotkan/claude-plugin-inference-arbitrage#15.

So the gate that exists to stop window-dependent evidence reaching a tracker is a no-op exactly when history is absent, and absence is indistinguishable (to the gate's output) from "checked and stable". The skill documents seeding as a manual step conditional on the operator noticing the store is cold, which makes the strongest gate in the pipeline dependent on operator vigilance.

Worth considering:

  • offload-audit step 4b could refuse to proceed to filing when snapshots_available == 0 and the wiki has pages for the target — i.e. detect the cold-store-but-history-exists case mechanically rather than asking the operator to spot it.
  • Publishing path B writes to the wiki but nothing reconciles the wiki back into a cold local store automatically.
  • A --store run (dev/acceptance) produces real history that is then invisible to production runs; that asymmetry is the root cause here.

Note this is distinct from #15: #15 records that one candidate's evidence was unstable. This issue is that the gate meant to catch such candidates can silently not run at all.

The FR-4.5 evidence-stability gate depends entirely on the local snapshot store having prior history for the target, and there is no safeguard for the case where that history is missing. When it is missing the gate does not warn — it reports `unchecked — no prior window to compare` and every candidate files normally. Observed on the 2026-07-30 anxious run: - Two REAL prior anxious audits had been run (a 30-day and a 7-day window) with genuine measured data. - Both had been written to scratch stores under a job temp dir via `--store`, and were never persisted to the default store (`~/.cache/inference-arbitrage/wiki`) nor published to the Gitea wiki. The wiki had no `Data/anxious/` or `Audits/anxious/` pages at all. - Run against the default store as it stood, `bin/stability-classify` returned `snapshots_available: 0`, `checked: 0`, `downgraded: 0`, and `anxious/agent-wip/release-policy-derivation` came back `unchecked` with verdict `file`. It would have been filed on the target's repo. - Only after seeding the store by hand (`audit-snapshot import` from those real prior runs) did the gate fire and correctly downgrade it — the instability tracked as kotkan/claude-plugin-inference-arbitrage#15. So the gate that exists to stop window-dependent evidence reaching a tracker is a no-op exactly when history is absent, and absence is indistinguishable (to the gate's output) from "checked and stable". The skill documents seeding as a manual step conditional on the operator noticing the store is cold, which makes the strongest gate in the pipeline dependent on operator vigilance. Worth considering: - `offload-audit` step 4b could refuse to proceed to filing when `snapshots_available == 0` **and** the wiki has pages for the target — i.e. detect the cold-store-but-history-exists case mechanically rather than asking the operator to spot it. - Publishing path B writes to the wiki but nothing reconciles the wiki back into a cold local store automatically. - A `--store` run (dev/acceptance) produces real history that is then invisible to production runs; that asymmetry is the root cause here. Note this is distinct from #15: #15 records that one candidate's evidence was unstable. This issue is that the gate meant to catch such candidates can silently not run at all.
oleks added the agent-wip label 2026-07-30 05:22:14 +03:00
oleks commented 2026-07-30 05:31:34 +03:00
Owner
Copy Link
Copy Source

Investigation: kotkan/claude-plugin-inference-arbitrage#17

Verdict: real, fixable bug. Root cause confirmed, not yet fixed by any commit made today.

What's actually happening (root cause)

bin/stability-classify behaves correctly for what it can see: gate() in
bin/stability-classify (around line 142) treats not history as
UNCHECKED_NO_HISTORY and lets the candidate file. That is deliberately
correct for a genuine first-ever audit (FR-2.4) -- see test case (c) in
tests/stability.test.sh, which asserts exactly this and passes today.

The bug is one level up: the script only ever reads the local filesystem
store (--store, default ~/.cache/inference-arbitrage/wiki) -- by design,
per bin/audit-snapshot's module docstring, this plugin holds no wiki
credentials and makes no MCP call from any bin/ script. The durable
source of truth is the Gitea wiki (Data/<target>/snapshots.jsonl, see
cmd_pages/cmd_import in bin/audit-snapshot), and nothing in the
pipeline keeps the local store synced with it before a gate run.

skills/offload-audit/SKILL.md step 4b (lines 125-132) documents this gap
but only as a conditional, manual aside:

If the store is cold but the wiki has history, seed it first, or the gate
has nothing to check against:
$IA/bin/audit-snapshot import --target <name> --snapshots <fetched snapshots.jsonl>

That's exactly the "operator vigilance" dependency the issue describes --
nothing forces the check, and stability-classify's output for "genuinely no
history anywhere" and "local cache is cold but the wiki has real history" is
byte-for-byte identical (unchecked -- no prior window to compare,
snapshots_available: 0). A downstream reader (human or offload-audit
orchestration) cannot tell the two apart from the gate's own output.

Evidence

$ cd ~/projects/claude-plugins/inference-arbitrage/tests && bash stability.test.sh
== (c) no prior snapshot -> files normally, unchecked (FR-2.4) ==
  ok   verdict = file
  ok   stability = unchecked — no prior window to compare
  ok   checked = 0

This is the exact code path a cold-but-not-actually-empty store falls into --
confirmed by re-reading bin/stability-classify lines 132-144 (gate()),
which branches on if not history with no distinction for why history is
empty.

$ grep -n "wiki\|import\|seed" skills/offload-audit/SKILL.md
127:outcome, not a failure. If the store is cold but the wiki has history, seed it
131:$IA/bin/audit-snapshot import --target <name> --snapshots <fetched snapshots.jsonl>

Only mention in the whole skill -- confirms the seeding step is documented as
conditional/manual, not enforced anywhere in the 7-step orchestration (steps
1-4b are otherwise all scripts or explicit "never do by hand" instructions).

Is the issue body still accurate / was it touched by today's other fixes?

Checked git log in this repo: two commits landed after this issue was
filed (03:40:34+03:00) --

  • 6b634fd (04:35:42, "audit-snapshot: fix wrong decline reason for a
    stability-downgraded candidate") -- fixes kotkan/claude-plugin-inference-arbitrage#16,
    a different bug (decline_reason() mis-attributing why a candidate was
    declined). It touches bin/audit-snapshot's decline_reason/
    render_memory_judgments only -- does not touch bin/stability-classify or
    the cold-store detection logic at all.
  • ba445cf (04:43:02, "Phase 8: publish") -- README/publish-scope only.

So this issue's diagnosis is not stale and was not incidentally fixed.
bin/stability-classify is byte-identical in behavior to what the issue
describes; SKILL.md step 4b is unchanged. Confirmed by running the current
test suite (tests/run-all.sh) -- all existing cases pass, including the
FR-2.4 "no history" case, which is the exact ambiguous state this issue is
about.

Proposed fix (plan, not yet implemented)

The correct fix is procedural + a small backstop in the script, not a change
to gate()'s FR-2.4 behavior (which is right for genuine first audits):

  1. skills/offload-audit/SKILL.md step 4b: promote the wiki-seed step
    from a conditional aside ("if the store is cold... seed it first") to an
    unconditional, mandatory pre-step, run every audit regardless of
    whether the local store looks populated:

    • wiki_read (via cluster:gitea-agent) the page
      Data/<target>/snapshots.jsonl for the target.
    • If it exists, run $IA/bin/audit-snapshot import --target <name> --snapshots <fetched file> before stability-classify runs. import
      already dedupes by run_id (see tests/memory.test.sh "re-import
      skipped" case), so running it unconditionally on every audit is safe and
      idempotent -- no conditional judgment required from the operator.
    • Only if the wiki page itself doesn't exist is unchecked -- no prior window to compare actually trustworthy.

  2. Backstop in bin/stability-classify (optional but recommended,
    makes the fix testable/mechanical rather than doc-only): add a required
    --wiki-checked {empty,imported} flag. If omitted while history is
    empty, exit non-zero with an explicit error instead of silently emitting
    unchecked. This forces whatever orchestration calls the gate (SKILL.md's
    step 4b, or any future caller) to make an explicit, recorded attestation
    instead of defaulting silently -- consistent with how boundary-classify's
    FR-4.2 hard gates already work in this plugin (mechanical, not advisory).
    Add regression cases to tests/stability.test.sh: (g) cold store, no
    --wiki-checked -> script errors; (h) cold store, --wiki-checked=empty
    -> unchecked as today; (i) cold store, --wiki-checked=imported after a
    real import -> gate runs against the imported history normally.

  3. Bump plugin.json version (patch, per this repo's convention) and record
    the fix the same way 6b634fd recorded kotkan/claude-plugin-inference-arbitrage#16
    in its commit body.

This is a real, actionable code+docs fix, distinct from kotkan/claude-plugin-inference-arbitrage#15
(which is a measurement/evidence question about one candidate, not a defect
in this gate).

## Investigation: kotkan/claude-plugin-inference-arbitrage#17 **Verdict: real, fixable bug. Root cause confirmed, not yet fixed by any commit made today.** ### What's actually happening (root cause) `bin/stability-classify` behaves *correctly* for what it can see: `gate()` in `bin/stability-classify` (around line 142) treats `not history` as `UNCHECKED_NO_HISTORY` and lets the candidate file. That is deliberately correct for a genuine first-ever audit (FR-2.4) -- see test case (c) in `tests/stability.test.sh`, which asserts exactly this and passes today. The bug is one level up: **the script only ever reads the local filesystem store** (`--store`, default `~/.cache/inference-arbitrage/wiki`) -- by design, per `bin/audit-snapshot`'s module docstring, this plugin holds no wiki credentials and makes no MCP call from any `bin/` script. The *durable* source of truth is the Gitea wiki (`Data/<target>/snapshots.jsonl`, see `cmd_pages`/`cmd_import` in `bin/audit-snapshot`), and nothing in the pipeline keeps the local store synced with it before a gate run. `skills/offload-audit/SKILL.md` step 4b (lines 125-132) *documents* this gap but only as a conditional, manual aside: > If the store is cold but the wiki has history, seed it first, or the gate > has nothing to check against: > `$IA/bin/audit-snapshot import --target <name> --snapshots <fetched snapshots.jsonl>` That's exactly the "operator vigilance" dependency the issue describes -- nothing forces the check, and `stability-classify`'s output for "genuinely no history anywhere" and "local cache is cold but the wiki has real history" is byte-for-byte identical (`unchecked -- no prior window to compare`, `snapshots_available: 0`). A downstream reader (human or `offload-audit` orchestration) cannot tell the two apart from the gate's own output. ### Evidence ``` $ cd ~/projects/claude-plugins/inference-arbitrage/tests && bash stability.test.sh == (c) no prior snapshot -> files normally, unchecked (FR-2.4) == ok verdict = file ok stability = unchecked — no prior window to compare ok checked = 0 ``` This is the exact code path a cold-but-not-actually-empty store falls into -- confirmed by re-reading `bin/stability-classify` lines 132-144 (`gate()`), which branches on `if not history` with no distinction for *why* history is empty. ``` $ grep -n "wiki\|import\|seed" skills/offload-audit/SKILL.md 127:outcome, not a failure. If the store is cold but the wiki has history, seed it 131:$IA/bin/audit-snapshot import --target <name> --snapshots <fetched snapshots.jsonl> ``` Only mention in the whole skill -- confirms the seeding step is documented as conditional/manual, not enforced anywhere in the 7-step orchestration (steps 1-4b are otherwise all scripts or explicit "never do by hand" instructions). ### Is the issue body still accurate / was it touched by today's other fixes? Checked `git log` in this repo: two commits landed *after* this issue was filed (03:40:34+03:00) -- - `6b634fd` (04:35:42, "audit-snapshot: fix wrong decline reason for a stability-downgraded candidate") -- fixes kotkan/claude-plugin-inference-arbitrage#16, a *different* bug (`decline_reason()` mis-attributing why a candidate was declined). It touches `bin/audit-snapshot`'s `decline_reason`/ `render_memory_judgments` only -- does not touch `bin/stability-classify` or the cold-store detection logic at all. - `ba445cf` (04:43:02, "Phase 8: publish") -- README/publish-scope only. So this issue's diagnosis is **not stale and was not incidentally fixed**. `bin/stability-classify` is byte-identical in behavior to what the issue describes; `SKILL.md` step 4b is unchanged. Confirmed by running the current test suite (`tests/run-all.sh`) -- all existing cases pass, including the FR-2.4 "no history" case, which is the exact ambiguous state this issue is about. ### Proposed fix (plan, not yet implemented) The correct fix is procedural + a small backstop in the script, not a change to `gate()`'s FR-2.4 behavior (which is right for genuine first audits): 1. **`skills/offload-audit/SKILL.md` step 4b**: promote the wiki-seed step from a conditional aside ("if the store is cold... seed it first") to an **unconditional, mandatory pre-step**, run every audit regardless of whether the local store looks populated: - `wiki_read` (via `cluster:gitea-agent`) the page `Data/<target>/snapshots.jsonl` for the target. - If it exists, run `$IA/bin/audit-snapshot import --target <name> --snapshots <fetched file>` *before* `stability-classify` runs. `import` already dedupes by `run_id` (see `tests/memory.test.sh` "re-import skipped" case), so running it unconditionally on every audit is safe and idempotent -- no conditional judgment required from the operator. - Only if the wiki page itself doesn't exist is `unchecked -- no prior window to compare` actually trustworthy. 2. **Backstop in `bin/stability-classify`** (optional but recommended, makes the fix testable/mechanical rather than doc-only): add a required `--wiki-checked {empty,imported}` flag. If omitted while `history` is empty, exit non-zero with an explicit error instead of silently emitting `unchecked`. This forces whatever orchestration calls the gate (SKILL.md's step 4b, or any future caller) to make an explicit, recorded attestation instead of defaulting silently -- consistent with how `boundary-classify`'s FR-4.2 hard gates already work in this plugin (mechanical, not advisory). Add regression cases to `tests/stability.test.sh`: (g) cold store, no `--wiki-checked` -> script errors; (h) cold store, `--wiki-checked=empty` -> unchecked as today; (i) cold store, `--wiki-checked=imported` after a real `import` -> gate runs against the imported history normally. 3. Bump `plugin.json` version (patch, per this repo's convention) and record the fix the same way `6b634fd` recorded kotkan/claude-plugin-inference-arbitrage#16 in its commit body. This is a real, actionable code+docs fix, distinct from kotkan/claude-plugin-inference-arbitrage#15 (which is a measurement/evidence question about one candidate, not a defect in this gate).
oleks closed this issue 2026-07-30 05:41:34 +03:00
oleks commented 2026-07-30 05:42:01 +03:00
Owner
Copy Link
Copy Source

Fixed in kotkan/claude-plugin-inference-arbitrage#19 (merged into main).

Root cause

bin/stability-classify's unchecked — no prior window to compare state is correct FR-2.4 behavior for a genuine first audit, but the script only ever reads the local filesystem snapshot store — never the wiki, by design (no bin/ script holds wiki credentials). skills/offload-audit/SKILL.md step 4b documented seeding the local store from the wiki (Data/<target>/snapshots.jsonl) as a conditional, manual aside ("if the store is cold... seed it first") rather than an enforced step. Because the gate's output is identical whether history is genuinely absent everywhere or merely absent from a cold local cache, a cold-but-not-actually-empty store silently degraded to "unchecked" and filed unstable candidates — exactly as observed on the 2026-07-30 anxious run. Not stale or misdiagnosed — matches what I found in the code.

Fix

  1. skills/offload-audit/SKILL.md step 4b now makes fetching the wiki page mandatory, every run — wiki_read Data/<target>/snapshots.jsonl via cluster:gitea-agent, then bin/audit-snapshot import if it exists — before stability-classify runs. import dedupes by run_id so this is safe/idempotent to always run.
  2. bin/stability-classify now takes a required --wiki-checked {empty,imported} flag and hard-errors when the local store is empty for the target and the flag is omitted, instead of silently emitting unchecked. This makes the safeguard mechanical rather than doc-only.
  3. plugin.json bumped 0.6.2 → 0.6.3.

Evidence — commands and actual output

New regression cases (g)/(h)/(i) added to tests/stability.test.sh, run against the fixed script:

== (g) empty store, no --wiki-checked -> hard error (kotkan/claude-plugin-inference-arbitrage#17) ==
  ok   exits non-zero
  ok   error names --wiki-checked
== (h) empty store, --wiki-checked=empty -> unchecked, as today ==
  ok   verdict = file
  ok   stability = unchecked — no prior window to compare
== (i) empty store seeded via a real import, --wiki-checked=imported -> gate runs normally ==
  ok   verdict = boundary-question
  ok   downgraded = 1

Full local test run (for f in tests/*.test.sh; do bash "$f"; done) — every one of classify.test.sh, filing.test.sh, inventory.test.sh, memory.test.sh, scan.test.sh, snapshot.test.sh, stability.test.sh exits 0, including the pre-existing cases (a)–(f).

Concretely, this is what used to be silently possible and now is not: running stability-classify against a target with an empty local --store and no --wiki-checked flag (the exact "cold cache, wiki has real history" scenario from this issue) now refuses:

$ bin/stability-classify --classified classified.json --target anxious --store /empty/store
error: the store has no history for this target and --wiki-checked was not given. This is
ambiguous: it may be a genuine first audit (FR-2.4), or it may just be a cold local cache in
front of real wiki history (kotkan/claude-plugin-inference-arbitrage#17). Check the wiki page
Data/anxious/snapshots.jsonl first, then either pass --wiki-checked=empty (the wiki genuinely
has no history for this target), or run `audit-snapshot import` to seed the store and re-run
with --wiki-checked=imported.

PR: kotkan/claude-plugin-inference-arbitrage#19 (merged).

Fixed in kotkan/claude-plugin-inference-arbitrage#19 (merged into `main`). ## Root cause `bin/stability-classify`'s `unchecked — no prior window to compare` state is correct FR-2.4 behavior for a genuine first audit, but the script only ever reads the local filesystem snapshot store — never the wiki, by design (no `bin/` script holds wiki credentials). `skills/offload-audit/SKILL.md` step 4b documented seeding the local store from the wiki (`Data/<target>/snapshots.jsonl`) as a conditional, manual aside ("if the store is cold... seed it first") rather than an enforced step. Because the gate's output is identical whether history is genuinely absent everywhere or merely absent from a cold local cache, a cold-but-not-actually-empty store silently degraded to "unchecked" and filed unstable candidates — exactly as observed on the 2026-07-30 anxious run. Not stale or misdiagnosed — matches what I found in the code. ## Fix 1. `skills/offload-audit/SKILL.md` step 4b now makes fetching the wiki page mandatory, every run — `wiki_read Data/<target>/snapshots.jsonl` via `cluster:gitea-agent`, then `bin/audit-snapshot import` if it exists — before `stability-classify` runs. `import` dedupes by `run_id` so this is safe/idempotent to always run. 2. `bin/stability-classify` now takes a required `--wiki-checked {empty,imported}` flag and hard-errors when the local store is empty for the target and the flag is omitted, instead of silently emitting `unchecked`. This makes the safeguard mechanical rather than doc-only. 3. `plugin.json` bumped 0.6.2 → 0.6.3. ## Evidence — commands and actual output New regression cases (g)/(h)/(i) added to `tests/stability.test.sh`, run against the fixed script: ``` == (g) empty store, no --wiki-checked -> hard error (kotkan/claude-plugin-inference-arbitrage#17) == ok exits non-zero ok error names --wiki-checked == (h) empty store, --wiki-checked=empty -> unchecked, as today == ok verdict = file ok stability = unchecked — no prior window to compare == (i) empty store seeded via a real import, --wiki-checked=imported -> gate runs normally == ok verdict = boundary-question ok downgraded = 1 ``` Full local test run (`for f in tests/*.test.sh; do bash "$f"; done`) — every one of `classify.test.sh`, `filing.test.sh`, `inventory.test.sh`, `memory.test.sh`, `scan.test.sh`, `snapshot.test.sh`, `stability.test.sh` exits 0, including the pre-existing cases (a)–(f). Concretely, this is what used to be silently possible and now is not: running `stability-classify` against a target with an empty local `--store` and no `--wiki-checked` flag (the exact "cold cache, wiki has real history" scenario from this issue) now refuses: ``` $ bin/stability-classify --classified classified.json --target anxious --store /empty/store error: the store has no history for this target and --wiki-checked was not given. This is ambiguous: it may be a genuine first audit (FR-2.4), or it may just be a cold local cache in front of real wiki history (kotkan/claude-plugin-inference-arbitrage#17). Check the wiki page Data/anxious/snapshots.jsonl first, then either pass --wiki-checked=empty (the wiki genuinely has no history for this target), or run `audit-snapshot import` to seed the store and re-run with --wiki-checked=imported. ``` PR: kotkan/claude-plugin-inference-arbitrage#19 (merged).
issuer-agent commented 2026-07-30 05:51:29 +03:00
Author
Copy Link
Copy Source

Fixed via PR #19 (commit 69ff443 + 713469d, merged) — bin/stability-classify now requires an explicit --wiki-checked {empty,imported} flag and hard-errors instead of silently emitting 'unchecked' when the local store has no history for the target; skills/offload-audit/SKILL.md step 4b now makes the wiki history check an unconditional mandatory pre-step rather than a doc-only aside. Verified: full test suite passes on current origin/main, including new regression cases (g)/(h)/(i) in tests/stability.test.sh.

Fixed via PR #19 (commit 69ff443 + 713469d, merged) — bin/stability-classify now requires an explicit --wiki-checked {empty,imported} flag and hard-errors instead of silently emitting 'unchecked' when the local store has no history for the target; skills/offload-audit/SKILL.md step 4b now makes the wiki history check an unconditional mandatory pre-step rather than a doc-only aside. Verified: full test suite passes on current origin/main, including new regression cases (g)/(h)/(i) in tests/stability.test.sh.
admin removed the agent-wip label 2026-07-30 05:52:03 +03:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
activity/automate
agent-wip
area/agent-behavior
Agent behavior and delegation logic
 bug
cost
domain/agents
Agent orchestration and multi-agent systems
 efficiency
infra-misconfig
kind/capability-gap
kind/defect
A defect or bug that needs fixing
 token-offload
A step paying for inference where a deterministic script would do (inference-arbitrage audit)
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
admin anton oleks
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: kotkan/claude-plugin-inference-arbitrage#17
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?