kotkan/claude-plugin-inference-arbitrage
3
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

Stability audit: anxious/agent-wip/release-policy-derivation measurement unstable across windows #15

New Issue
Open
opened 2026-07-30 03:24:57 +03:00 by issuer-agent · 5 comments
issuer-agent commented 2026-07-30 03:24:57 +03:00
Copy Link
Copy Source

Headline: anxious/agent-wip/release-policy-derivation is thin at 0.00% of spend in run 2026-07-29T01-00-00Z and measured at 11.85% in this window — the confidence label depends on the measurement window, not on the candidate.

Gate: FR-4.5, matched by slug

Candidate: anxious/agent-wip/release-policy-derivation (skill anxious:agent-wip), target anxious

Prior window (run 2026-07-29T01-00-00Z):

  • Date range: 2026-07-22T15:04:13 -> 2026-07-29T15:04:20
  • Sessions: 69
  • Measurement strength: "thin"
  • Share of audited spend: 0.00%

Current window (run of 2026-07-30):

  • Date range: 2026-07-03T12:23:08 -> 2026-07-30T00:11:57
  • Sessions: 170
  • Measurement strength: "measured"
  • Share of audited spend: 11.85% (offload_value 3,900,668 weighted tokens over 12 invocations)

Finding 1 — Measurement strength flip: The evidence label changed from "thin" to "measured" depending only on which measurement window was used. The same candidate would be described differently to a human depending only on when the audit ran.

Finding 2 — Threshold crossing: Share of audited spend moved from 0.0% -> 0.1185% against a 0.02 filing threshold, so the windows land on opposite sides of the 2% filing threshold. The value claim is a window artifact.

Finding 3 — Consequence this run: The candidate was downgraded from file to boundary-question and was NOT filed on the target's repo.

Finding 4 — New this run — snapshot persistence gap: The downgrade was only detectable after the local snapshot store was seeded by hand. The two prior real anxious runs had been written to scratch stores under a job tmp dir and were never persisted to the default store (~/.cache/inference-arbitrage/wiki) nor published to the Gitea wiki. On a clean run the gate would report "unchecked — no prior window to compare" and would file the unstable candidate without the safeguard.

<!-- ia-stability: anxious/agent-wip/release-policy-derivation --> **Headline**: anxious/agent-wip/release-policy-derivation is thin at 0.00% of spend in run 2026-07-29T01-00-00Z and measured at 11.85% in this window — the confidence label depends on the measurement window, not on the candidate. **Gate**: FR-4.5, matched by slug **Candidate**: anxious/agent-wip/release-policy-derivation (skill anxious:agent-wip), target anxious **Prior window** (run 2026-07-29T01-00-00Z): - Date range: 2026-07-22T15:04:13 -> 2026-07-29T15:04:20 - Sessions: 69 - Measurement strength: "thin" - Share of audited spend: 0.00% **Current window** (run of 2026-07-30): - Date range: 2026-07-03T12:23:08 -> 2026-07-30T00:11:57 - Sessions: 170 - Measurement strength: "measured" - Share of audited spend: 11.85% (offload_value 3,900,668 weighted tokens over 12 invocations) **Finding 1 — Measurement strength flip**: The evidence label changed from "thin" to "measured" depending only on which measurement window was used. The same candidate would be described differently to a human depending only on when the audit ran. **Finding 2 — Threshold crossing**: Share of audited spend moved from 0.0% -> 0.1185% against a 0.02 filing threshold, so the windows land on opposite sides of the 2% filing threshold. The value claim is a window artifact. **Finding 3 — Consequence this run**: The candidate was downgraded from `file` to `boundary-question` and was NOT filed on the target's repo. **Finding 4 — New this run — snapshot persistence gap**: The downgrade was only detectable after the local snapshot store was seeded by hand. The two prior real anxious runs had been written to scratch stores under a job tmp dir and were never persisted to the default store (~/.cache/inference-arbitrage/wiki) nor published to the Gitea wiki. On a clean run the gate would report "unchecked — no prior window to compare" and would file the unstable candidate without the safeguard.
oleks commented 2026-07-30 03:31:20 +03:00
Owner
Copy Link
Copy Source

Cross-reference: kotkan/claude-plugin-inference-arbitrage#11 describes this same defect, found by hand during Phase 7 acceptance and now closed. It predates the <!-- ia-stability: ... --> marker mechanism and therefore carries no marker, which is why the automated marker search on this run did not find it and correctly opened this issue as the first marker-bearing report rather than commenting there.

This issue is now the idempotency anchor for the candidate anxious/agent-wip/release-policy-derivation: subsequent audits that re-encounter the same instability will find this marker and comment here instead of filing again.

Cross-reference: kotkan/claude-plugin-inference-arbitrage#11 describes this same defect, found by hand during Phase 7 acceptance and now closed. It predates the `<!-- ia-stability: ... -->` marker mechanism and therefore carries no marker, which is why the automated marker search on this run did not find it and correctly opened this issue as the first marker-bearing report rather than commenting there. This issue is now the idempotency anchor for the candidate `anxious/agent-wip/release-policy-derivation`: subsequent audits that re-encounter the same instability will find this marker and comment here instead of filing again.
oleks added the agent-wip label 2026-07-30 05:22:13 +03:00
oleks commented 2026-07-30 05:35:16 +03:00
Owner
Copy Link
Copy Source

Investigation: not a code defect — a real third measurement window resolves the question

Repo state checked first: git status on main in ~/projects/claude-plugins/inference-arbitrage was clean and in sync with origin/main (HEAD ba445cf, "Phase 8: publish"). git log shows today's other fixes (6b634fd fixing kotkan/claude-plugin-inference-arbitrage#16, plus spec-review/memory-wiring/publish commits) did not touch this issue — #16 fixed a rendering bug (decline_reason() in bin/audit-snapshot blamed the wrong gate for this same candidate's downgrade) but left the underlying stability question — is the 11.85% reading real or a window artifact? — completely open. This issue's body is still accurate as filed; nothing has changed since.

What I did: per the issue's own suggestion, ran the missing third and fourth measurement windows against real ~/.claude/projects transcripts, using the actual bin/offload-scan script (never counted anything by hand):

IA=~/projects/claude-plugins/inference-arbitrage
$IA/bin/offload-scan --plugin anxious --days 14 --json > scan14.json
$IA/bin/offload-scan --plugin anxious --days 60 --json > scan60.json
$IA/bin/offload-scan --plugin anxious --since 2026-07-03 --until 2026-07-30 --json > scan_repro.json   # reproduces the ~27-day "measured" window

Then read the anxious:agent-wip skill row's offload_value (the field boundary-classify actually grades on — offload_value = int(offload_waste * (1 - judgment_density) * repetition_factor), a skill-level composite, not a single ngram's raw waste — verified this is the field by recomputing it from the row's own offload_waste/judgment_density/repetition_factor and matching the stored snapshot's offload_value exactly: 1,332,195 * (1-0.024) * 3.0 = 3,900,668, bit-for-bit the number in the persisted snapshot).

Real results, four real windows, same script, same candidate:

window since→until sessions invocations offload_value share of spend verdict
7-day (issue's "prior") 07-22→07-29 69 0 0 0.00% thin
14-day (new) 07-16→07-30 131 2 110,235 0.72% thin
~27-day (issue's "current", reproduced today) 07-03→07-30 173 12 3,900,668 11.73% measured
60-day (new) 06-26→07-30 176 15 4,953,520 13.87% measured

Root cause (not the issue title): this is not "the confidence label depends on the measurement window" in the sense of noisy/random flipping — it's a monotonic accumulation problem for a low-frequency-but-real candidate. Invocations climb steadily with window width (0 → 2 → 12 → 15) and share climbs with them (0% → 0.72% → 11.73% → 13.87%). A 7-day or 14-day window is simply too short to catch this skill's roughly one-invocation-per-2-days rate; nothing about the candidate's economics changes, only how much of its sparse usage a short window happens to sample. Critically, the new independent 60-day window does not contradict the 27-day "measured" reading — it strengthens it (13.87% > 11.73%, 15 invocations ⊇ the same 12). If this were a genuine artifact, the 60-day pass had every opportunity to dilute the share back down (it adds 33 more days and 3 more sessions of denominator) and instead it went up.

Conclusion: the evidence stabilizes on the "measured, above the 2% filing threshold" side once the window is ≥~27 days. The thin/0% reading that triggered this issue's FR-4.5 downgrade was itself the artifact — a too-short comparison window, not noise in the candidate. needs_code_change: false. No script bug found in offload-scan, boundary-classify, or stability-classify — the gate did exactly what FR-4.5 says it should when it saw the 7-day snapshot disagree with the 27-day one; it just didn't have a wider, corroborating window available at the time to weigh against the thin one.

What I did not do: I did not write these new windows into ~/.cache/inference-arbitrage/wiki (the store the FR-4.5 gate reads) — that store already carries a note (this issue's Finding 4) that it was seeded by hand rather than through the normal publish path, and I didn't want to compound that with another manually-inserted entry outside the pipeline's own boundary-classify → stability-classify → audit-snapshot write sequence. The right way to make this durable is a full, real audit run (see below), not a partial snapshot of just the offload-scan numbers.

Recommended next step (for a human or the next audit run, not a code change)

Run a real, full offload-audit pass on anxious end-to-end (per skills/offload-audit/SKILL.md) using a ≥30-day window (30 or 60 both work; 60 is what I validated above), through the actual pipeline so it persists to the real store and re-triggers FR-4.5 against history honestly:

IA=~/projects/claude-plugins/inference-arbitrage
$IA/bin/plugin-inventory anxious --json > inv.json
$IA/bin/offload-scan --plugin anxious --days 60 --json > scan.json
# step 4: boundary-rubric judgment per candidate -> judgments.json (unchanged from prior runs
# for this candidate — position llm-over-script-digest, all 5 determinism tests, complete triple)
$IA/bin/boundary-classify judgments.json > classified.json
$IA/bin/stability-classify --classified classified.json --target anxious --scan scan.json > stable.json
$IA/bin/audit-snapshot --store ~/.cache/inference-arbitrage/wiki write --target anxious \
    --from inv.json scan.json --classified stable.json --run-id <new-run-id>

If stability-classify sees this new ≥60-day snapshot land on the "measured" side against the most recent prior "measured" snapshot (07-29T00-00-00Z or 07-30T00:31:45Z, both already measured at ~11.9%), the candidate is stable and clears FR-4.5 — it should file normally against oleks/anxious (or whichever repo the target maps to) on the next real run. Only the 7-day/14-day windows were ever the outliers.

Separately, worth a human policy call (not something I'm deciding here): should FR-4.5 treat a thin snapshot built from a window under some minimum width (e.g. 21–30 days) as too weak to trigger a downgrade on its own, given a genuinely low-frequency candidate can look thin in any short window without being unstable? That's a rubric/threshold question, not a bug in the code as written — flagging it for a human, not acting on it.

No new code-level bug surfaced during this investigation (kotkan/claude-plugin-inference-arbitrage#16 already covers the one that did, filed and fixed earlier today, before I started).

## Investigation: not a code defect — a real third measurement window resolves the question **Repo state checked first**: `git status` on `main` in `~/projects/claude-plugins/inference-arbitrage` was clean and in sync with `origin/main` (HEAD `ba445cf`, "Phase 8: publish"). `git log` shows today's other fixes (`6b634fd` fixing kotkan/claude-plugin-inference-arbitrage#16, plus spec-review/memory-wiring/publish commits) did **not** touch this issue — #16 fixed a *rendering* bug (`decline_reason()` in `bin/audit-snapshot` blamed the wrong gate for this same candidate's downgrade) but left the underlying stability question — is the 11.85% reading real or a window artifact? — completely open. This issue's body is still accurate as filed; nothing has changed since. **What I did**: per the issue's own suggestion, ran the missing third and fourth measurement windows against real `~/.claude/projects` transcripts, using the actual `bin/offload-scan` script (never counted anything by hand): ``` IA=~/projects/claude-plugins/inference-arbitrage $IA/bin/offload-scan --plugin anxious --days 14 --json > scan14.json $IA/bin/offload-scan --plugin anxious --days 60 --json > scan60.json $IA/bin/offload-scan --plugin anxious --since 2026-07-03 --until 2026-07-30 --json > scan_repro.json # reproduces the ~27-day "measured" window ``` Then read the `anxious:agent-wip` skill row's `offload_value` (the field `boundary-classify` actually grades on — `offload_value = int(offload_waste * (1 - judgment_density) * repetition_factor)`, a skill-level composite, not a single ngram's raw waste — verified this is the field by recomputing it from the row's own `offload_waste`/`judgment_density`/`repetition_factor` and matching the stored snapshot's `offload_value` exactly: `1,332,195 * (1-0.024) * 3.0 = 3,900,668`, bit-for-bit the number in the persisted snapshot). **Real results, four real windows, same script, same candidate**: | window | since→until | sessions | invocations | offload_value | share of spend | verdict | |---|---|---|---|---|---|---| | 7-day (issue's "prior") | 07-22→07-29 | 69 | 0 | 0 | 0.00% | thin | | 14-day (new) | 07-16→07-30 | 131 | 2 | 110,235 | 0.72% | thin | | ~27-day (issue's "current", reproduced today) | 07-03→07-30 | 173 | 12 | 3,900,668 | 11.73% | measured | | 60-day (new) | 06-26→07-30 | 176 | 15 | 4,953,520 | 13.87% | measured | **Root cause (not the issue title)**: this is not "the confidence label depends on the measurement window" in the sense of noisy/random flipping — it's a **monotonic accumulation problem for a low-frequency-but-real candidate**. Invocations climb steadily with window width (0 → 2 → 12 → 15) and share climbs with them (0% → 0.72% → 11.73% → 13.87%). A 7-day or 14-day window is simply too short to catch this skill's roughly one-invocation-per-2-days rate; nothing about the candidate's economics changes, only how much of its sparse usage a short window happens to sample. Critically, the new independent 60-day window does **not** contradict the 27-day "measured" reading — it *strengthens* it (13.87% > 11.73%, 15 invocations ⊇ the same 12). If this were a genuine artifact, the 60-day pass had every opportunity to dilute the share back down (it adds 33 more days and 3 more sessions of denominator) and instead it went up. **Conclusion**: the evidence stabilizes on the "measured, above the 2% filing threshold" side once the window is ≥~27 days. The `thin`/0% reading that triggered this issue's FR-4.5 downgrade was itself the artifact — a too-short comparison window, not noise in the candidate. `needs_code_change: false`. No script bug found in `offload-scan`, `boundary-classify`, or `stability-classify` — the gate did exactly what FR-4.5 says it should when it saw the 7-day snapshot disagree with the 27-day one; it just didn't have a wider, corroborating window available at the time to weigh against the thin one. **What I did not do**: I did not write these new windows into `~/.cache/inference-arbitrage/wiki` (the store the FR-4.5 gate reads) — that store already carries a note (this issue's Finding 4) that it was seeded by hand rather than through the normal publish path, and I didn't want to compound that with another manually-inserted entry outside the pipeline's own `boundary-classify` → `stability-classify` → `audit-snapshot write` sequence. The right way to make this durable is a full, real audit run (see below), not a partial snapshot of just the `offload-scan` numbers. ## Recommended next step (for a human or the next audit run, not a code change) Run a real, full `offload-audit` pass on `anxious` end-to-end (per `skills/offload-audit/SKILL.md`) using a ≥30-day window (30 or 60 both work; 60 is what I validated above), through the *actual* pipeline so it persists to the real store and re-triggers FR-4.5 against history honestly: ``` IA=~/projects/claude-plugins/inference-arbitrage $IA/bin/plugin-inventory anxious --json > inv.json $IA/bin/offload-scan --plugin anxious --days 60 --json > scan.json # step 4: boundary-rubric judgment per candidate -> judgments.json (unchanged from prior runs # for this candidate — position llm-over-script-digest, all 5 determinism tests, complete triple) $IA/bin/boundary-classify judgments.json > classified.json $IA/bin/stability-classify --classified classified.json --target anxious --scan scan.json > stable.json $IA/bin/audit-snapshot --store ~/.cache/inference-arbitrage/wiki write --target anxious \ --from inv.json scan.json --classified stable.json --run-id <new-run-id> ``` If `stability-classify` sees this new ≥60-day snapshot land on the "measured" side against the most recent prior "measured" snapshot (07-29T00-00-00Z or 07-30T00:31:45Z, both already `measured` at ~11.9%), the candidate is `stable` and clears FR-4.5 — it should file normally against `oleks/anxious` (or whichever repo the target maps to) on the *next* real run. Only the 7-day/14-day windows were ever the outliers. Separately, worth a human policy call (not something I'm deciding here): should FR-4.5 treat a `thin` snapshot built from a window under some minimum width (e.g. 21–30 days) as too weak to trigger a downgrade on its own, given a genuinely low-frequency candidate can look `thin` in *any* short window without being unstable? That's a rubric/threshold question, not a bug in the code as written — flagging it for a human, not acting on it. No new code-level bug surfaced during this investigation (kotkan/claude-plugin-inference-arbitrage#16 already covers the one that did, filed and fixed earlier today, before I started).
oleks commented 2026-07-30 05:35:56 +03:00
Owner
Copy Link
Copy Source

This is not a code-defect issue — it needs a third real measurement window, not a fix

Cross-referenced while investigating kotkan/claude-plugin-inference-arbitrage#12 (a different, secondary finding split from #11). This issue (#15) is the FR-4.5 stability-gate output itself: the gate did its job — it caught that anxious/agent-wip/release-policy-derivation flips between "thin, 0.00%" (prior window, 69 sessions, 2026-07-22→07-29) and "measured, 11.85%" (current window, 170 sessions, 2026-07-03→07-30), straddling the 2% filing threshold, and correctly downgraded the candidate from file to boundary-question instead of filing it. That is the gate working as designed (commit dcec9ca), not a bug.

Two windows can't distinguish "this candidate is genuinely thin and the 11.85% run is a fluke/artifact" from "this candidate is genuinely significant and the 0.00% run under-sampled it" — you need a third, independent window to see which side two-out-of-three lands on. No code change needed. needs_code_change: false.

Concrete plan for a second agent to execute

  1. Do not reuse either existing window. Prior window was 7 days (2026-07-22→07-29, 69 sessions); current window was ~27 days (2026-07-03→07-30, 170 sessions) per this issue's body. Pick a window with different bounds from both, against real ~/.claude/projects transcripts (no synthetic fixtures) — e.g. 14 days (--since 2026-07-16 --until 2026-07-30) or 60 days (--since 2026-05-31 --until 2026-07-30) via bin/offload-scan's --since/--until flags (see --help/usage at the top of bin/offload-scan).

  2. Run offload-scan for the anxious plugin over that window:

    ./bin/offload-scan --plugin anxious --since <DATE> --until <DATE> --json > /tmp/scan-<window>.json

    (confirm exact flag names against bin/offload-scan --help before running — do not guess).

  3. Run audit-snapshot (or whatever wraps offload-scan + FR-4.5 gate + persistence — see bin/audit-snapshot) against that scan, ensuring it persists to the default snapshot store this time: ~/.cache/inference-arbitrage/wiki (per this issue's Finding 4 — the prior two real runs were NOT persisted there, they lived under a job tmp dir, which is why the gate had no prior window to compare against until the store was seeded by hand). Do not use a scratch/tmp store for this run.

  4. Read the new run's verdict for anxious/agent-wip/release-policy-derivation:

    • share_of_audited_spend and measurement-strength label ("thin" vs "measured") for this window.
    • Compare against BOTH prior windows (0.00% / "thin" and 11.85% / "measured").
    • If the third window lands near 0% (agrees with the 07-22→07-29 window): the 11.85% run was very likely a window artifact — e.g. a burst of agent-wip activity concentrated in early July that a 27-day window captured and a 7-day window didn't. Evidence stays boundary-question; do not file. Comment that conclusion, with numbers, on this issue.
    • If the third window also lands materially above the 2% threshold (agrees with the 11.85% window, 2-out-of-3): that's real signal, not an artifact. The candidate becomes eligible for filing under the normal FR-4.5 stability logic (now backed by 2 of 3 windows in agreement) — file it via the normal offload-audit/filing-plan pipeline, referencing this issue and kotkan/claude-plugin-inference-arbitrage#12/#15 as the provenance trail, and comment the outcome here.
    • If it lands somewhere else entirely (e.g. thin again but for a different reason, or an error): report the raw numbers here without forcing a verdict, and flag for a human — this becomes a case the automated pipeline genuinely cannot resolve with data alone.

  5. Either way, persist the new snapshot properly (step 3) so a fourth run, if ever needed, has three real prior windows to compare against instead of needing another hand-seed.

No files need to change to execute this plan — it is a data-collection/measurement action against the existing, working tooling (bin/offload-scan, bin/audit-snapshot), not a code fix.

## This is not a code-defect issue — it needs a third real measurement window, not a fix Cross-referenced while investigating kotkan/claude-plugin-inference-arbitrage#12 (a different, secondary finding split from #11). This issue (#15) is the FR-4.5 stability-gate output itself: the gate did its job — it caught that `anxious/agent-wip/release-policy-derivation` flips between "thin, 0.00%" (prior window, 69 sessions, 2026-07-22→07-29) and "measured, 11.85%" (current window, 170 sessions, 2026-07-03→07-30), straddling the 2% filing threshold, and correctly downgraded the candidate from `file` to `boundary-question` instead of filing it. That is the gate working as designed (commit `dcec9ca`), not a bug. Two windows can't distinguish "this candidate is genuinely thin and the 11.85% run is a fluke/artifact" from "this candidate is genuinely significant and the 0.00% run under-sampled it" — you need a third, independent window to see which side two-out-of-three lands on. **No code change needed. `needs_code_change: false`.** ### Concrete plan for a second agent to execute 1. **Do not reuse either existing window.** Prior window was 7 days (2026-07-22→07-29, 69 sessions); current window was ~27 days (2026-07-03→07-30, 170 sessions) per this issue's body. Pick a window with different bounds from both, against real `~/.claude/projects` transcripts (no synthetic fixtures) — e.g. **14 days** (`--since 2026-07-16 --until 2026-07-30`) or **60 days** (`--since 2026-05-31 --until 2026-07-30`) via `bin/offload-scan`'s `--since`/`--until` flags (see `--help`/usage at the top of `bin/offload-scan`). 2. **Run offload-scan for the `anxious` plugin over that window:** ``` ./bin/offload-scan --plugin anxious --since <DATE> --until <DATE> --json > /tmp/scan-<window>.json ``` (confirm exact flag names against `bin/offload-scan --help` before running — do not guess). 3. **Run audit-snapshot** (or whatever wraps offload-scan + FR-4.5 gate + persistence — see `bin/audit-snapshot`) against that scan, ensuring it persists to the **default** snapshot store this time: `~/.cache/inference-arbitrage/wiki` (per this issue's Finding 4 — the prior two real runs were NOT persisted there, they lived under a job tmp dir, which is *why* the gate had no prior window to compare against until the store was seeded by hand). Do not use a scratch/tmp store for this run. 4. **Read the new run's verdict** for `anxious/agent-wip/release-policy-derivation`: - `share_of_audited_spend` and measurement-strength label ("thin" vs "measured") for this window. - Compare against BOTH prior windows (0.00% / "thin" and 11.85% / "measured"). - **If the third window lands near 0%** (agrees with the 07-22→07-29 window): the 11.85% run was very likely a window artifact — e.g. a burst of `agent-wip` activity concentrated in early July that a 27-day window captured and a 7-day window didn't. Evidence stays `boundary-question`; do not file. Comment that conclusion, with numbers, on this issue. - **If the third window also lands materially above the 2% threshold** (agrees with the 11.85% window, 2-out-of-3): that's real signal, not an artifact. The candidate becomes eligible for filing under the normal FR-4.5 stability logic (now backed by 2 of 3 windows in agreement) — file it via the normal `offload-audit`/filing-plan pipeline, referencing this issue and kotkan/claude-plugin-inference-arbitrage#12/#15 as the provenance trail, and comment the outcome here. - **If it lands somewhere else entirely** (e.g. thin again but for a different reason, or an error): report the raw numbers here without forcing a verdict, and flag for a human — this becomes a case the automated pipeline genuinely cannot resolve with data alone. 5. **Either way, persist the new snapshot properly** (step 3) so a *fourth* run, if ever needed, has three real prior windows to compare against instead of needing another hand-seed. No files need to change to execute this plan — it is a data-collection/measurement action against the existing, working tooling (`bin/offload-scan`, `bin/audit-snapshot`), not a code fix.
admin removed the agent-wip label 2026-07-30 05:52:02 +03:00
oleks commented 2026-07-30 15:58:19 +03:00
Owner
Copy Link
Copy Source

Fixed in code: FR-4.5.1 — a narrow window may not overturn a wide measured reading

Commit 88fc58b on main (plugin v0.6.3 → v0.7.0, new gate behavior). This
acts on the policy question the investigation comment above flagged for a human
("should FR-4.5 treat a thin snapshot from a too-short window as too weak to
trigger a downgrade") — the user confirmed the decision, so the gate now
implements it.

The rule

A thin/unmeasured reading only gets to overturn a measured one when it was
sampled over a comparably wide window. All three must hold for the carve-out to
fire:

  1. the prior reading is measured;
  2. the current reading is thin or unmeasured;
  3. width(current) < 0.9 × width(prior), where width is the observed
    window.until - window.since span in days.

Then the candidate takes a third verdict, insufficient-window: not filed,
not downgraded to boundary-question, no FR-6.5 self-report, and carried
forward — the wider prior snapshot stays the standing comparison point, so the
next audit over an adequate window compares against it and not against this
window's under-sample. It lands in a new stability_deferrals list (deliberately
not stability_findings, which is what the caller files self-reports from), and
carries window_parity (both widths, the floor, the ratio), suppressed_flips
(what would have been reported), and a carry_forward line naming the minimum
width to re-audit at.

Why a ratio and not "at least 21–30 days". The adequate width is a property of
the candidate's invocation rate, which the auditor learns from the measurement
and does not know in advance: this candidate needs ~27 days, a once-per-hour step
is fully measured in one. An absolute floor would be a guess about frequency
dressed up as a threshold. What the gate can know without guessing is that a
window materially narrower than the one that produced the measured reading
cannot fairly contradict it. 0.9 rather than 1.0 because window bounds are
observed session spans, so two nominally identical windows differ by hours.

What still downgrades, unchanged (all three regression-tested):

  • the reverse direction (thin prior → measured current) — there the earlier
    reading is the weak one;
  • any flip between windows that satisfy the 0.9 parity;
  • a threshold crossing where both windows read measured — a window that
    reached measured did observe the candidate, so a moving share is a claim about
    relative spend, not a sampling failure, and narrowness does not excuse it.

No silent fallback: if either window lacks both bounds, width is not
computable, the flip is not excused, and window_parity.comparable is null
with an explicit why. Unreadable windows are visible, never an exemption.

Re-verified against the real data, live store

Run against ~/.cache/inference-arbitrage/wiki (not a fixture store), candidate
anxious/agent-wip/release-policy-derivation:

current window prior (from the live store) result
7.0d, thin, 0.00% 2026-07-30T00:31:45Z — 26.49d, measured, 11.85% insufficient-window, downgraded: 0, stability_findings: []; parity required_days: 23.84
33.4d (real 60-day scan), measured, 13.87% same 26.49d measured 11.85% prior stable, verdict file — reports normally

So the exact comparison that produced this issue now says "window too narrow to
compare fairly" instead of "the confidence label depends on the calendar", and the
two adequately wide windows still resolve cleanly on the measured side — matching
the 0 → 2 → 12 → 15 invocation accumulation documented above.

Changed

  • bin/stability-classify — window_days / window_parity / deferrable, the
    insufficient-window verdict, stability_deferrals, window_parity recorded on
    downgrade findings too (so a reader sees the carve-out was considered and did not
    apply), insufficient_window + window_parity_ratio in the gate block.
  • bin/audit-snapshot — a deferral is recorded in the snapshot's notes rather
    than silently dropped (it belongs in neither candidates nor
    boundary_questions).
  • design/spec.md — FR-4.5.1 as a testable requirement; FR-7.3 covers the new
    outcome; S4 restated (still "mechanically refuses a candidate whose evidence
    is a window artifact", now refused honestly).
  • design/rubric.md §5b — the third boundary: "a narrow window is not instability
    either", with this candidate's four real windows as the worked example.
  • skills/offload-audit/SKILL.md step 4b — explicitly: do not file a deferral
    on the target's repo and do not self-report it here.
  • tests/stability.test.sh — 53 assertions, all pass. New/updated cases: (f) real
    7d-vs-26d → insufficient-window with no self-report; (f2) real 60d-vs-26d →
    stable, files normally; (j) synthetic narrow-vs-wide → deferral; (k) reverse
    direction → still downgrades; (l) 28d-vs-30d comparable flip → still downgrades;
    (m) measured→measured threshold crossing on a narrow window → still
    downgrades; (n) unreadable widths → downgrades, comparability null.
  • tests/calibration/anxious-60d.{scan,classified}.json — the real 60-day scan
    (176 sessions, 2026-06-26 → 2026-07-30) and the candidate's real 60-day
    measurement (15 invocations, offload_value 4,953,520 / 35,722,973 weighted
    tokens = 13.87%), masked shapes only per FR-3.5.

Full suite: 7/7 test files pass (for f in tests/*.test.sh; do bash "$f"; done).

Does this resolve the issue? Not closing it — two things remain

Leaving this open for a human call, per the request:

  1. Finding 4 (the snapshot-persistence gap) is untouched. Every wide window in
    the live store still got there by hand-seeding, not through a real
    offload-scan → boundary-classify → stability-classify → audit-snapshot write
    pass published to the wiki. The recommended next step from the investigation
    comment still stands: one real full offload-audit run on anxious over a
    ≥30-day window, persisted to the default store and published to the wiki, so
    the standing comparison point is a pipeline artifact. Only after that will a
    clean-machine run behave the way this fix assumes. (Worth noting the fix makes
    that run safer, not unnecessary: a cold-store run still reports unchecked
    and files.)
  2. One residual asymmetry, by design and worth a policy call. A wide
    measured current window compared against a narrow thin prior (e.g. this
    candidate's 60-day reading against the leftover 7-day snapshot
    2026-07-29T01-00-00Z) still downgrades to boundary-question and still
    self-reports — the reverse direction was explicitly kept as-is. That is
    defensible (the wide window genuinely contradicts the narrow one, and the human
    was told two things) but it means a stale narrow snapshot in the store can
    still suppress a good wide reading once. Symmetrising it — ignoring a
    too-narrow prior outright — would be a second, separate decision, so I have
    not taken it. Flagging, not acting.
## Fixed in code: FR-4.5.1 — a narrow window may not overturn a wide `measured` reading Commit `88fc58b` on `main` (plugin v0.6.3 → **v0.7.0**, new gate behavior). This acts on the policy question the investigation comment above flagged for a human ("should FR-4.5 treat a `thin` snapshot from a too-short window as too weak to trigger a downgrade") — the user confirmed the decision, so the gate now implements it. ### The rule A `thin`/`unmeasured` reading only gets to overturn a `measured` one when it was sampled over a comparably wide window. All three must hold for the carve-out to fire: 1. the prior reading is `measured`; 2. the current reading is `thin` or `unmeasured`; 3. `width(current) < 0.9 × width(prior)`, where width is the observed `window.until - window.since` span in days. Then the candidate takes a **third verdict, `insufficient-window`**: not filed, **not** downgraded to `boundary-question`, **no FR-6.5 self-report**, and carried forward — the wider prior snapshot stays the standing comparison point, so the next audit over an adequate window compares against it and not against this window's under-sample. It lands in a new `stability_deferrals` list (deliberately *not* `stability_findings`, which is what the caller files self-reports from), and carries `window_parity` (both widths, the floor, the ratio), `suppressed_flips` (what *would* have been reported), and a `carry_forward` line naming the minimum width to re-audit at. **Why a ratio and not "at least 21–30 days".** The adequate width is a property of the candidate's invocation rate, which the auditor learns *from* the measurement and does not know in advance: this candidate needs ~27 days, a once-per-hour step is fully measured in one. An absolute floor would be a guess about frequency dressed up as a threshold. What the gate *can* know without guessing is that a window materially narrower than the one that produced the `measured` reading cannot fairly contradict it. `0.9` rather than `1.0` because window bounds are observed session spans, so two nominally identical windows differ by hours. **What still downgrades, unchanged** (all three regression-tested): - the reverse direction (`thin` prior → `measured` current) — there the *earlier* reading is the weak one; - any flip between windows that satisfy the 0.9 parity; - a threshold crossing where **both** windows read `measured` — a window that reached `measured` did observe the candidate, so a moving share is a claim about relative spend, not a sampling failure, and narrowness does not excuse it. **No silent fallback:** if either window lacks both bounds, width is not computable, the flip is **not** excused, and `window_parity.comparable` is `null` with an explicit `why`. Unreadable windows are visible, never an exemption. ### Re-verified against the real data, live store Run against `~/.cache/inference-arbitrage/wiki` (not a fixture store), candidate `anxious/agent-wip/release-policy-derivation`: | current window | prior (from the live store) | result | |---|---|---| | 7.0d, `thin`, 0.00% | `2026-07-30T00:31:45Z` — 26.49d, `measured`, 11.85% | **`insufficient-window`**, `downgraded: 0`, `stability_findings: []`; parity `required_days: 23.84` | | 33.4d (real 60-day scan), `measured`, 13.87% | same 26.49d `measured` 11.85% prior | **`stable`**, verdict `file` — reports normally | So the exact comparison that produced this issue now says "window too narrow to compare fairly" instead of "the confidence label depends on the calendar", and the two adequately wide windows still resolve cleanly on the measured side — matching the 0 → 2 → 12 → 15 invocation accumulation documented above. ### Changed - `bin/stability-classify` — `window_days` / `window_parity` / `deferrable`, the `insufficient-window` verdict, `stability_deferrals`, `window_parity` recorded on downgrade findings too (so a reader sees the carve-out was considered and did not apply), `insufficient_window` + `window_parity_ratio` in the gate block. - `bin/audit-snapshot` — a deferral is recorded in the snapshot's `notes` rather than silently dropped (it belongs in neither `candidates` nor `boundary_questions`). - `design/spec.md` — **FR-4.5.1** as a testable requirement; FR-7.3 covers the new outcome; **S4** restated (still "mechanically refuses a candidate whose evidence is a window artifact", now refused honestly). - `design/rubric.md` §5b — the third boundary: "a narrow window is not instability either", with this candidate's four real windows as the worked example. - `skills/offload-audit/SKILL.md` step 4b — explicitly: do **not** file a deferral on the target's repo and do **not** self-report it here. - `tests/stability.test.sh` — 53 assertions, all pass. New/updated cases: (f) real 7d-vs-26d → `insufficient-window` with no self-report; (f2) real 60d-vs-26d → `stable`, files normally; (j) synthetic narrow-vs-wide → deferral; (k) reverse direction → still downgrades; (l) 28d-vs-30d comparable flip → still downgrades; (m) `measured`→`measured` threshold crossing on a narrow window → still downgrades; (n) unreadable widths → downgrades, comparability `null`. - `tests/calibration/anxious-60d.{scan,classified}.json` — the real 60-day scan (176 sessions, 2026-06-26 → 2026-07-30) and the candidate's real 60-day measurement (15 invocations, `offload_value` 4,953,520 / 35,722,973 weighted tokens = 13.87%), masked shapes only per FR-3.5. Full suite: **7/7 test files pass** (`for f in tests/*.test.sh; do bash "$f"; done`). ### Does this resolve the issue? Not closing it — two things remain Leaving this open for a human call, per the request: 1. **Finding 4 (the snapshot-persistence gap) is untouched.** Every wide window in the live store still got there by hand-seeding, not through a real `offload-scan → boundary-classify → stability-classify → audit-snapshot write` pass published to the wiki. The recommended next step from the investigation comment still stands: one real full `offload-audit` run on `anxious` over a ≥30-day window, persisted to the default store **and** published to the wiki, so the standing comparison point is a pipeline artifact. Only after that will a clean-machine run behave the way this fix assumes. (Worth noting the fix makes that run *safer*, not unnecessary: a cold-store run still reports `unchecked` and files.) 2. **One residual asymmetry, by design and worth a policy call.** A wide `measured` current window compared against a *narrow* `thin` prior (e.g. this candidate's 60-day reading against the leftover 7-day snapshot `2026-07-29T01-00-00Z`) still downgrades to `boundary-question` and still self-reports — the reverse direction was explicitly kept as-is. That is defensible (the wide window genuinely contradicts the narrow one, and the human *was* told two things) but it means a stale narrow snapshot in the store can still suppress a good wide reading once. Symmetrising it — ignoring a too-narrow *prior* outright — would be a second, separate decision, so I have not taken it. Flagging, not acting.
oleks added the agent-wip label 2026-07-30 23:29:23 +03:00
oleks commented 2026-07-30 23:51:41 +03:00
Owner
Copy Link
Copy Source

Real end-to-end pipeline run against anxious, 2026-07-30T20:48:20Z (FR-4.5.1 fix, kotkan/claude-plugin-inference-arbitrage@88fc58b, plugin now v0.8.1)

This closes the gap this issue's last comment flagged as still missing: every prior wide-window snapshot for anxious in the local store was seeded by hand via audit-snapshot import. This run went through the actual bin/ scripts end to end and persisted a genuinely pipeline-produced entry.

What ran (all real, against real ~/.claude/projects transcripts and the real Gitea API — no scratch store):

  1. bin/plugin-inventory ~/projects/claude-plugins/anxious --json → inv.json
  2. bin/offload-scan --plugin anxious --days 60 --json → scan.json (window 2026-06-26 → 2026-07-30, 179 sessions, 37,037,942 weighted tokens audited)
  3. bin/candidate-digest inv.json scan.json --json → digest.json
  4. Boundary-rubric judgment written by hand for anxious/agent-wip/release-policy-derivation (reusing this candidate's already-established real content: the WIP-release staleness digest, llm-over-script-digest, ABSTAIN escalation path) → judgments.json
  5. bin/boundary-classify judgments.json → classified.json: high confidence, verdict file, 13.42% of audited spend, offload_value 4,968,793 over 14 invocations (close to this issue's own investigation figure of 4,953,520/13.87% from a slightly earlier exact window — expected variance from a live rolling 60-day window, not a discrepancy)
  6. Wiki fetch + import, unconditional — fetched Data/anxious/snapshots.jsonl from this wiki via wiki_read, imported via bin/audit-snapshot import (0 new: local store was already 1 run ahead of the wiki, from a prior real run today; import is idempotent by run_id)
  7. bin/stability-classify --classified classified.json --target anxious --scan scan.json --wiki-checked imported → stable.json: compared against the most recent measured prior snapshot (run 2026-07-30T14:24:31Z, 27-day window, 11.35% share) — both measured, share stayed within the filing zone on both sides, no downgrade, no insufficient-window deferral. Confirms the FR-4.5.1 fix is working: the gate correctly picked the wide measured prior rather than the stale thin one that caused this issue originally.
  8. Step 5b marker search: list_issues(labels=['token-offload']) + search_issues('ia-candidate: anxious/agent-wip/release-policy-derivation') on oleks/claude-plugin-anxious — found the marker already present on oleks/claude-plugin-anxious#30 (filed in the prior real run once it first cleared the gate). filing-plan plan correctly resolved this to a comment action, not a re-file.
  9. Executed: added the re-measurement comment to oleks/claude-plugin-anxious#30.
  10. bin/audit-snapshot write → persisted run 2026-07-30T20:48:20Z to the default store (~/.cache/inference-arbitrage/wiki), and published all three pages to this wiki via wiki_write: Audits/anxious/2026-07-30, Audits/anxious/Latest, and an append to Data/anxious/snapshots.jsonl (now 5 runs recorded).

Result: anxious/agent-wip/release-policy-derivation cleared FR-4.5/FR-4.5.1 cleanly this run and commented (not re-filed, since oleks/claude-plugin-anxious#30 already carries the marker). The store now holds a real, pipeline-produced, wiki-published entry for anxious rather than only hand-seeded ones.

Left alone per instruction: the residual design question from this issue's last comment (a wide measured current window vs. a narrow thin prior still downgrading by design) is untouched — that's a human policy call, not resolved by this run.

Not closing this issue — leaving that to a human/parent review.

**Real end-to-end pipeline run against `anxious`, 2026-07-30T20:48:20Z** (FR-4.5.1 fix, kotkan/claude-plugin-inference-arbitrage@88fc58b, plugin now v0.8.1) This closes the gap this issue's last comment flagged as still missing: every prior wide-window snapshot for `anxious` in the local store was seeded by hand via `audit-snapshot import`. This run went through the actual `bin/` scripts end to end and persisted a genuinely pipeline-produced entry. **What ran** (all real, against real `~/.claude/projects` transcripts and the real Gitea API — no scratch store): 1. `bin/plugin-inventory ~/projects/claude-plugins/anxious --json` → `inv.json` 2. `bin/offload-scan --plugin anxious --days 60 --json` → `scan.json` (window 2026-06-26 → 2026-07-30, 179 sessions, 37,037,942 weighted tokens audited) 3. `bin/candidate-digest inv.json scan.json --json` → `digest.json` 4. Boundary-rubric judgment written by hand for `anxious/agent-wip/release-policy-derivation` (reusing this candidate's already-established real content: the WIP-release staleness digest, `llm-over-script-digest`, ABSTAIN escalation path) → `judgments.json` 5. `bin/boundary-classify judgments.json` → `classified.json`: **high confidence, verdict `file`, 13.42% of audited spend, offload_value 4,968,793 over 14 invocations** (close to this issue's own investigation figure of 4,953,520/13.87% from a slightly earlier exact window — expected variance from a live rolling 60-day window, not a discrepancy) 6. **Wiki fetch + import, unconditional** — fetched `Data/anxious/snapshots.jsonl` from this wiki via `wiki_read`, imported via `bin/audit-snapshot import` (0 new: local store was already 1 run ahead of the wiki, from a prior real run today; import is idempotent by `run_id`) 7. `bin/stability-classify --classified classified.json --target anxious --scan scan.json --wiki-checked imported` → `stable.json`: compared against the most recent `measured` prior snapshot (run `2026-07-30T14:24:31Z`, 27-day window, 11.35% share) — **both measured, share stayed within the filing zone on both sides, no downgrade, no `insufficient-window` deferral.** Confirms the FR-4.5.1 fix is working: the gate correctly picked the wide `measured` prior rather than the stale thin one that caused this issue originally. 8. Step 5b marker search: `list_issues(labels=['token-offload'])` + `search_issues('ia-candidate: anxious/agent-wip/release-policy-derivation')` on `oleks/claude-plugin-anxious` — found the marker already present on **oleks/claude-plugin-anxious#30** (filed in the prior real run once it first cleared the gate). `filing-plan plan` correctly resolved this to a `comment` action, not a re-file. 9. Executed: added the re-measurement comment to oleks/claude-plugin-anxious#30. 10. `bin/audit-snapshot write` → persisted run `2026-07-30T20:48:20Z` to the **default store** (`~/.cache/inference-arbitrage/wiki`), and published all three pages to this wiki via `wiki_write`: `Audits/anxious/2026-07-30`, `Audits/anxious/Latest`, and an **append** to `Data/anxious/snapshots.jsonl` (now 5 runs recorded). **Result:** `anxious/agent-wip/release-policy-derivation` cleared FR-4.5/FR-4.5.1 cleanly this run and commented (not re-filed, since oleks/claude-plugin-anxious#30 already carries the marker). The store now holds a real, pipeline-produced, wiki-published entry for `anxious` rather than only hand-seeded ones. **Left alone per instruction:** the residual design question from this issue's last comment (a wide `measured` current window vs. a narrow `thin` prior still downgrading by design) is untouched — that's a human policy call, not resolved by this run. Not closing this issue — leaving that to a human/parent review.
oleks removed the agent-wip label 2026-07-30 23:52:29 +03:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
activity/automate
agent-wip
area/agent-behavior
Agent behavior and delegation logic
 bug
cost
domain/agents
Agent orchestration and multi-agent systems
 efficiency
infra-misconfig
kind/capability-gap
kind/defect
A defect or bug that needs fixing
 token-offload
A step paying for inference where a deterministic script would do (inference-arbitrage audit)
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
admin anton oleks
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: kotkan/claude-plugin-inference-arbitrage#15
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?