kotkan/claude-plugin-inference-arbitrage
3
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

FR-4.5.1: a narrow window may not overturn a wide measured reading
ci/woodpecker/push/test Pipeline was successful
Details

Browse Source 
The stability gate treated every evidence flip as symmetric, and that
produced a false downgrade on the candidate it was written for
(kotkan/claude-plugin-inference-arbitrage#15). Re-measuring
anxious/agent-wip/release-policy-derivation over four real windows with
the same script gave 0 -> 2 -> 12 -> 15 invocations at 7/14/27/60 days:
monotonic accumulation of a step that runs about once every two days, not
a label flipping about. The 7-day window had not caught an unstable
candidate, it had failed to observe a real one.

So a thin/unmeasured reading now only overturns a `measured` one when its
window is at least 90% as wide (observed since->until span). Below that
the comparison takes a third verdict, `insufficient-window`: not filed,
not downgraded to a boundary question, no FR-6.5 self-report, and carried
forward with the wider snapshot left as the standing comparison point.
The parity is relative rather than an absolute day count because the
adequate width is a property of the candidate's invocation rate, which
the auditor does not know in advance.

Unchanged, and tested: the reverse direction (thin prior -> measured now)
still downgrades, a flip between comparably wide windows still downgrades,
and a threshold crossing with `measured` on both sides is never excused by
narrowness. An unreadable window is not an exemption either — the flip
stands and window_parity records that the check could not run.

- bin/stability-classify: window_days/window_parity/deferrable, the
  insufficient-window verdict, stability_deferrals, window_parity on
  findings too, insufficient_window + window_parity_ratio in the gate block
- bin/audit-snapshot: record deferrals in the snapshot notes so the row is
  visible on the wiki page instead of silently dropped
- design/spec.md: FR-4.5.1 as a testable requirement; S4 restated
- design/rubric.md: the third boundary in §5b
- skills/offload-audit/SKILL.md: do not file and do not self-report a deferral
- tests: real 60-day anxious fixture; cases (f)/(f2) on real data, (j)-(n)
  synthetic — 7 test files pass

Verified against the live store (~/.cache/inference-arbitrage/wiki): the
real 7.0d-vs-26.49d comparison now reports insufficient-window with zero
findings, while the real 33.4d-vs-26.49d one reports stable at 13.87% and
files normally.
This commit is contained in:
Oleks
2026-07-30 15:56:30 +03:00
parent 4230782aaa
commit 88fc58b2fd
10 changed files with 892 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files
bin/audit-snapshot
+6 -1
View File
@@ -180,9 +180,14 @@ def build_snapshot(target, inventory, scan, classified, issue_map, explicit, run
# the filing pass — `filing: unavailable` and why (FR-6.4) — which this
# script cannot see and which have to reach the wiki page, not just the
# user's terminal.
# `insufficient-window` is in this list, not in `candidates` or
# `boundary_questions`: an FR-4.5 deferral is neither a filed candidate nor
# a question put to a human, and the wider snapshot it was deferred against
# remains the standing comparison point. Recording it as a note keeps the
# deferral visible on the wiki page instead of silently dropping the row.
notes = [
f"{c['candidate_id']}: {c['verdict']} — {c['reasons'][-1] if c.get('reasons') else ''}"
for c in rows if c.get("verdict") in ("no-offload", "drift-note")
for c in rows if c.get("verdict") in ("no-offload", "drift-note", "insufficient-window")
] + list(extra_notes)
return {
bin/stability-classify
+203
View File
@@ -40,6 +40,66 @@ window and 5% the next is stable — same label, same side of the threshold —
that variation is what `audit-snapshot diff` exists to report as a trend. This
gate fires only when the *description a human receives* changes.
A NARROW WINDOW IS NOT EVIDENCE OF INSTABILITY EITHER
-----------------------------------------------------
The gate as first written treated every flip as symmetric, and that produced a
false downgrade on the very candidate that motivated it
(kotkan/claude-plugin-inference-arbitrage#15). Re-measuring
`anxious/agent-wip/release-policy-derivation` across four real windows with the
same script showed a monotonic accumulation, not a flip:
7 days -> 0 invocations, 0.00% of spend -> thin
14 days -> 2 invocations, 0.72% -> thin
27 days -> 12 invocations, 11.73% -> measured
60 days -> 15 invocations, 13.87% -> measured
The candidate runs about once every two days. A 7-day window did not observe an
unstable candidate; it failed to observe a real one. The wider the window, the
higher the reading — the 60-day pass had every opportunity to dilute the share
back down and instead raised it. Calling that "the confidence label moves with
the calendar" gets the direction of the artifact backwards: the *narrow* window
was the artifact, and the gate punished the candidate for it.
So a weak reading only gets to overturn a `measured` one when it was taken over
a comparably wide window. Concretely, all three must hold:
* the prior reading is `measured`,
* the current reading is `thin` or `unmeasured`,
* the current window is narrower than WINDOW_PARITY_RATIO of the prior
window's observed width (`until - since`),
and then the comparison is not read as instability at all. The candidate takes
a third verdict, `insufficient-window`: it is neither filed nor downgraded, no
self-report is emitted, and it is carried forward — the wider prior snapshot
stays the standing comparison point, so the next audit run over an adequate
window compares against it rather than against this window's under-sample.
The parity is relative, not an absolute day count, on purpose. "At least 21
days" would be a guess about candidate frequency dressed up as a threshold —
this candidate needs ~27 days, a once-a-week step would need more, and a
once-an-hour step is fully measured in one. What the gate can know without
guessing is that a window materially narrower than the one that produced the
`measured` reading cannot fairly contradict it. WINDOW_PARITY_RATIO is 0.9
rather than 1.0 because window bounds are observed session spans, not requested
ranges, so two nominally identical windows differ by hours.
Three things stay downgrades, unchanged:
1. the reverse direction (`thin` prior, `measured` current) — a wider or
equally wide current window that now sees the candidate genuinely
contradicts the earlier reading, and the earlier reading is the weak one;
2. any flip between comparably wide windows;
3. a threshold-side flip in which BOTH windows read `measured`. If the
current window still reached `measured`, it observed the candidate often
enough; a share that then crosses the filing threshold is a claim about
relative spend, not a sampling failure, and narrowness does not excuse it.
Width is read off the snapshot's own `window.since`/`window.until`. When either
window does not carry both bounds the widths are not comparable, and the gate
does NOT quietly excuse the flip: it downgrades as before and says so in the
finding's `window_parity` block, so an unreadable window is visible rather than
a silent exemption.
NO PRIOR WINDOW MEANS UNCHECKED, NOT UNSTABLE
---------------------------------------------
A first-ever audit of a target, or a candidate seen for the first time, has
@@ -80,6 +140,7 @@ of `classified.json`.
"""
import argparse
import datetime
import json
import sys
@@ -92,6 +153,21 @@ GATE = "FR-4.5"
UNCHECKED_NO_HISTORY = "unchecked — no prior window to compare"
UNCHECKED_NO_MATCH = "unchecked — no prior window recorded this candidate"
# The verdict a comparison takes when it is too lopsided to read either way.
# Distinct from `boundary-question` on purpose: a boundary question is something
# a human is asked to judge, and this is the auditor declining to make a claim.
INSUFFICIENT_WINDOW = "insufficient-window"
# Readings that mean "this window did not observe the candidate enough", as
# opposed to `measured`, which means it did.
WEAK_STRENGTHS = ("thin", "unmeasured")
# How wide the current window must be, as a fraction of the prior window that
# produced the `measured` reading, before a weak reading may overturn it. Not
# 1.0: window bounds are observed session spans, so two nominally equal windows
# differ by hours and an exact-parity rule would fire on clock noise.
WINDOW_PARITY_RATIO = 0.9
def keyed(cand, scan, explicit, issue_map):
"""The identity fields FR-5.2 matches on, computed for a candidate that is
@@ -149,10 +225,78 @@ def flips(prior, curr_strength, curr_share):
return out
def window_days(window):
"""Observed width of a window in days, or None when it cannot be read.
None is a real answer, not a default: `window_parity` reports it as
"not comparable" and lets the flip stand, rather than treating an
unreadable window as narrow (which would excuse every flip) or as wide
(which would hide that the check could not run)."""
window = window or {}
bounds = []
for key in ("since", "until"):
raw = window.get(key)
if not isinstance(raw, str):
return None
try:
bounds.append(datetime.datetime.fromisoformat(raw))
except ValueError:
return None
since, until = bounds
# One bound tz-aware and the other naive cannot be subtracted, and guessing
# a zone for the naive one would invent a width.
if (since.tzinfo is None) != (until.tzinfo is None):
return None
return (until - since).total_seconds() / 86400.0
def window_parity(prior_window, curr_window):
"""Whether the current window is wide enough to contradict the prior one.
Always returned, whether or not it changes the outcome, so the finding
records the widths it was decided on."""
prior_days = window_days(prior_window)
curr_days = window_days(curr_window)
out = {
"parity_ratio": WINDOW_PARITY_RATIO,
"prior_days": round(prior_days, 2) if prior_days is not None else None,
"current_days": round(curr_days, 2) if curr_days is not None else None,
"required_days": (round(prior_days * WINDOW_PARITY_RATIO, 2)
if prior_days is not None else None),
}
if prior_days is None or curr_days is None:
out["comparable"] = None
out["why"] = ("window width is unreadable — one of the two windows does not carry "
"both `since` and `until` — so narrowness cannot be established and "
"the flip is not excused; this is recorded rather than defaulted")
return out
out["comparable"] = curr_days >= out["required_days"]
out["why"] = (
f"this window spans {out['current_days']}d against the prior window's "
f"{out['prior_days']}d; a weak reading needs at least "
f"{out['required_days']}d ({WINDOW_PARITY_RATIO:.0%} of the prior width) to "
"contradict a `measured` one"
if not out["comparable"] else
f"this window spans {out['current_days']}d against the prior window's "
f"{out['prior_days']}d, at or above the {out['required_days']}d parity floor, so "
"the two readings are comparably sampled and the flip is real")
return out
def deferrable(prior_strength, curr_strength):
"""The only direction narrowness can excuse: a wide `measured` reading that a
narrow window failed to observe. The reverse (`thin` prior, `measured` now)
is a genuine contradiction in which the *earlier* reading is the weak one,
and a `measured` -> `measured` threshold crossing means both windows saw the
candidate, so neither is excused here."""
return prior_strength == "measured" and curr_strength in WEAK_STRENGTHS
def gate(doc, history, scan, explicit, issue_map):
target = doc.get("target")
window = scan.get("window") or {}
findings = []
deferrals = []
checked = 0
for cand in doc.get("candidates", []):
@@ -180,6 +324,54 @@ def gate(doc, history, scan, explicit, issue_map):
f"{curr_strength}, {curr_share:.2%} of audited spend either side")
continue
parity = window_parity(snap.get("window"), window)
if deferrable(prior.get("measurement_strength"), curr_strength) \
and parity["comparable"] is False:
# Not filed, not downgraded, not self-reported: this comparison
# cannot be made fairly, and saying so is the honest third answer.
cand["verdict"] = INSUFFICIENT_WINDOW
cand["stability"] = (
f"insufficient-window against {snap.get('run_id')} (matched by {by}): "
f"{parity['current_days']}d window read {curr_strength} against a "
f"{parity['prior_days']}d window's measured "
f"{((prior.get('measurement') or {}).get('share_of_plugin') or 0):.2%} — "
"too narrow to compare fairly")
cand.setdefault("reasons", []).append(
f"DEFERRED ({GATE}): the current window ({parity['current_days']}d) is "
f"narrower than the {parity['required_days']}d parity floor set by the "
f"{parity['prior_days']}d window that produced the prior `measured` "
"reading, so this window's weak reading cannot fairly contradict it. "
"Neither filed nor downgraded to a boundary question this run; carried "
"forward for a comparison over an adequately wide window.")
deferrals.append({
"gate": GATE,
"candidate_id": cand.get("candidate_id"),
"target": target,
"skill": cand.get("skill"),
"matched_by": by,
"prior": {
"run_id": snap.get("run_id"),
"window": snap.get("window"),
"measurement_strength": prior.get("measurement_strength"),
"share_of_audited_spend": (prior.get("measurement") or {}).get("share_of_plugin"),
},
"current": {
"window": window,
"measurement_strength": curr_strength,
"share_of_audited_spend": curr_share,
},
# What WOULD have been reported as instability, kept so the
# deferral is auditable rather than an unexplained absence.
"suppressed_flips": moved,
"window_parity": parity,
"carry_forward": (
f"re-audit over a window of at least {parity['required_days']}d and "
f"compare against {snap.get('run_id')}, which remains the standing "
"comparison point — this window contributes no reading that could "
"overturn it"),
})
continue
cand["verdict"] = "boundary-question"
cand["stability"] = f"unstable against {snap.get('run_id')} (matched by {by})"
cand.setdefault("reasons", []).append(
@@ -206,6 +398,10 @@ def gate(doc, history, scan, explicit, issue_map):
"share_of_audited_spend": curr_share,
},
"flips": moved,
# Recorded on downgrades too, so a reader can see the widths this
# was decided on and that the narrow-window carve-out was
# considered and did not apply.
"window_parity": parity,
"headline": (
f"{cand.get('candidate_id')} is {prior.get('measurement_strength')} at "
f"{((prior.get('measurement') or {}).get('share_of_plugin') or 0):.2%} of spend in "
@@ -226,14 +422,21 @@ def gate(doc, history, scan, explicit, issue_map):
summary["boundary_questions"] = sum(1 for c in rows if c.get("verdict") == "boundary-question")
summary["stability_checked"] = checked
summary["stability_downgraded"] = len(findings)
summary["stability_insufficient_window"] = len(deferrals)
doc["stability_gate"] = {
"gate": GATE,
"snapshots_available": len(history),
"checked": checked,
"downgraded": len(findings),
"insufficient_window": len(deferrals),
"window_parity_ratio": WINDOW_PARITY_RATIO,
}
doc["stability_findings"] = findings
# Kept out of `stability_findings` deliberately: those are FR-6.5
# self-reports the caller files against this plugin's repo, and a deferral
# is not a defect to report — it is a comparison that was not made.
doc["stability_deferrals"] = deferrals
return doc