Audit pipeline: evidence-strength classifications unstable across measurement windows #11

New Issue

2026-07-29T18:30:58+03:00

issuer-agent commented

2026-07-29 18:30:58 +03:00

Why: During Phase 7 acceptance testing, the pipeline (bin/offload-scan -> bin/boundary-classify) produced a candidate with two DIFFERENT evidence-strength classifications depending only on the measurement window. This directly undermines the plugin's core value proposition — confidently recommending a candidate as "measured" when a different window shows it as "thin" is exactly the "plausible-looking wrong recommendation" failure mode that design/rubric.md warns is worse than missing a candidate.

Symptom: Candidate anxious/agent-wip/release-policy-derivation:

Full 30-day window: measurement_strength: "measured", cost 3.9M weighted tokens = 12% of audited spend — issue body presented confidently with no caveat
Same candidate, 7-day window: share_of_audited_spend: 0.0, offload_value: 0, measurement_strength: "thin" — classifier reason text explicitly says "evidence is thin ... must be stated as such in the issue body"

Impact: A human reviewing only the full-window body sees "measured" + confident 12% with no hint that a shorter window shows nothing. Window-sensitivity is not surfaced, violating the audit's transparency contract. Found during Phase 7 (S4 scenario) acceptance testing by adversarial triage review.

Secondary: Per-invocation weighted-token figure for anxious:agent-wip is ~989k (for a skill whose actual job is ~10 MCP calls). Suggests whole-session attribution may be bleeding into skill-level cost figures rather than isolating actual mechanical/release-related turns — investigate whether offload-scan's invocation-reconstruction or attribution logic is over-attributing session-wide activity.

Acceptance:

Investigate window-sensitivity in bin/boundary-classify — re-run classification across 2+ windows on failing candidate and confirm consistency
Either surface window-sensitivity explicitly in issue-body template with caveat, or require consistency across windows before filing
Investigate per-invocation attribution in offload-scan — verify skill-level cost figures isolate only relevant turns, not session-wide bleed
Confirm corrected evidence-strength labeling is stable and caveat-bearing for single-window "measured" claims

Links: Phase 7 acceptance testing (S4 scenario); design/rubric.md failure-mode warning

**Why**: During Phase 7 acceptance testing, the pipeline (`bin/offload-scan` -> `bin/boundary-classify`) produced a candidate with two DIFFERENT evidence-strength classifications depending only on the measurement window. This directly undermines the plugin's core value proposition — confidently recommending a candidate as "measured" when a different window shows it as "thin" is exactly the "plausible-looking wrong recommendation" failure mode that design/rubric.md warns is worse than missing a candidate. **Symptom**: Candidate `anxious/agent-wip/release-policy-derivation`: - Full 30-day window: `measurement_strength: "measured"`, cost `3.9M weighted tokens = 12% of audited spend` — issue body presented confidently with no caveat - Same candidate, 7-day window: `share_of_audited_spend: 0.0`, `offload_value: 0`, `measurement_strength: "thin"` — classifier reason text explicitly says "evidence is thin ... must be stated as such in the issue body" **Impact**: A human reviewing only the full-window body sees "measured" + confident 12% with no hint that a shorter window shows nothing. Window-sensitivity is not surfaced, violating the audit's transparency contract. Found during Phase 7 (S4 scenario) acceptance testing by adversarial triage review. **Secondary**: Per-invocation weighted-token figure for `anxious:agent-wip` is ~989k (for a skill whose actual job is ~10 MCP calls). Suggests whole-session attribution may be bleeding into skill-level cost figures rather than isolating actual mechanical/release-related turns — investigate whether `offload-scan`'s invocation-reconstruction or attribution logic is over-attributing session-wide activity. **Acceptance**: - [ ] Investigate window-sensitivity in `bin/boundary-classify` — re-run classification across 2+ windows on failing candidate and confirm consistency - [ ] Either surface window-sensitivity explicitly in issue-body template with caveat, or require consistency across windows before filing - [ ] Investigate per-invocation attribution in `offload-scan` — verify skill-level cost figures isolate only relevant turns, not session-wide bleed - [ ] Confirm corrected evidence-strength labeling is stable and caveat-bearing for single-window "measured" claims **Links**: Phase 7 acceptance testing (S4 scenario); design/rubric.md failure-mode warning

oleks referenced this issue from a commit

2026-07-29 20:25:09 +03:00

FR-4.5: evidence-stability gate — refuse to file a candidate whose confidence moved with the window

oleks closed this issue

2026-07-29 20:25:09 +03:00

issuer-agent commented

2026-07-29 20:28:39 +03:00

Fixed by commit dcec9ca on main — the instability is now caught mechanically before filing, rather than by a human running a second review.

What was built. bin/stability-classify, a hard gate in the same register as FR-4.2's overrule-case gate. It runs after bin/boundary-classify and before bin/filing-plan. For every candidate whose verdict is file, it matches the candidate against the most recent prior snapshot that recorded it (FR-5.2 identity: issue > signature hash > slug) and downgrades it to a boundary question — never filed — if either:

measurement_strength differs between the two windows (measured / thin / unmeasured), or
share_of_audited_spend lands on opposite sides of the 2% filing threshold.

Movement within a label is explicitly not instability (12% → 8% stays filed; that is a trend, and audit-snapshot diff already reports it). No prior window means stability: unchecked and the candidate files normally — an absence of history is not evidence of instability, and doing otherwise would contradict FR-2.4.

Proof on the real data from this issue. Both Phase 7 windows are kept verbatim as fixtures. Running the gate on the 7-day classification with the 30-day snapshot in the store:

verdict:   file -> boundary-question
stability: unstable against 2026-07-29T00-00-00Z (matched by slug)
finding:   measurement_strength  measured -> thin
           share_of_audited_spend  0.1194 -> 0.0

It is symmetric: classifying the 30-day window against the 7-day snapshot downgrades it too. Both windows cannot be right — which is the point.

Self-reporting. A downgrade produces a stability_findings entry that is a finding against this repo, not the target's: the target did not change, the auditor described it two ways. It carries an  marker so a re-run comments instead of duplicating, mirroring FR-6.3.

Acceptance checklist from this issue:

Investigate window-sensitivity — reproduced across both windows, confirmed the flip is real and symmetric.
Require consistency across windows before filing — chosen over the caveat option. A hedge a reader can skip is not a gate; the candidate is now not filed at all.
Confirm evidence-strength labelling is stable — enforced mechanically rather than confirmed by inspection.
Not addressed, split out: the secondary finding in this issue — anxious:agent-wip showing ~989k weighted tokens per invocation, suggesting bin/offload-scan may be attributing session-wide activity to a skill rather than isolating its own turns. That is a separate defect in the dynamic pass and is filed separately (see below); it does not block this gate, which works on whatever numbers the scan produces.

Documented as: spec FR-4.5, rubric §5b ("The evidence-stability gate"), shipped verbatim in references/boundary-rubric.md. Wired into skills/offload-audit/SKILL.md as step 4b. bin/ia_store.py factors store access and identity resolution out of bin/audit-snapshot so both scripts hash signatures identically — a divergence there would make the gate match nothing and fail open.

Tests: tests/stability.test.sh, six cases including (f), the real two-window anxious evidence. Full suite 178 assertions, exit 0.

Fixed by commit `dcec9ca` on `main` — the instability is now caught mechanically before filing, rather than by a human running a second review. **What was built.** `bin/stability-classify`, a hard gate in the same register as FR-4.2's overrule-case gate. It runs after `bin/boundary-classify` and before `bin/filing-plan`. For every candidate whose verdict is `file`, it matches the candidate against the most recent prior snapshot that recorded it (FR-5.2 identity: issue > signature hash > slug) and downgrades it to a boundary question — never filed — if either: - `measurement_strength` differs between the two windows (`measured` / `thin` / `unmeasured`), or - `share_of_audited_spend` lands on opposite sides of the 2% filing threshold. Movement *within* a label is explicitly not instability (12% → 8% stays filed; that is a trend, and `audit-snapshot diff` already reports it). No prior window means `stability: unchecked` and the candidate files normally — an absence of history is not evidence of instability, and doing otherwise would contradict FR-2.4. **Proof on the real data from this issue.** Both Phase 7 windows are kept verbatim as fixtures. Running the gate on the 7-day classification with the 30-day snapshot in the store: ``` verdict: file -> boundary-question stability: unstable against 2026-07-29T00-00-00Z (matched by slug) finding: measurement_strength measured -> thin share_of_audited_spend 0.1194 -> 0.0 ``` It is symmetric: classifying the 30-day window against the 7-day snapshot downgrades it too. Both windows cannot be right — which is the point. **Self-reporting.** A downgrade produces a `stability_findings` entry that is a finding against *this* repo, not the target's: the target did not change, the auditor described it two ways. It carries an `` marker so a re-run comments instead of duplicating, mirroring FR-6.3. **Acceptance checklist from this issue:** - [x] Investigate window-sensitivity — reproduced across both windows, confirmed the flip is real and symmetric. - [x] Require consistency across windows before filing — chosen over the caveat option. A hedge a reader can skip is not a gate; the candidate is now not filed at all. - [x] Confirm evidence-strength labelling is stable — enforced mechanically rather than confirmed by inspection. - [ ] **Not addressed, split out:** the secondary finding in this issue — `anxious:agent-wip` showing ~989k weighted tokens per invocation, suggesting `bin/offload-scan` may be attributing session-wide activity to a skill rather than isolating its own turns. That is a separate defect in the dynamic pass and is filed separately (see below); it does not block this gate, which works on whatever numbers the scan produces. **Documented as:** spec FR-4.5, rubric §5b ("The evidence-stability gate"), shipped verbatim in `references/boundary-rubric.md`. Wired into `skills/offload-audit/SKILL.md` as step 4b. `bin/ia_store.py` factors store access and identity resolution out of `bin/audit-snapshot` so both scripts hash signatures identically — a divergence there would make the gate match nothing and fail open. **Tests:** `tests/stability.test.sh`, six cases including (f), the real two-window `anxious` evidence. Full suite 178 assertions, exit 0.

issuer-agent referenced this issue

2026-07-29 20:28:58 +03:00

offload-scan: per-invocation cost may include session-wide activity, not just the skill's own turns #12

oleks referenced this issue

2026-07-30 03:31:20 +03:00

Stability audit: anxious/agent-wip/release-policy-derivation measurement unstable across windows #15

oleks referenced this issue

2026-07-30 05:35:56 +03:00

Stability audit: anxious/agent-wip/release-policy-derivation measurement unstable across windows #15

Sign in to join this conversation.