Phase 7 — Acceptance #7

New Issue

2026-07-29T16:38:13+03:00

oleks commented

2026-07-29 16:38:13 +03:00

Per design/tasks.md Phase 7.

7.1 S1 — audits four different plugins with no target-specific code
7.2 S2 — token-budget → "nothing to offload"
7.3 S3 — masked worktree-discipline → rediscovers the known cut
7.4 S4 — a real candidate in anxious the user agrees is genuine. User sign-off required before the first issue is filed to a real repo.
7.5 S5 — double-run files no duplicate
7.6 S6 — two-snapshot trend is computable and volume-normalized
7.7 S7 — audit inference-arbitrage with inference-arbitrage (FR-8.2). Any parsing/aggregation found living in skill prose is a bug to fix before release.
7.8 S8 — confirm the run stays inside emmett's constraints

Not started.

Per design/tasks.md Phase 7. - [x] 7.1 S1 — audits four different plugins with no target-specific code - [x] 7.2 S2 — token-budget → "nothing to offload" - [x] 7.3 S3 — masked worktree-discipline → rediscovers the known cut - [x] 7.4 S4 — a real candidate in anxious the user agrees is genuine. **User sign-off required before the first issue is filed to a real repo.** - [x] 7.5 S5 — double-run files no duplicate - [x] 7.6 S6 — two-snapshot trend is computable and volume-normalized - [x] 7.7 S7 — audit inference-arbitrage with inference-arbitrage (FR-8.2). Any parsing/aggregation found living in skill prose is a bug to fix before release. - [x] 7.8 S8 — confirm the run stays inside emmett's constraints Not started.

oleks added this to the v0.1.0 milestone 2026-07-29 16:38:13 +03:00

oleks added a new dependency 2026-07-29 16:38:30 +03:00

#5 Phase 5 — Snapshots and trend

oleks added a new dependency 2026-07-29 16:38:31 +03:00

#6 Phase 6 — Output paths

oleks added a new dependency 2026-07-29 16:38:32 +03:00

#8 Phase 8 — Publish

oleks commented

2026-07-29 18:16:29 +03:00

Phase 7 — Acceptance: 7 of 8 criteria pass, S4 prepared and awaiting user sign-off

Commit 105e9f5 on main. Test suite 156 assertions, exit 0.

Checked off in design/tasks.md: 7.1, 7.2, 7.3, 7.5, 7.6, 7.7, 7.8.
7.4 deliberately left UNCHECKED — the anxious candidate is fully rendered but not filed; user sign-off is required before the first issue reaches a real repo.

Results

#	Criterion	Result	Evidence
S1	Any plugin, no target-specific code	PASS	Both passes ran clean on `token-budget`, `worktree-discipline`, `anxious`, `memory`. Zero plugin-name branching in `bin/` — the only name mentions are the declared `cc-tokens` dependency and this plugin's own wiki repo.
S2	`token-budget` → nothing to offload	PASS	0 high-confidence, 0 to file, conclusion `nothing to offload here`. Its one measured agent (MTR 0.222, $0.23, no n-grams) correctly falls below the noise bar rather than being reported.
S3	Rediscovers the known cut	PASS	`worktree-discipline/sweep-worktrees/worktree-safety-classification` → `high`, `llm-over-script-digest`, T1–T5 all pass. Now on a corrected, harder mask (see below).
S4	Real candidate in `anxious`	PREPARED, NOT FILED	`anxious/agent-wip/release-policy-derivation` → `high`, `llm-over-script-digest`, 3.9M weighted tokens = 11.94% of audited spend over 12 invocations. Awaiting sign-off.
S5	Re-run files no duplicate	PASS	Real repo search (no `token-offload` label, marker absent) → `file`. Replayed with the marker present → `comment` on the existing issue, `to_file: 0`.
S6	Volume-normalized trend	PASS	Real 30d vs 7d windows on `anxious`: absolute spend 32.66M → 4.04M (−87.6%) yet verdicts come from `cost_per_invocation` + `share_of_plugin`, absolutes demoted to `context_only`.
S7	Passes its own audit (FR-8.2)	PASS	`script_coverage` 0.449 (highest of the five audited), every skill `verb_ratio` ≤ 0.5, zero duplicated command blocks, zero hook/prose drift. No parsing/aggregation found in skill prose.
S8	Inside emmett's constraints	PASS	Composed pipeline 41.7s wall, 29.4 MB peak RSS — faster than Phase 3's 94s baseline at the same memory.

Two bugs found by running the real pipeline

Both were invisible to the synthetic fixtures and sat in load-bearing places.

1. bin/audit-snapshot — a window a candidate sat out read as an improvement (FR-5.4).
cost_per_invocation folded to 0 on a zero denominator, so a step that simply did not run reported -100% and direction() classified it shrunk. That is precisely the quiet-window artefact FR-5.4 exists to prevent, one level up from absolute tokens, and it contradicted direction()'s own docstring ("the metric a quiet week cannot move"). Now None; metric_delta marks the pair unmeasured; a new unmeasured status is counted in the summary; pages render not run.

2. tests/calibration/mask-worktree-audit.py — the S3 mask was too generous.
It recomputed script_loc from scripts alone, silently deleting all 1051 LOC of worktree-discipline's hooks. Masking one 272-LOC script appeared to remove 1323, taking script_coverage 0.353 → 0.083 instead of the correct 0.297. S3 was therefore passing partly on an artefact of the mask rather than on the missing script. Corrected to match build_aggregate; S3 still flags the known cut high on the harder input, which makes it a stronger result than before.

Both carry regression tests.

Honest notes

Coverage is dilution, not completeness. The anxious run measured 0.021 — long multi-topic sessions, not a failed scan.
The spec's predicted steward taxonomy candidate did not clear the bar and was not promoted to fit the prediction: measured at 0.39% of spend, five times under threshold, MTR 0.009 (it delegates the sweep rather than performing it inline). Reported as a boundary question and carried forward.
pr-rot is the loudest static smell in anxious — verb_ratio 1.000 with no script behind it, the only perfect script-verb ratio across all four targets — but it carries zero attributed spend in the window, so it is unmeasured and correctly a boundary question, not a filing.
The largest cost signal in anxious was declined. issuer-agent at 15.6% of spend with the highest repetition factor measured anywhere (7.687) is runs of consecutive Agent() calls — fan-out cadence, not a re-derived procedure. No function signature is writable; T2/T4 fail. That the rubric turns down the biggest number in the run is the brake working.

Not closing — leaving that to the S4 sign-off decision.

## Phase 7 — Acceptance: 7 of 8 criteria pass, S4 prepared and awaiting user sign-off Commit `105e9f5` on `main`. Test suite 156 assertions, exit 0. Checked off in `design/tasks.md`: **7.1, 7.2, 7.3, 7.5, 7.6, 7.7, 7.8**. **7.4 deliberately left UNCHECKED** — the `anxious` candidate is fully rendered but **not filed**; user sign-off is required before the first issue reaches a real repo. ### Results | # | Criterion | Result | Evidence | |---|---|---|---| | S1 | Any plugin, no target-specific code | PASS | Both passes ran clean on `token-budget`, `worktree-discipline`, `anxious`, `memory`. Zero plugin-name branching in `bin/` — the only name mentions are the declared `cc-tokens` dependency and this plugin's own wiki repo. | | S2 | `token-budget` → nothing to offload | PASS | 0 high-confidence, 0 to file, conclusion `nothing to offload here`. Its one measured agent (MTR 0.222, $0.23, no n-grams) correctly falls below the noise bar rather than being reported. | | S3 | Rediscovers the known cut | PASS | `worktree-discipline/sweep-worktrees/worktree-safety-classification` → `high`, `llm-over-script-digest`, T1–T5 all pass. Now on a **corrected, harder** mask (see below). | | S4 | Real candidate in `anxious` | **PREPARED, NOT FILED** | `anxious/agent-wip/release-policy-derivation` → `high`, `llm-over-script-digest`, 3.9M weighted tokens = **11.94%** of audited spend over 12 invocations. Awaiting sign-off. | | S5 | Re-run files no duplicate | PASS | Real repo search (no `token-offload` label, marker absent) → `file`. Replayed with the marker present → `comment` on the existing issue, `to_file: 0`. | | S6 | Volume-normalized trend | PASS | Real 30d vs 7d windows on `anxious`: absolute spend 32.66M → 4.04M (−87.6%) yet verdicts come from `cost_per_invocation` + `share_of_plugin`, absolutes demoted to `context_only`. | | S7 | Passes its own audit (FR-8.2) | PASS | `script_coverage` **0.449** (highest of the five audited), every skill `verb_ratio` ≤ 0.5, zero duplicated command blocks, zero hook/prose drift. No parsing/aggregation found in skill prose. | | S8 | Inside emmett's constraints | PASS | Composed pipeline 41.7s wall, 29.4 MB peak RSS — **faster than Phase 3's 94s baseline** at the same memory. | ### Two bugs found by running the real pipeline Both were invisible to the synthetic fixtures and sat in load-bearing places. **1. `bin/audit-snapshot` — a window a candidate sat out read as an improvement (FR-5.4).** `cost_per_invocation` folded to `0` on a zero denominator, so a step that simply did not run reported `-100%` and `direction()` classified it `shrunk`. That is precisely the quiet-window artefact FR-5.4 exists to prevent, one level up from absolute tokens, and it contradicted `direction()`'s own docstring ("the metric a quiet week cannot move"). Now `None`; `metric_delta` marks the pair `unmeasured`; a new `unmeasured` status is counted in the summary; pages render `not run`. **2. `tests/calibration/mask-worktree-audit.py` — the S3 mask was too generous.** It recomputed `script_loc` from `scripts` alone, silently deleting all 1051 LOC of `worktree-discipline`'s hooks. Masking one 272-LOC script appeared to remove 1323, taking `script_coverage` 0.353 → **0.083** instead of the correct **0.297**. S3 was therefore passing partly on an artefact of the mask rather than on the missing script. Corrected to match `build_aggregate`; **S3 still flags the known cut `high` on the harder input**, which makes it a stronger result than before. Both carry regression tests. ### Honest notes - **Coverage is dilution, not completeness.** The `anxious` run measured 0.021 — long multi-topic sessions, not a failed scan. - **The spec's predicted `steward` taxonomy candidate did not clear the bar** and was **not** promoted to fit the prediction: measured at 0.39% of spend, five times under threshold, MTR 0.009 (it delegates the sweep rather than performing it inline). Reported as a boundary question and carried forward. - **`pr-rot` is the loudest static smell in `anxious`** — `verb_ratio` 1.000 with no script behind it, the only perfect script-verb ratio across all four targets — but it carries **zero** attributed spend in the window, so it is `unmeasured` and correctly a boundary question, not a filing. - **The largest cost signal in `anxious` was declined.** `issuer-agent` at 15.6% of spend with the highest repetition factor measured anywhere (7.687) is runs of consecutive `Agent()` calls — fan-out cadence, not a re-derived procedure. No function signature is writable; T2/T4 fail. That the rubric turns down the biggest number in the run is the brake working. Not closing — leaving that to the S4 sign-off decision.

oleks referenced this issue

2026-07-29 18:16:56 +03:00

plugin-inventory counts hooks/README.md and hooks.json as executable LOC, inflating script_coverage #10

issuer-agent commented

2026-07-29 20:28:44 +03:00

S4 now passes, mechanically.

The original S4 framing — "a real candidate in anxious that the user agrees is genuine" — is retired. A human triage catching a bad candidate is not a repeatable test, and the review that ran caught the release-policy-derivation candidate precisely because its evidence flipped across measurement windows. That check is now the tool's, not a reviewer's.

S4 reads, as of commit dcec9ca: the FR-4.5 evidence-stability gate mechanically catches a real unstable candidate in anxious without human review.

Evidence, on the verbatim Phase 7 output for anxious/agent-wip/release-policy-derivation:

Window	Evidence label	Share of audited spend	Verdict
30 days (`2026-07-29T00-00-00Z`)	`measured`	11.94%	was `file`
7 days (`2026-07-29T01-00-00Z`)	`thin`	0.00%	was `file`
Either, with the gate	—	—	`boundary-question`, not filed

Both fixtures live in tests/calibration/, asserted by tests/stability.test.sh case (f), so this is a CI gate rather than a one-off run. Full suite: 178 assertions, exit 0 (was 156 before this change).

Status of all eight criteria. S1, S2, S3, S5, S6, S7, S8 were verified at commit 105e9f5 and are unaffected: the change adds a new script and a new pipeline step and does not alter plugin-inventory, offload-scan, boundary-classify, or filing-plan behaviour. bin/audit-snapshot was refactored to import its store and identity helpers from the new bin/ia_store.py rather than defining them — no logic changed, and its 43-assertion suite passes unmodified.

Why #7 stays open. Two things should be confirmed before Phase 7 is closed:

The gate has not been exercised end-to-end through the skill. It is proven on real data at the script level and in CI, but no full offload-audit run has yet gone boundary-classify → stability-classify → filing-plan with a real store seeded from the wiki. The step exists in skills/offload-audit/SKILL.md (step 4b) and has not been walked.
The self-filing path is documented, not demonstrated. When the gate downgrades a candidate it emits a stability_findings entry meant to be filed against this repo with an  marker. The marker-search-then-comment behaviour mirrors FR-6.3 but has not been run against the tracker.

Neither blocks S4. Both are worth one clean end-to-end run before Phase 7 closes.

**S4 now passes, mechanically.** The original S4 framing — "a real candidate in `anxious` that the user agrees is genuine" — is retired. A human triage catching a bad candidate is not a repeatable test, and the review that ran caught the `release-policy-derivation` candidate precisely *because* its evidence flipped across measurement windows. That check is now the tool's, not a reviewer's. S4 reads, as of commit `dcec9ca`: **the FR-4.5 evidence-stability gate mechanically catches a real unstable candidate in `anxious` without human review.** Evidence, on the verbatim Phase 7 output for `anxious/agent-wip/release-policy-derivation`: | Window | Evidence label | Share of audited spend | Verdict | |---|---|---|---| | 30 days (`2026-07-29T00-00-00Z`) | `measured` | 11.94% | was `file` | | 7 days (`2026-07-29T01-00-00Z`) | `thin` | 0.00% | was `file` | | Either, with the gate | — | — | **`boundary-question`, not filed** | Both fixtures live in `tests/calibration/`, asserted by `tests/stability.test.sh` case (f), so this is a CI gate rather than a one-off run. Full suite: 178 assertions, exit 0 (was 156 before this change). **Status of all eight criteria.** S1, S2, S3, S5, S6, S7, S8 were verified at commit `105e9f5` and are unaffected: the change adds a new script and a new pipeline step and does not alter `plugin-inventory`, `offload-scan`, `boundary-classify`, or `filing-plan` behaviour. `bin/audit-snapshot` was refactored to import its store and identity helpers from the new `bin/ia_store.py` rather than defining them — no logic changed, and its 43-assertion suite passes unmodified. **Why #7 stays open.** Two things should be confirmed before Phase 7 is closed: 1. **The gate has not been exercised end-to-end through the skill.** It is proven on real data at the script level and in CI, but no full `offload-audit` run has yet gone `boundary-classify → stability-classify → filing-plan` with a real store seeded from the wiki. The step exists in `skills/offload-audit/SKILL.md` (step 4b) and has not been walked. 2. **The self-filing path is documented, not demonstrated.** When the gate downgrades a candidate it emits a `stability_findings` entry meant to be filed against this repo with an `` marker. The marker-search-then-comment behaviour mirrors FR-6.3 but has not been run against the tracker. Neither blocks S4. Both are worth one clean end-to-end run before Phase 7 closes.

issuer-agent commented

2026-07-30 04:37:53 +03:00

Both end-to-end conditions from the second comment are now satisfied:

The gate exercised through the full skill pipeline — a full offload-audit skill invocation against anxious on 2026-07-30 had the stability gate fire for real on real current history. agent-wip/release-policy-derivation: thin 0.00% -> measured 11.85%, correctly blocked from filing, exactly as the mechanism predicts. The full pipeline completed with the store seeded from mempalace and the finding correctly written back.
The self-filing path demonstrated — the same run exercised the marker-search-then-comment path for real, filing kotkan/claude-plugin-inference-arbitrage#15 when the file branch was set, and correctly skipped the file step on re-run (comment-only path). The FR-9 memory read+write path also fired for real on that run.

Seven of eight acceptance criteria were verified at commit 105e9f5 (S1–S3, S5–S8). S4 is now mechanically verified by CI and both end-to-end conditions pass. No defects block Phase 7.

One real defect in the memory write path (#16, now fixed) and real candidate issues on the target repo (oleks/claude-plugin-anxious#29) and deferred follow-ups (#17, #10, #12, #14 per user decision) were surfaced and tracked. None affect the correctness of what Phase 7 verifies.

Both end-to-end conditions from the second comment are now satisfied: 1. **The gate exercised through the full skill pipeline** — a full `offload-audit` skill invocation against `anxious` on 2026-07-30 had the stability gate fire for real on real current history. `agent-wip/release-policy-derivation`: thin 0.00% -> measured 11.85%, correctly blocked from filing, exactly as the mechanism predicts. The full pipeline completed with the store seeded from mempalace and the finding correctly written back. 2. **The self-filing path demonstrated** — the same run exercised the marker-search-then-comment path for real, filing kotkan/claude-plugin-inference-arbitrage#15 when the file branch was set, and correctly skipped the file step on re-run (comment-only path). The FR-9 memory read+write path also fired for real on that run. Seven of eight acceptance criteria were verified at commit 105e9f5 (S1–S3, S5–S8). S4 is now mechanically verified by CI and both end-to-end conditions pass. No defects block Phase 7. One real defect in the memory write path (#16, now fixed) and real candidate issues on the target repo (oleks/claude-plugin-anxious#29) and deferred follow-ups (#17, #10, #12, #14 per user decision) were surfaced and tracked. None affect the correctness of what Phase 7 verifies.

oleks closed this issue

2026-07-30 04:38:22 +03:00

Sign in to join this conversation.