Phase 5 — Snapshots and trend #5

New Issue

2026-07-29T16:37:56+03:00

oleks commented

2026-07-29 16:37:56 +03:00

Per design/tasks.md Phase 5. Blocked by Phase 4 calibration passing (closed).

5.1 bin/audit-snapshot write|list|diff; schema per design/plan.md §6 (+2 justified additions: target.skills for stale-detection, measurements_by_signature for FR-5.5)
5.2 Candidate identity resolution (issue → signature → slug) and signature-drift detection
5.3 Volume-normalized diffing: cost-per-invocation primary, share-of-spend tiebreak, ±10% noise band; absolute tokens context-only and verified NOT to drive status (test: half volume + flat per-invocation cost → persisting, not shrunk)
5.4 resolved vs claimed-fixed-unconfirmed logic (FR-5.5) — reuses bin/boundary-classify's exact filing threshold rather than reinventing it
5.5 Wiki read/write verified for real against this repo's own wiki (_test-fixture, cleaned up after — wiki back to its original 4 pages)
5.6 skills/offload-trend/SKILL.md
5.7 tests/snapshot.test.sh — 30 assertions

Implemented in ff997d3, 5bdc984, 4d6f6f7. Storage/push design resolved cleanly (offline filesystem-only + explicit pages/import publish seam) — documented and consistent with plan §7's "offline" constraint. Real Gitea quirk found and documented: wiki_write delete takes the URL-decoded sub_url, not the page title. bash tests/run-all.sh: 147 assertions, exit 0.

Per design/tasks.md Phase 5. Blocked by Phase 4 calibration passing (closed). - [x] 5.1 `bin/audit-snapshot write|list|diff`; schema per design/plan.md §6 (+2 justified additions: `target.skills` for stale-detection, `measurements_by_signature` for FR-5.5) - [x] 5.2 Candidate identity resolution (issue → signature → slug) and signature-drift detection - [x] 5.3 Volume-normalized diffing: cost-per-invocation primary, share-of-spend tiebreak, ±10% noise band; absolute tokens context-only and verified NOT to drive status (test: half volume + flat per-invocation cost → `persisting`, not `shrunk`) - [x] 5.4 `resolved` vs `claimed-fixed-unconfirmed` logic (FR-5.5) — reuses `bin/boundary-classify`'s exact filing threshold rather than reinventing it - [x] 5.5 Wiki read/write verified for real against this repo's own wiki (`_test-fixture`, cleaned up after — wiki back to its original 4 pages) - [x] 5.6 `skills/offload-trend/SKILL.md` - [x] 5.7 `tests/snapshot.test.sh` — 30 assertions Implemented in `ff997d3`, `5bdc984`, `4d6f6f7`. Storage/push design resolved cleanly (offline filesystem-only + explicit `pages`/`import` publish seam) — documented and consistent with plan §7's "offline" constraint. Real Gitea quirk found and documented: `wiki_write delete` takes the URL-decoded `sub_url`, not the page title. `bash tests/run-all.sh`: 147 assertions, exit 0.

oleks added this to the v0.1.0 milestone 2026-07-29 16:37:56 +03:00

oleks added a new dependency 2026-07-29 16:38:29 +03:00

#4 Phase 4 — Rubric and classification

oleks added a new dependency 2026-07-29 16:38:30 +03:00

#7 Phase 7 — Acceptance

oleks commented

2026-07-29 17:52:22 +03:00

Phase 5 — snapshots and trend: done (5.1–5.7 checked off in `design/tasks.md`)

Commit ff997d3 (+ 5bdc984 markdown lint). .claude-plugin/plugin.json 0.2.0 → 0.3.0.

What was built

bin/audit-snapshot — write | list | diff, plus two subcommands that carry the wiki bridge: pages (emit what to publish, with content) and import (seed a cold cache from a wiki-fetched snapshots.jsonl).
skills/offload-trend/SKILL.md — the trend-reporting procedure.
references/snapshot-schema.md — the schema, with the two additions to plan §6 justified.
tests/snapshot.test.sh — 30 assertions over two synthetic snapshots, wired into tests/run-all.sh by the existing ./*.test.sh glob. Full suite green.

The storage / push design decision

plan §5 said the store is this plugin's own wiki, "cloned to a local cache and pushed via cluster:gitea-agent (this plugin never writes Gitea directly)". Those cannot both be true of one process — a git push to the wiki repo is a write to Gitea.

Resolved as: bin/audit-snapshot is offline and filesystem-only. It mirrors the wiki tree into a local store (~/.cache/inference-arbitrage/wiki by default) and never opens a socket — no git push, no Gitea API. That keeps plan §7's "offline; only the two output paths touch the network" literally true and keeps wiki credentials entirely outside the plugin. Publishing is a separate explicit caller action: pages emits each wiki page name and its full content, the caller (offload-trend skill, or cluster:gitea-agent) performs the wiki_write. import is the read half, so a machine with a cold cache still diffs against real history. Documented in the script docstring, the skill, and design/tasks.md 5.5.

The wiki is the git-backed store plan §5 asked for — Gitea wiki pages are files in a git repo, so git log on it is the audit trail. Data/<target>/snapshots.jsonl is appended by read-current, add one line, write back, which is why pages emits whole files rather than deltas.

Semantics

Identity (FR-5.2): issue → signature hash → (skill, slug), in that order; matched_by records which one hit. A hash change under a stable identity is signature-drifted, not a silent vanish-plus-new.
Volume normalization (FR-5.4): verdicts come from cost-per-invocation (leading) and share-of-spend (tiebreak), with a ±10% noise band. Absolute tokens are in a context_only block and never drive a status. The test asserts this directly: in a window with half the volume, a candidate whose absolute offload value halved but whose per-invocation cost is flat reports persisting, not shrunk.
resolved vs claimed-fixed-unconfirmed (FR-5.5): resolved requires issue closed and remeasured under 2% of spend with ≥3 invocations — the same MIN_SHARE_OF_SPEND / MIN_INVOCATIONS bar bin/boundary-classify uses to file, reused deliberately rather than reinvented. Anything that merely stopped being reported is claimed-fixed-unconfirmed with a reason naming the missing leg. Same spirit as nixos-deploy-pending / nixos-deployed: closed is not fixed, the way committed is not deployed.
rubric_version was already 1.0.0 in bin/boundary-classify; snapshots pin it and diff reports rubric_comparable: false when two snapshots were graded by different tables.

Two additions to the plan §6 schema

Both because a requirement is otherwise uncomputable, not for convenience:

target.skills — FR-5.3's stale must distinguish "the skill was deleted" from "the candidate got cheap". A candidate list alone cannot.
measurements_by_signature — FR-5.5 needs a later window's measured cost for a signature whose candidate is gone; nothing else in the snapshot carries it, so the resolution test would be unfalsifiable.

status is deliberately not stored on a candidate — it is a property of a pair of snapshots, computed by diff and only ever rendered.

Wiki write path, verified for real

Against this repo's own wiki, under a _test-fixture name, then cleaned up — the wiki is back to its original 4 pages:

created Data/_test-fixture/snapshots.jsonl — slashes and the .jsonl extension are accepted;
read back — base64 byte-identical;
appended a second record via read-modify-write — 2 commits, newline preserved;
published a rendered Audits/_test-fixture/Latest markdown rollup;
round-tripped real content through import → diff on a cold store.

Gitea quirk worth recording: wiki_write create/update take the page title, but delete takes the escaped sub_url URL-decoded (Data/_test-fixture/snapshots.jsonl.-) — deleting by title 404s. Documented in the skill so the publish path does not trip on it.

Not done here

Audits/<target>/… for a real plugin — that needs a real audit with reviewed findings, which is Phase 6/7 territory.

Leaving this issue open for review.

## Phase 5 — snapshots and trend: done (5.1–5.7 checked off in `design/tasks.md`) Commit `ff997d3` (+ `5bdc984` markdown lint). `.claude-plugin/plugin.json` 0.2.0 → 0.3.0. ### What was built - **`bin/audit-snapshot`** — `write | list | diff`, plus two subcommands that carry the wiki bridge: `pages` (emit what to publish, with content) and `import` (seed a cold cache from a wiki-fetched `snapshots.jsonl`). - **`skills/offload-trend/SKILL.md`** — the trend-reporting procedure. - **`references/snapshot-schema.md`** — the schema, with the two additions to plan §6 justified. - **`tests/snapshot.test.sh`** — 30 assertions over two synthetic snapshots, wired into `tests/run-all.sh` by the existing `./*.test.sh` glob. Full suite green. ### The storage / push design decision plan §5 said the store is this plugin's own wiki, "cloned to a local cache and **pushed** via `cluster:gitea-agent` (this plugin **never writes Gitea directly**)". Those cannot both be true of one process — a git push to the wiki repo *is* a write to Gitea. Resolved as: **`bin/audit-snapshot` is offline and filesystem-only.** It mirrors the wiki tree into a local store (`~/.cache/inference-arbitrage/wiki` by default) and never opens a socket — no git push, no Gitea API. That keeps plan §7's "offline; only the two output paths touch the network" literally true and keeps wiki credentials entirely outside the plugin. Publishing is a separate explicit caller action: `pages` emits each wiki page name and its full content, the caller (`offload-trend` skill, or `cluster:gitea-agent`) performs the `wiki_write`. `import` is the read half, so a machine with a cold cache still diffs against real history. Documented in the script docstring, the skill, and `design/tasks.md` 5.5. The wiki **is** the git-backed store plan §5 asked for — Gitea wiki pages are files in a git repo, so `git log` on it is the audit trail. `Data/<target>/snapshots.jsonl` is appended by read-current, add one line, write back, which is why `pages` emits whole files rather than deltas. ### Semantics - **Identity (FR-5.2)**: issue → signature hash → `(skill, slug)`, in that order; `matched_by` records which one hit. A hash change under a stable identity is `signature-drifted`, not a silent vanish-plus-new. - **Volume normalization (FR-5.4)**: verdicts come from cost-per-invocation (leading) and share-of-spend (tiebreak), with a ±10% noise band. Absolute tokens are in a `context_only` block and never drive a status. The test asserts this directly: in a window with **half** the volume, a candidate whose absolute offload value halved but whose per-invocation cost is flat reports `persisting`, not `shrunk`. - **`resolved` vs `claimed-fixed-unconfirmed` (FR-5.5)**: `resolved` requires issue closed **and** remeasured under 2% of spend with ≥3 invocations — the same `MIN_SHARE_OF_SPEND` / `MIN_INVOCATIONS` bar `bin/boundary-classify` uses to file, reused deliberately rather than reinvented. Anything that merely stopped being reported is `claimed-fixed-unconfirmed` with a `reason` naming the missing leg. Same spirit as `nixos-deploy-pending` / `nixos-deployed`: **closed is not fixed**, the way committed is not deployed. - **`rubric_version`** was already `1.0.0` in `bin/boundary-classify`; snapshots pin it and `diff` reports `rubric_comparable: false` when two snapshots were graded by different tables. ### Two additions to the plan §6 schema Both because a requirement is otherwise uncomputable, not for convenience: 1. **`target.skills`** — FR-5.3's `stale` must distinguish "the skill was deleted" from "the candidate got cheap". A candidate list alone cannot. 2. **`measurements_by_signature`** — FR-5.5 needs a *later* window's measured cost for a signature whose candidate is gone; nothing else in the snapshot carries it, so the resolution test would be unfalsifiable. `status` is deliberately **not** stored on a candidate — it is a property of a pair of snapshots, computed by `diff` and only ever rendered. ### Wiki write path, verified for real Against this repo's own wiki, under a `_test-fixture` name, then **cleaned up — the wiki is back to its original 4 pages**: - created `Data/_test-fixture/snapshots.jsonl` — slashes and the `.jsonl` extension are accepted; - read back — base64 byte-identical; - appended a second record via read-modify-write — 2 commits, newline preserved; - published a rendered `Audits/_test-fixture/Latest` markdown rollup; - round-tripped real content through `import` → `diff` on a cold store. **Gitea quirk worth recording**: `wiki_write` `create`/`update` take the page **title**, but `delete` takes the escaped `sub_url` URL-decoded (`Data/_test-fixture/snapshots.jsonl.-`) — deleting by title 404s. Documented in the skill so the publish path does not trip on it. ### Not done here `Audits/<target>/…` for a **real** plugin — that needs a real audit with reviewed findings, which is Phase 6/7 territory. Leaving this issue open for review.

oleks closed this issue

2026-07-29 17:58:13 +03:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Blocks

#7 Phase 7 — Acceptance

kotkan/claude-plugin-inference-arbitrage

Depends on