Phase 3 — Dynamic pass #3

New Issue

2026-07-29T16:37:41+03:00

oleks commented

2026-07-29 16:37:41 +03:00

Per design/tasks.md Phase 3. Depends on decision 0.1 (closed — cc-tokens via PATH + cache-glob fallback).

3.1 bin/offload-scan skeleton: streaming JSONL reader, attribution filter, (message.id, requestId) de-dup — delegate token/cost arithmetic to cc-tokens, don't reimplement the pricing table or streaming-snapshot max
3.2 Invocation reconstruction (contiguous attributed runs per session)
3.3 Turn classification: mechanical / judgment / retry
3.4 Tool-signature normalization and masking (also satisfies FR-3.5)
3.5 N-gram mining: recurring sequences ≥3 long in ≥3 invocations
3.6 Metrics: MTR, offload_waste, read amplification, retry density, judgment density, fan-out multiplier, composite offload_value
3.7 Attribution coverage computation (FR-3.4)
3.8 tests/scan.test.sh against synthetic fixture transcripts only — never real transcripts, they contain private content
3.9 Verify runtime/memory on real history: one process, streaming, bounded; record wall time in Methodology/Calibration.md

Implemented in bdb62da. See the comment below for the two spec deviations the real transcripts forced, and the calibration numbers for Methodology/Calibration.md.

Per design/tasks.md Phase 3. Depends on decision 0.1 (closed — `cc-tokens` via PATH + cache-glob fallback). - [x] 3.1 `bin/offload-scan` skeleton: streaming JSONL reader, attribution filter, `(message.id, requestId)` de-dup — delegate token/cost arithmetic to `cc-tokens`, don't reimplement the pricing table or streaming-snapshot max - [x] 3.2 Invocation reconstruction (contiguous attributed runs per session) - [x] 3.3 Turn classification: mechanical / judgment / retry - [x] 3.4 Tool-signature normalization and masking (also satisfies FR-3.5) - [x] 3.5 N-gram mining: recurring sequences ≥3 long in ≥3 invocations - [x] 3.6 Metrics: MTR, `offload_waste`, read amplification, retry density, judgment density, fan-out multiplier, composite `offload_value` - [x] 3.7 Attribution coverage computation (FR-3.4) - [x] 3.8 `tests/scan.test.sh` against **synthetic** fixture transcripts only — never real transcripts, they contain private content - [x] 3.9 Verify runtime/memory on real history: one process, streaming, bounded; record wall time in Methodology/Calibration.md Implemented in bdb62da. See the comment below for the two spec deviations the real transcripts forced, and the calibration numbers for Methodology/Calibration.md.

oleks added this to the v0.1.0 milestone 2026-07-29 16:37:41 +03:00

oleks added a new dependency 2026-07-29 16:38:27 +03:00

#4 Phase 4 — Rubric and classification

oleks commented

2026-07-29 16:59:17 +03:00

Phase 3 landed in bdb62da (3.1–3.9 all checked above). Leaving open for review.

What shipped

bin/offload-scan (~600 lines, Python 3 stdlib only, executable bit set) — one streaming pass over ~/.claude/projects/**/*.jsonl with an mtime prefilter, attribution filter, (message.id, requestId) de-dup taking max on every usage field, invocation reconstruction, turn classification, signature masking, n-gram mining, all §5.2 metrics, and the FR-3.4 coverage headline.
references/signals-catalog.md — four new sections read only by offload-scan: mechanical-tool-calls, read-tool-calls, read-only-bash, scan-thresholds. Same fenced-block contract as the static pass, extended to accept * globs so MCP tools match on their last __ segment. plugin-inventory ignores them; nothing in Phase 2 changed behaviourally.
tests/scan.test.sh + tests/fixtures/transcripts/ — 30 assertions, all passing, against four hand-built synthetic transcripts (generate.py regenerates them). No real transcript content is in this repo, and the suite asserts that: it greps the output for five fixture literals and fails if any survives masking. tests/run-all.sh picks it up automatically via its ./*.test.sh glob; Phase 2's suite still passes; shellcheck clean.

Deviations from design/plan.md §4.1 — both forced by what the transcripts actually contain

Agent attribution is attributionAgent, not agentName. agentName exists but on a separate type: "agent-name" line that maps a sessionId to a human display label ("cluster-grinder", "price-hunter") — it is a Workflow session name, not a subagent type, and it never appears on an assistant line. The real field is attributionAgent on the subagent sidechain transcripts under <session>/subagents/*.jsonl, carrying values like cluster:gitea-agent, anxious:issuer-agent. Those lines also carry attributionPlugin, so agent turns are attributed without needing plugin-inventory --json to supply the agent list at all — the plan's step-2 dependency on the static pass turned out to be unnecessary.
attributionSkill is sometimes bare, sometimes plugin-qualified (planning-with-files vs memory:save). Normalized: an unqualified value is prefixed with the attributed plugin.

The `cc-tokens` dependency — resolution order held, mechanism adjusted

Resolution order is exactly as decided in 0.1: PATH → ~/.claude/plugins/cache/*/token-budget/*/bin/cc-tokens (newest semver) → fail loudly with an install hint. Nothing is vendored and no pricing table is reimplemented.

But it imports the module instead of shelling out with --json, and that gap is worth recording. cc-tokens' actual CLI surface is daily | sessions | projects | models | composition | hogs | context, all of which aggregate by day/session/project/model. None of them can answer "give me weighted tokens for this specific set of turns" — the attribution fields this pass filters on are not on its CLI surface at all, and context --json (the only per-turn output) is scoped to one session and omits message.id/requestId, so there is no join key. Shelling out cannot express the query.

Importing keeps the property that decision 0.1 actually cares about: every token, cost and weighted number in the output is computed by cc_tokens.Rec.cost / Rec.weighted, so the pricing table, the cache multipliers, and the tier normalization still have exactly one home. The import is guarded — a missing Rec, price_for or parse_day fails loudly rather than silently degrading.

This makes the plan's preferred long-term fix concrete: cc-tokens attribute --by skill|plugin|agent --json upstream in token-budget would let offload-scan drop the import for a subprocess call. Worth filing against kotkan/claude-plugin-token-budget when Phase 3 is signed off.

Calibration numbers for `Methodology/Calibration.md` (3.9)

Real-history sanity run, --plugin anxious --days 30, against the live ~/.claude/projects (1.7 GB, ~4800 transcript files):

Measure	Value
Wall clock	94 s
Peak RSS	29 MB
Processes	1 (streaming, no fan-out)
Sessions touched by target	165
Attributed turns	4,915
Candidate turns (coverage denominator)	237,900
Attribution coverage	0.021
Audited weighted tokens	32.6 M
Plugin-wide mechanical share	0.101
Invocations reconstructed	665

Memory is the headline: 29 MB peak over a 1.7 GB history confirms the O(attributed turns + sessions) bound, well inside the emmett constraint (spec S8). 94 s is acceptable for an on-demand audit.

Coverage of 0.021 is a real finding, not a bug, and it is exactly what FR-3.4 exists to surface. The denominator is "every assistant turn in any session where the target was active at all" — and this environment's sessions are long, multi-topic marathons, so a skill invoked once inside a 5,000-turn session drags the ratio down hard. Two consequences for Phase 4: the analyst must lead with this number and treat a dynamic pass at this coverage as weak evidence; and it may be worth reporting a second, tighter denominator (turns within the invocation windows) alongside it, so "we saw 2% of the sessions" is not confused with "we saw 2% of the plugin's work". Flagging for a decision rather than changing the spec unilaterally.

One more observation for the rubric's calibration: turns attributed to anxious show up under agents that are not anxious' own (repo-worker, cluster:gitea-agent) — an anxious-attributed turn executing inside someone else's subagent. The grouping reports this honestly rather than discarding it, and it is likely to matter for the fan-out analysis in Phase 4.

fanout_multiplier is reported unclamped and can come out below 1 (0.2–0.66 on the real run): a subagent spawned with a tight brief genuinely carries less context than a marathon main session. That is fan-out done right, and it is more useful reported than hidden. It is a diagnostic only — it is deliberately not an input to offload_value.

Not done here (by design)

No output from the real-history run is committed anywhere, no snapshot is persisted (Phase 5), and nothing is classified or filed (Phases 4 and 6).

Phase 3 landed in `bdb62da` (3.1–3.9 all checked above). Leaving open for review. ## What shipped - **`bin/offload-scan`** (~600 lines, Python 3 stdlib only, executable bit set) — one streaming pass over `~/.claude/projects/**/*.jsonl` with an mtime prefilter, attribution filter, `(message.id, requestId)` de-dup taking `max` on every usage field, invocation reconstruction, turn classification, signature masking, n-gram mining, all §5.2 metrics, and the FR-3.4 coverage headline. - **`references/signals-catalog.md`** — four new sections read only by `offload-scan`: `mechanical-tool-calls`, `read-tool-calls`, `read-only-bash`, `scan-thresholds`. Same fenced-block contract as the static pass, extended to accept `*` globs so MCP tools match on their last `__` segment. `plugin-inventory` ignores them; nothing in Phase 2 changed behaviourally. - **`tests/scan.test.sh`** + **`tests/fixtures/transcripts/`** — 30 assertions, all passing, against four hand-built synthetic transcripts (`generate.py` regenerates them). No real transcript content is in this repo, and the suite asserts that: it greps the output for five fixture literals and fails if any survives masking. `tests/run-all.sh` picks it up automatically via its `./*.test.sh` glob; Phase 2's suite still passes; shellcheck clean. ## Deviations from design/plan.md §4.1 — both forced by what the transcripts actually contain 1. **Agent attribution is `attributionAgent`, not `agentName`.** `agentName` exists but on a separate `type: "agent-name"` line that maps a `sessionId` to a human display label ("cluster-grinder", "price-hunter") — it is a Workflow session name, not a subagent type, and it never appears on an assistant line. The real field is `attributionAgent` on the subagent sidechain transcripts under `<session>/subagents/*.jsonl`, carrying values like `cluster:gitea-agent`, `anxious:issuer-agent`. Those lines also carry `attributionPlugin`, so agent turns are attributed without needing `plugin-inventory --json` to supply the agent list at all — the plan's step-2 dependency on the static pass turned out to be unnecessary. 2. **`attributionSkill` is sometimes bare, sometimes plugin-qualified** (`planning-with-files` vs `memory:save`). Normalized: an unqualified value is prefixed with the attributed plugin. ## The `cc-tokens` dependency — resolution order held, mechanism adjusted Resolution order is exactly as decided in 0.1: PATH → `~/.claude/plugins/cache/*/token-budget/*/bin/cc-tokens` (newest semver) → **fail loudly** with an install hint. Nothing is vendored and no pricing table is reimplemented. **But it imports the module instead of shelling out with `--json`, and that gap is worth recording.** `cc-tokens`' actual CLI surface is `daily | sessions | projects | models | composition | hogs | context`, all of which aggregate by day/session/project/model. None of them can answer *"give me weighted tokens for this specific set of turns"* — the attribution fields this pass filters on are not on its CLI surface at all, and `context --json` (the only per-turn output) is scoped to one session and omits `message.id`/`requestId`, so there is no join key. Shelling out cannot express the query. Importing keeps the property that decision 0.1 actually cares about: every token, cost and weighted number in the output is computed by `cc_tokens.Rec.cost` / `Rec.weighted`, so the pricing table, the cache multipliers, and the tier normalization still have exactly one home. The import is guarded — a missing `Rec`, `price_for` or `parse_day` fails loudly rather than silently degrading. This makes the plan's **preferred long-term fix concrete**: `cc-tokens attribute --by skill|plugin|agent --json` upstream in `token-budget` would let `offload-scan` drop the import for a subprocess call. Worth filing against `kotkan/claude-plugin-token-budget` when Phase 3 is signed off. ## Calibration numbers for `Methodology/Calibration.md` (3.9) Real-history sanity run, `--plugin anxious --days 30`, against the live `~/.claude/projects` (1.7 GB, ~4800 transcript files): | Measure | Value | |---|---| | Wall clock | **94 s** | | Peak RSS | **29 MB** | | Processes | 1 (streaming, no fan-out) | | Sessions touched by target | 165 | | Attributed turns | 4,915 | | Candidate turns (coverage denominator) | 237,900 | | **Attribution coverage** | **0.021** | | Audited weighted tokens | 32.6 M | | Plugin-wide mechanical share | 0.101 | | Invocations reconstructed | 665 | Memory is the headline: 29 MB peak over a 1.7 GB history confirms the O(attributed turns + sessions) bound, well inside the emmett constraint (spec S8). 94 s is acceptable for an on-demand audit. **Coverage of 0.021 is a real finding, not a bug**, and it is exactly what FR-3.4 exists to surface. The denominator is "every assistant turn in any session where the target was active at all" — and this environment's sessions are long, multi-topic marathons, so a skill invoked once inside a 5,000-turn session drags the ratio down hard. Two consequences for Phase 4: the analyst must lead with this number and treat a dynamic pass at this coverage as weak evidence; and it may be worth reporting a second, tighter denominator (turns within the invocation windows) alongside it, so "we saw 2% of the sessions" is not confused with "we saw 2% of the plugin's work". Flagging for a decision rather than changing the spec unilaterally. One more observation for the rubric's calibration: turns attributed to `anxious` show up under agents that are not `anxious`' own (`repo-worker`, `cluster:gitea-agent`) — an anxious-attributed turn executing inside someone else's subagent. The grouping reports this honestly rather than discarding it, and it is likely to matter for the fan-out analysis in Phase 4. `fanout_multiplier` is reported unclamped and can come out **below 1** (0.2–0.66 on the real run): a subagent spawned with a tight brief genuinely carries less context than a marathon main session. That is fan-out done right, and it is more useful reported than hidden. It is a diagnostic only — it is deliberately not an input to `offload_value`. ## Not done here (by design) No output from the real-history run is committed anywhere, no snapshot is persisted (Phase 5), and nothing is classified or filed (Phases 4 and 6).

oleks closed this issue

2026-07-29 17:12:38 +03:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Blocks

#4 Phase 4 — Rubric and classification

kotkan/claude-plugin-inference-arbitrage

Reference: kotkan/claude-plugin-inference-arbitrage#3

Phase 3 — Dynamic pass #3

What shipped

Deviations from design/plan.md §4.1 — both forced by what the transcripts actually contain

The cc-tokens dependency — resolution order held, mechanism adjusted

Calibration numbers for Methodology/Calibration.md (3.9)

Not done here (by design)

The `cc-tokens` dependency — resolution order held, mechanism adjusted

Calibration numbers for `Methodology/Calibration.md` (3.9)