oleks/analytics-playbook
1
0
Fork 0
Code Issues 8 Pull Requests Actions Packages Projects 1 Releases Wiki Activity

playbook omits the drill-down tools its own conclusions need #2

New Issue
Closed
opened 2026-09-06 01:31:42 +03:00 by oleks · 4 comments
oleks commented 2026-09-06 01:31:42 +03:00
Owner
Copy Link
Copy Source

What

playbook.xsh's docstring says it collects "every token-usage and plugin-workflow report in one run". It does not. Present in the audit plugins, never invoked:

tool what it gives that the playbook lacks
cc-tokens context --session <id> per-turn context-growth curve for one session — where context stepped up
pa firings [--slowest] [--errors-only] [--slower-than MS] the individual slow/failing hook invocations behind pa hooks' aggregates
pa inventory, pa tags --all static fleet inventory; version/tag gaps across every plugin
instrument report --all-profiles --group-by plugin,event,hook measured hook CPU cost per invocation — the whole hook-instrument plugin is absent from the run
instrument report --failures / --bursts / --gaps non-zero-exit hooks, host-wide stall windows, holes in the record stream
instrument check-wiring -root <plugin> static proof a hook actually dispatches (dispatch fails open, so exit 0 proves nothing)
boundary-classify, stability-classify, filing-plan the deterministic half of the offload decision — the run stops at candidate-digest

Why it matters

The two gaps that cost the most:

  1. No cc-tokens context. The run establishes that 97.7% of all tokens are context re-read at 246.8k mean per turn, and hogs names the culprit sessions (b7b84b10, 1,498 turns, peak 999.6k). It never asks what pushed those contexts up. context is the only tool that answers that, and it is the answer any deterministic gate would key on.
  2. candidate-digest is the last IA step run. boundary-classify and stability-classify exist precisely so the rubric and the evidence-stability gate are applied mechanically. Stopping at the digest hands a model the job two scripts already do.

hook-instrument is not in DEFAULT_PLUGINS at all, so hook cost — the economics of the thing the playbook exists to inform — is unmeasured.

Acceptance

Add the steps, or narrow the docstring's claim. Suggested shape: a context step for the top N sessions from hogs, a hi group wrapping instrument report/check-wiring, and the two IA classifiers chained after candidate-digest.

## What `playbook.xsh`'s docstring says it collects "every token-usage and plugin-workflow report in one run". It does not. Present in the audit plugins, never invoked: | tool | what it gives that the playbook lacks | |---|---| | `cc-tokens context --session <id>` | per-turn context-growth curve for one session — **where** context stepped up | | `pa firings [--slowest] [--errors-only] [--slower-than MS]` | the individual slow/failing hook invocations behind `pa hooks`' aggregates | | `pa inventory`, `pa tags --all` | static fleet inventory; version/tag gaps across every plugin | | `instrument report --all-profiles --group-by plugin,event,hook` | measured hook CPU cost per invocation — the whole `hook-instrument` plugin is absent from the run | | `instrument report --failures / --bursts / --gaps` | non-zero-exit hooks, host-wide stall windows, holes in the record stream | | `instrument check-wiring -root <plugin>` | static proof a hook actually dispatches (dispatch fails open, so exit 0 proves nothing) | | `boundary-classify`, `stability-classify`, `filing-plan` | the deterministic half of the offload decision — the run stops at `candidate-digest` | ## Why it matters The two gaps that cost the most: 1. **No `cc-tokens context`.** The run establishes that 97.7% of all tokens are context re-read at 246.8k mean per turn, and `hogs` names the culprit sessions (`b7b84b10`, 1,498 turns, peak 999.6k). It never asks what pushed those contexts up. `context` is the only tool that answers that, and it is the answer any deterministic gate would key on. 2. **`candidate-digest` is the last IA step run.** `boundary-classify` and `stability-classify` exist precisely so the rubric and the evidence-stability gate are applied mechanically. Stopping at the digest hands a model the job two scripts already do. `hook-instrument` is not in `DEFAULT_PLUGINS` at all, so hook cost — the economics of the thing the playbook exists to inform — is unmeasured. ## Acceptance Add the steps, or narrow the docstring's claim. Suggested shape: a `context` step for the top N sessions from `hogs`, a `hi` group wrapping `instrument report`/`check-wiring`, and the two IA classifiers chained after `candidate-digest`.
oleks added the enhancement label 2026-09-06 01:31:42 +03:00
oleks commented 2026-09-06 01:44:24 +03:00
Author
Owner
Copy Link
Copy Source

Sequencing note — the tier below this one

Everything in this issue is runnable today: cc-tokens context, pa firings/inventory/tags, all of hook-instrument, and IA's boundary-classify / stability-classify / filing-plan all exist and work. Nothing here waits on anything.

Worth recording alongside it, though, that closing this issue still will not answer the question the playbook exists to serve — which tool calls to gate on. That measurement does not exist anywhere in the fleet and is tracked separately:

The relationship, so a later reader does not have to reconstruct it:

tier question status
1 how much, and whose covered — cc-tokens + pa utilization/cost/heatmap
2 where inside a session; what the existing hooks cost this issue — tools exist, playbook does not run them
3 which tool result caused the growth kotkan/claude-plugin-token-budget#16 — no tool anywhere

Deliberately not adding a blocked label: this issue is not blocked by #16, and marking it so would stall six actionable steps behind one that has not started.

If #16 lands, this issue gains one more step (collect the payload ranking per hog session). If it never lands, everything listed here is still worth doing.

## Sequencing note — the tier below this one Everything in this issue is runnable **today**: `cc-tokens context`, `pa firings`/`inventory`/`tags`, all of `hook-instrument`, and IA's `boundary-classify` / `stability-classify` / `filing-plan` all exist and work. Nothing here waits on anything. Worth recording alongside it, though, that closing this issue still will not answer the question the playbook exists to serve — *which tool calls to gate on*. That measurement does not exist anywhere in the fleet and is tracked separately: - **kotkan/claude-plugin-token-budget#16** — nothing attributes context growth to tool-result payload size (labeled `priority/high`). The relationship, so a later reader does not have to reconstruct it: | tier | question | status | |---|---|---| | 1 | how much, and whose | covered — `cc-tokens` + `pa utilization`/`cost`/`heatmap` | | 2 | where inside a session; what the existing hooks cost | **this issue** — tools exist, playbook does not run them | | 3 | which tool result caused the growth | kotkan/claude-plugin-token-budget#16 — no tool anywhere | Deliberately **not** adding a `blocked` label: this issue is not blocked by #16, and marking it so would stall six actionable steps behind one that has not started. If #16 lands, this issue gains one more step (collect the payload ranking per hog session). If it never lands, everything listed here is still worth doing.
oleks added the agent/wip label 2026-09-06 12:18:41 +03:00
oleks added this to the analytics-playbook-board project 2026-09-06 12:18:42 +03:00
oleks commented 2026-09-06 12:19:13 +03:00
Author
Owner
Copy Link
Copy Source

Plan (posted before the first code edit) — and two corrections to the table above

Worked as one pass with oleks/analytics-playbook#1 and oleks/analytics-playbook#3 — same file.

I re-derived every named tool's real surface from its own --help/source rather than from the table, because a sibling issue in this same batch (kotkan/claude-plugin-plugin-analytics#14) asserted seven affected subcommands where there were eight — the evidence came from a playbook run, and the playbook never called the eighth. That check paid off twice here.

Correction 1 — the table is SHORT: cc-tokens payloads exists and is missing from it

cc-tokens --help lists eight subcommands, not the seven the playbook knows about plus context:

{composition,context,daily,hogs,models,payloads,projects,sessions}

  payloads     WHICH tool results filled the context, ranked by re-read
               cost (approx size x turns remaining) — the per-tool-call
               axis the five aggregate axes above cannot provide

This matters more than a missing row, because it falsifies the tier table in this issue's own sequencing comment. That comment says tier 3 — "which tool result caused the growth" — is a measurement that "does not exist anywhere in the fleet", tracked as kotkan/claude-plugin-token-budget#16. It exists now. Verified running:

$ cc-tokens payloads --root ~/.claude --days 2 --json
{"estimate": "sizes are APPROXIMATE (~4 chars/token, fleet rule)",
 "rank_key": "rereads = approx_tokens * turns_after",
 "totals": {"tool_results": 2415, "approx_reread_tokens": 72970033,
            "measured_cache_read_tokens": 653310712,
            "explained_pct": 11.2, "consistent": true}, "payloads": [...]}

I am adding it to the run. I am not touching token-budget (a peer holds kotkan/claude-plugin-token-budget#16); whether that issue should now close is the peer's and the shepherd's call, not mine — flagging it, not acting on it.

Correction 2 — the table is WRONG about the IA classifiers: they cannot be chained here

The issue proposes "the two IA classifiers chained after candidate-digest" and calls them "the deterministic half of the offload decision". They are deterministic, but their input is not. From inference-arbitrage/bin/boundary-classify's own docstring:

Applies rubric §6's confidence table and the FR-4.2 filing gate to candidates whose determinism-test outcomes and falsifiability triple were judged by the analyst agent. The judgments are inference (T2 and T4 are irreducibly semantic).

Its CLI confirms it: boundary-classify [judgments] takes a candidate judgments JSON, not candidate-digest output. stability-classify --classified then consumes boundary-classify's output, and filing-plan sits downstream of that. So the whole three-tool chain is gated behind a model pass.

The playbook is "Deliberately collection-only: no claude -p". Chaining these would either break that property or feed the classifiers fabricated judgments and produce confident grades from nothing. Neither is acceptable, so I am not adding them — and the docstring will say so explicitly rather than leave the gap unexplained.

What I am adding (all verified working before promising)

step why it is cheap enough to add
cc-tokens payloads per profile same cost class as the six cc-tokens steps already run
cc-tokens context --session <id> for the top N hog sessions N is small and bounded; this is the issue's headline gap
pa inventory, pa tags --all, pa firings --slowest all --json, same cost class as the seven pa steps already run
new hi group: instrument report --all-profiles --group-by plugin,event,hook, --failures, --bursts, --gaps; instrument check-wiring per plugin reads a state log, not transcripts

What I am NOT adding, and why (this is the narrowing half)

  • The three IA classifiers — correction 2 above. The docstring will name them as deliberately excluded and say the reason is that they consume model judgments.
  • hook-instrument into DEFAULT_PLUGINS — this is explicitly an operator decision, since it changes what every group iterates. It also turns out to be unnecessary for the measurement: instrument report --all-profiles reads the state log directly and needs no entry in the plugin list. Recommendation: add it separately, on its own merits, not as a side effect of this issue. The hi group works either way.

Per the issue's acceptance ("add the steps, or narrow the claim"), I am doing both halves honestly: adding what is genuinely collectible, and narrowing the docstring to name what it deliberately excludes and why. A quietly shortened claim would be the same defect pointing the other way.

Incidental observation

instrument report --all-profiles reads a third profile this repo's tooling does not otherwise mention: /home/oleks/.claude-old (157 records, against 199,927 and 188,121 for the two live ones). Recording it because a coverage claim that says "both profiles" while a tool reads three is exactly the class of thing oleks/analytics-playbook#3 is about. Not acting on it.

Wall-clock cost of everything added will be measured against the baseline run and reported here before this closes.

## Plan (posted before the first code edit) — and two corrections to the table above Worked as one pass with oleks/analytics-playbook#1 and oleks/analytics-playbook#3 — same file. I re-derived every named tool's real surface from its own `--help`/source rather than from the table, because a sibling issue in this same batch (kotkan/claude-plugin-plugin-analytics#14) asserted seven affected subcommands where there were eight — the evidence came from a playbook run, and the playbook never called the eighth. That check paid off twice here. ### Correction 1 — the table is SHORT: `cc-tokens payloads` exists and is missing from it `cc-tokens --help` lists **eight** subcommands, not the seven the playbook knows about plus `context`: ``` {composition,context,daily,hogs,models,payloads,projects,sessions} payloads WHICH tool results filled the context, ranked by re-read cost (approx size x turns remaining) — the per-tool-call axis the five aggregate axes above cannot provide ``` This matters more than a missing row, because **it falsifies the tier table in this issue's own sequencing comment.** That comment says tier 3 — "which tool result caused the growth" — is a measurement that "does not exist anywhere in the fleet", tracked as kotkan/claude-plugin-token-budget#16. It exists now. Verified running: ``` $ cc-tokens payloads --root ~/.claude --days 2 --json {"estimate": "sizes are APPROXIMATE (~4 chars/token, fleet rule)", "rank_key": "rereads = approx_tokens * turns_after", "totals": {"tool_results": 2415, "approx_reread_tokens": 72970033, "measured_cache_read_tokens": 653310712, "explained_pct": 11.2, "consistent": true}, "payloads": [...]} ``` I am adding it to the run. I am **not** touching `token-budget` (a peer holds kotkan/claude-plugin-token-budget#16); whether that issue should now close is the peer's and the shepherd's call, not mine — flagging it, not acting on it. ### Correction 2 — the table is WRONG about the IA classifiers: they cannot be chained here The issue proposes "the two IA classifiers chained after `candidate-digest`" and calls them "the deterministic half of the offload decision". They are deterministic, but their **input is not**. From `inference-arbitrage/bin/boundary-classify`'s own docstring: > Applies rubric §6's confidence table and the FR-4.2 filing gate to candidates whose determinism-test outcomes and falsifiability triple **were judged by the analyst agent**. **The judgments are inference (T2 and T4 are irreducibly semantic).** Its CLI confirms it: `boundary-classify [judgments]` takes a *candidate judgments JSON*, not `candidate-digest` output. `stability-classify --classified` then consumes `boundary-classify`'s output, and `filing-plan` sits downstream of that. So the whole three-tool chain is gated behind a model pass. The playbook is **"Deliberately collection-only: no claude -p"**. Chaining these would either break that property or feed the classifiers fabricated judgments and produce confident grades from nothing. **Neither is acceptable, so I am not adding them** — and the docstring will say so explicitly rather than leave the gap unexplained. ### What I am adding (all verified working before promising) | step | why it is cheap enough to add | |---|---| | `cc-tokens payloads` per profile | same cost class as the six `cc-tokens` steps already run | | `cc-tokens context --session <id>` for the top N hog sessions | N is small and bounded; this is the issue's headline gap | | `pa inventory`, `pa tags --all`, `pa firings --slowest` | all `--json`, same cost class as the seven `pa` steps already run | | new `hi` group: `instrument report --all-profiles --group-by plugin,event,hook`, `--failures`, `--bursts`, `--gaps`; `instrument check-wiring` per plugin | reads a state log, not transcripts | ### What I am NOT adding, and why (this is the narrowing half) - **The three IA classifiers** — correction 2 above. The docstring will name them as deliberately excluded and say the reason is that they consume model judgments. - **`hook-instrument` into `DEFAULT_PLUGINS`** — this is explicitly an operator decision, since it changes what *every* group iterates. It also turns out to be unnecessary for the measurement: `instrument report --all-profiles` reads the state log directly and needs no entry in the plugin list. **Recommendation: add it separately, on its own merits, not as a side effect of this issue.** The `hi` group works either way. Per the issue's acceptance ("add the steps, **or** narrow the claim"), I am doing both halves honestly: adding what is genuinely collectible, and narrowing the docstring to name what it deliberately excludes and why. A quietly shortened claim would be the same defect pointing the other way. ### Incidental observation `instrument report --all-profiles` reads a **third** profile this repo's tooling does not otherwise mention: `/home/oleks/.claude-old` (157 records, against 199,927 and 188,121 for the two live ones). Recording it because a coverage claim that says "both profiles" while a tool reads three is exactly the class of thing oleks/analytics-playbook#3 is about. Not acting on it. Wall-clock cost of everything added will be measured against the baseline run and reported here before this closes.
oleks commented 2026-09-06 12:42:25 +03:00
Author
Owner
Copy Link
Copy Source

Done — 938e09d + 04cf5f4, landed on main

Both halves of the acceptance: added what is genuinely collectible, and narrowed the docstring to name what it excludes and why.

Added (25 new steps per run, all verified in a real run)

step outcome
cc-tokens payloads × 2 profiles ok — 22.3s / 26.5s
cc-tokens context --session × top-3 hogs × 2 profiles ok — 11.2–39.4s each
pa inventory ok — 0.83s
pa tags --all ok — 0.61s
pa firings --slowest ok — 4.44s
new hi group: instrument report / --failures / --bursts / --gaps ok — 1.8–2.3s each
instrument check-wiring per hook-declaring plugin ok — 4 checked, all rc=0

NOT added, stated in the docstring rather than quietly dropped

boundary-classify, stability-classify, filing-plan — for the reason in the plan comment: their input is model-produced judgments, so chaining them in a collection-only run would grade fabricated input. The docstring now carries a WHAT THIS RUN DELIBERATELY DOES NOT COLLECT section naming them and giving that reason, and the summary header repeats it. The issue's suggested shape was wrong on this point and the correction is the substantive finding here — see the plan comment for the tool's own docstring.

hook-instrument is not in DEFAULT_PLUGINS. It turned out not to need to be: instrument report --all-profiles reads the state log directly. Recommendation to the operator: add it on its own merits, not as a side effect of this issue — it changes what every group iterates.

The one round-trip the additions caught immediately

check-wiring exits 64 on a plugin with no hooks/hooks.json, which is neither a pass nor a wiring failure. Six of the ten default plugins declare no hooks, so the first cut logged six failed steps and the summary read "6 did not exit 0" for a non-finding. Fixed in 04cf5f4: those plugins are skipped explicitly and reported as such, keeping declares no hooks distinct from checked and passed —

`check-wiring` covers the 4 plugins that declare hooks; 6 declare none and were not
checked (`token-budget`, `inference-arbitrage`, `plugin-analytics`, `hooks-to-go`,
`reflection`, `load-trend`). All checked plugins exited 0.

Wall-clock cost — the number the operator needs

Two full runs, emmett, --days 30, two profiles, ten plugins:

baseline after delta
total 438.2s 790.6s +352.4s (+80%)
steps 93 118 +25

Attributed, because the two issues should not be billed to each other:

  • This issue's additions: +164.2s. Of which context curves 100.9s (6 sessions), payloads 48.8s, hi group 8.4s, new pa subcommands 5.9s.
  • oleks/analytics-playbook#3's second --root: +192.3s (the ia group goes 176.8s → 369.1s — twice the transcript volume).

So the honest headline is ~7min → ~13min, and the larger half of that is #3, not this issue. If the run's length becomes a problem, the cheapest lever is --hogs-context (default 3; each curve costs 11–39s), then --days. Nothing added here is in the hot path of the other groups.

No step exceeded 60s in either run — the slowest was 39.4s (context on a 464-turn session).

The correction worth carrying forward

cc-tokens payloads exists and is now collected. That falsifies this issue's own sequencing comment, which lists tier 3 — "which tool result caused the growth" — as a measurement that "does not exist anywhere in the fleet", tracked at kotkan/claude-plugin-token-budget#16. It exists and works; the run reports explained_pct: 11.2 against measured cache_read for ~/.claude over 2 days. I have not touched token-budget and take no view on whether #16 should close — that is its holder's and the shepherd's call, but somebody should look.

Limits

Two runs, one host, one point in time. --json was verified per tool before adding, but I did not audit every added subcommand's output schema for downstream consumers — the steps are collected, not yet digested by anything.

## Done — `938e09d` + `04cf5f4`, landed on `main` Both halves of the acceptance: added what is genuinely collectible, and narrowed the docstring to **name** what it excludes and why. ### Added (25 new steps per run, all verified in a real run) | step | outcome | |---|---| | `cc-tokens payloads` × 2 profiles | ok — 22.3s / 26.5s | | `cc-tokens context --session` × top-3 hogs × 2 profiles | ok — 11.2–39.4s each | | `pa inventory` | ok — 0.83s | | `pa tags --all` | ok — 0.61s | | `pa firings --slowest` | ok — 4.44s | | new `hi` group: `instrument report` / `--failures` / `--bursts` / `--gaps` | ok — 1.8–2.3s each | | `instrument check-wiring` per hook-declaring plugin | ok — 4 checked, all rc=0 | ### NOT added, stated in the docstring rather than quietly dropped `boundary-classify`, `stability-classify`, `filing-plan` — for the reason in the plan comment: their input is model-produced judgments, so chaining them in a collection-only run would grade fabricated input. The docstring now carries a `WHAT THIS RUN DELIBERATELY DOES NOT COLLECT` section naming them and giving that reason, and the summary header repeats it. **The issue's suggested shape was wrong on this point** and the correction is the substantive finding here — see the plan comment for the tool's own docstring. `hook-instrument` is **not** in `DEFAULT_PLUGINS`. It turned out not to need to be: `instrument report --all-profiles` reads the state log directly. **Recommendation to the operator: add it on its own merits, not as a side effect of this issue** — it changes what every group iterates. ### The one round-trip the additions caught immediately `check-wiring` exits **64** on a plugin with no `hooks/hooks.json`, which is neither a pass nor a wiring failure. Six of the ten default plugins declare no hooks, so the first cut logged six failed steps and the summary read "6 did not exit 0" for a non-finding. Fixed in `04cf5f4`: those plugins are skipped explicitly and reported as such, keeping *declares no hooks* distinct from *checked and passed* — ``` `check-wiring` covers the 4 plugins that declare hooks; 6 declare none and were not checked (`token-budget`, `inference-arbitrage`, `plugin-analytics`, `hooks-to-go`, `reflection`, `load-trend`). All checked plugins exited 0. ``` ### Wall-clock cost — the number the operator needs Two full runs, emmett, `--days 30`, two profiles, ten plugins: | | baseline | after | delta | |---|---:|---:|---:| | total | **438.2s** | **790.6s** | **+352.4s (+80%)** | | steps | 93 | 118 | +25 | Attributed, because the two issues should not be billed to each other: - **This issue's additions: +164.2s.** Of which `context` curves 100.9s (6 sessions), `payloads` 48.8s, `hi` group 8.4s, new `pa` subcommands 5.9s. - **oleks/analytics-playbook#3's second `--root`: +192.3s** (the `ia` group goes 176.8s → 369.1s — twice the transcript volume). So the honest headline is ~7min → ~13min, and **the larger half of that is #3, not this issue.** If the run's length becomes a problem, the cheapest lever is `--hogs-context` (default 3; each curve costs 11–39s), then `--days`. Nothing added here is in the hot path of the other groups. **No step exceeded 60s in either run** — the slowest was 39.4s (`context` on a 464-turn session). ### The correction worth carrying forward `cc-tokens payloads` exists and is now collected. That **falsifies this issue's own sequencing comment**, which lists tier 3 — "which tool result caused the growth" — as a measurement that "does not exist anywhere in the fleet", tracked at kotkan/claude-plugin-token-budget#16. It exists and works; the run reports `explained_pct: 11.2` against measured `cache_read` for `~/.claude` over 2 days. I have not touched `token-budget` and take no view on whether #16 should close — that is its holder's and the shepherd's call, but somebody should look. ### Limits Two runs, one host, one point in time. `--json` was verified per tool before adding, but I did not audit every added subcommand's *output schema* for downstream consumers — the steps are collected, not yet digested by anything.
oleks closed this issue 2026-09-06 12:42:57 +03:00
oleks added spent time 25 minutes 2026-09-06 12:43:14 +03:00
oleks commented 2026-09-06 12:43:14 +03:00
Author
Owner
Copy Link
Copy Source

⏱ worked | 06-09-26 | session oleks/2fede70c | +25min | session-total 25min (landed at 04cf5f4 on origin/main; issue closed with evidence) | elapsed from timeline

⏱ worked | 06-09-26 | session oleks/2fede70c | +25min | session-total 25min (landed at 04cf5f4 on origin/main; issue closed with evidence) | elapsed from timeline
oleks added agent/worked:oleks/2fede70c and removed agent/wip labels 2026-09-06 12:43:14 +03:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agent/wip
agent/worked:oleks/0ba3d568
agent/worked:oleks/245e3e60
agent/worked:oleks/2fede70c
agent/worked:oleks/948115a1
agent/worked:oleks/b1b00333
bug
enhancement
epic
measurement
nixos-deploy-pending
nixos-deployed
No labels agent/worked:oleks/2fede70c enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No projects
analytics-playbook-board
Done
Assignees
Clear assignees
archivist-agent coach-agent docs-agent (Documentation Agent) issuer-agent repo-worker steward-agent oleks
No Assignees
1 Participants
Notifications
Total Time Spent: 25 minutes
oleks
25 minutes
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: oleks/analytics-playbook#2
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?