oleks/analytics-playbook
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects 1 Releases Wiki Activity

collect_ia still scans only profiles[0]; offload-scan --root is repeatable now #3

New Issue
Closed
opened 2026-09-06 11:38:49 +03:00 by oleks · 3 comments
oleks commented 2026-09-06 11:38:49 +03:00
Owner
Copy Link
Copy Source

What

playbook.xsh:126-127 still takes the first profile only:

def collect_ia(run, profiles, days, plugins):
    prof = profiles[0]

The standard invocation passes two (--profile ~/.claude --profile ~/.claude-vlad). The tokens and pa groups cover both; ia covers only ~/.claude.

Why this is now actionable (it was not when kotkan/claude-plugin-inference-arbitrage#55 was filed)

#55 was filed against inference-arbitrage because the fix was thought to need offload-scan to accept multiple roots. It now does. Verified at origin/main (v0.23.0), bin/offload-scan:985:

ap.add_argument("--root", required=True, action="append", metavar="PATH",
                help="REQUIRED absolute path to the profile dir or its "
                     "projects/ store; no default. REPEATABLE — pass it "
                     "once per profile; scanning one profile out of "
                     "several is a partial measurement")

Landed in adb0309 / v0.16.0, CI pipeline #65 pass. coverage.roots in the emitted document now names every store actually scanned, and coverage.caveat states the document covers only those stores.

#55 is correctly closed — its producer half is complete and was independently verified. Both the landing peer and the verifying shepherd stated explicitly on that issue that the consumer half lives here and is the shepherd's to route. This issue is that routing, not a reopening.

Why it matters

Offload candidates are ranked by observed invocation cost. Half the evidence is missing and nothing in SUMMARY.md or manifest.json says so, so the digest reads as a whole-fleet measurement. A skill used mostly from the vlad profile is systematically under-ranked and filing decisions inherit that bias. ~/.claude-vlad is not marginal: in run 20260905-214353 its five sampled sessions ran 100-181 turns at 270k-524k context each.

Acceptance

  1. collect_ia passes every profile as a repeated --root, in one invocation (no merging needed — offload-scan's (message.id, requestId) seen set spans the whole scan, and roots are de-duplicated after projects/ descent).
  2. SUMMARY.md and manifest.json surface coverage.roots from the scan document, so a subset scan cannot present as a whole-fleet one. This is the half that matters more: widening coverage without recording it fixes the smaller problem.
  3. Note the known upstream gap rather than working around it — bin/candidate-digest emits a bare JSON array with no header object, so it carries no coverage field; coverage must be read from the scan.json the digest joins.

Scope note

Same file as oleks/analytics-playbook#1 and oleks/analytics-playbook#2 (playbook.xsh). Should be worked in one pass with them, not separately.

## What `playbook.xsh:126-127` still takes the first profile only: ```python def collect_ia(run, profiles, days, plugins): prof = profiles[0] ``` The standard invocation passes two (`--profile ~/.claude --profile ~/.claude-vlad`). The `tokens` and `pa` groups cover both; `ia` covers only `~/.claude`. ## Why this is now actionable (it was not when kotkan/claude-plugin-inference-arbitrage#55 was filed) #55 was filed against `inference-arbitrage` because the fix was thought to need `offload-scan` to accept multiple roots. **It now does.** Verified at `origin/main` (v0.23.0), `bin/offload-scan:985`: ```python ap.add_argument("--root", required=True, action="append", metavar="PATH", help="REQUIRED absolute path to the profile dir or its " "projects/ store; no default. REPEATABLE — pass it " "once per profile; scanning one profile out of " "several is a partial measurement") ``` Landed in `adb0309` / v0.16.0, CI pipeline #65 pass. `coverage.roots` in the emitted document now names every store actually scanned, and `coverage.caveat` states the document covers only those stores. #55 is **correctly closed** — its producer half is complete and was independently verified. Both the landing peer and the verifying shepherd stated explicitly on that issue that the consumer half lives here and is the shepherd's to route. This issue is that routing, not a reopening. ## Why it matters Offload candidates are ranked by observed invocation cost. Half the evidence is missing and nothing in `SUMMARY.md` or `manifest.json` says so, so the digest reads as a whole-fleet measurement. A skill used mostly from the vlad profile is systematically under-ranked and filing decisions inherit that bias. `~/.claude-vlad` is not marginal: in run `20260905-214353` its five sampled sessions ran 100-181 turns at 270k-524k context each. ## Acceptance 1. `collect_ia` passes **every** profile as a repeated `--root`, in one invocation (no merging needed — `offload-scan`'s `(message.id, requestId)` `seen` set spans the whole scan, and roots are de-duplicated after `projects/` descent). 2. `SUMMARY.md` and `manifest.json` surface `coverage.roots` from the scan document, so a subset scan cannot present as a whole-fleet one. This is the half that matters more: widening coverage without recording it fixes the smaller problem. 3. Note the known upstream gap rather than working around it — `bin/candidate-digest` emits a bare JSON array with no header object, so it carries no coverage field; coverage must be read from the `scan.json` the digest joins. ## Scope note Same file as oleks/analytics-playbook#1 and oleks/analytics-playbook#2 (`playbook.xsh`). Should be worked in one pass with them, not separately.
oleks added the bug label 2026-09-06 11:38:49 +03:00
oleks added the agent/wip label 2026-09-06 12:18:43 +03:00
oleks added this to the analytics-playbook-board project 2026-09-06 12:18:43 +03:00
oleks commented 2026-09-06 12:19:30 +03:00
Author
Owner
Copy Link
Copy Source

Plan (posted before the first code edit)

Worked as one pass with oleks/analytics-playbook#1 and oleks/analytics-playbook#2 — same file, as the issue's scope note asks.

Verified the premise before building on it

The issue's claim that offload-scan --root is now repeatable is the load-bearing one, so I checked it rather than inheriting it. Confirmed at inference-arbitrage/bin/offload-scan: --root carries action="append", and its help text says "REPEATABLE — pass it once per profile; scanning one profile out of several is a partial measurement".

kotkan/claude-plugin-inference-arbitrage#55 stays closed and I am editing nothing in inference-arbitrage. The change is entirely on this side.

Both halves, per acceptance

  1. Widen. collect_ia drops prof = profiles[0] and passes every profile as a repeated --root in one invocation. No merging: offload-scan's (message.id, requestId) seen set spans the whole scan and roots are de-duplicated after projects/ descent, exactly as the issue states.
  2. Record. SUMMARY.md and manifest.json surface coverage.roots read from each scan.json. This is the half that matters more — widening coverage without recording it would make the number better and the report no more honest.
  3. Respect the upstream boundary. bin/candidate-digest emits a bare JSON array with no header object and therefore carries no coverage field. That is by contract; adding one is a breaking schema change for every consumer. Coverage is read from the scan.json the digest joins, not from the digest. I am not touching candidate-digest.

The check that makes it credible

Per the acceptance evidence I was briefed on, a two-profile scan must find more sessions, not merely a bigger number — a missed root shows up as a missing session. I will report per-plugin session counts for one profile vs. both, from the before/after runs. If the totals do not increase I will say so and establish which of the two explanations holds (the second profile genuinely empty for that plugin, vs. roots being de-duplicated wrongly) rather than reporting the widening as a success.

Related observation, not acted on

instrument report --all-profiles (from hook-instrument) reads a third profile: /home/oleks/.claude-old, 157 records against 199,927 and 188,121 for the two live ones. So even after this fix, "every profile passed" means the two the playbook is invoked with, not every profile on the host. The coverage.roots line will state the roots actually scanned, which is precisely the property that keeps that from becoming the next version of this bug. Recorded here rather than widened, since which profiles the playbook should cover is an operator decision.

A separate fail-open found while working oleks/analytics-playbook#1

route-stats <nonexistent-path> --json exits 0 and emits an all-zeros document, making an unreadable path indistinguishable from an empty transcript. Same family of defect as this issue — a partial or failed measurement presenting as a complete one — but it belongs to agents-routing. Filing separately.

## Plan (posted before the first code edit) Worked as one pass with oleks/analytics-playbook#1 and oleks/analytics-playbook#2 — same file, as the issue's scope note asks. ### Verified the premise before building on it The issue's claim that `offload-scan --root` is now repeatable is the load-bearing one, so I checked it rather than inheriting it. Confirmed at `inference-arbitrage/bin/offload-scan`: `--root` carries `action="append"`, and its help text says "REPEATABLE — pass it once per profile; scanning one profile out of several is a partial measurement". kotkan/claude-plugin-inference-arbitrage#55 stays closed and I am editing nothing in `inference-arbitrage`. The change is entirely on this side. ### Both halves, per acceptance 1. **Widen.** `collect_ia` drops `prof = profiles[0]` and passes **every** profile as a repeated `--root` in one invocation. No merging: `offload-scan`'s `(message.id, requestId)` `seen` set spans the whole scan and roots are de-duplicated after `projects/` descent, exactly as the issue states. 2. **Record.** `SUMMARY.md` and `manifest.json` surface `coverage.roots` read from each `scan.json`. This is the half that matters more — widening coverage without recording it would make the number better and the report no more honest. 3. **Respect the upstream boundary.** `bin/candidate-digest` emits a bare JSON array with no header object and therefore carries no coverage field. That is by contract; adding one is a breaking schema change for every consumer. Coverage is read from the `scan.json` the digest joins, not from the digest. I am not touching `candidate-digest`. ### The check that makes it credible Per the acceptance evidence I was briefed on, a two-profile scan must find **more sessions**, not merely a bigger number — a missed root shows up as a missing session. I will report per-plugin session counts for one profile vs. both, from the before/after runs. If the totals do not increase I will say so and establish which of the two explanations holds (the second profile genuinely empty for that plugin, vs. roots being de-duplicated wrongly) rather than reporting the widening as a success. ### Related observation, not acted on `instrument report --all-profiles` (from `hook-instrument`) reads a **third** profile: `/home/oleks/.claude-old`, 157 records against 199,927 and 188,121 for the two live ones. So even after this fix, "every profile passed" means the two the playbook is invoked with, not every profile on the host. The `coverage.roots` line will state the roots actually scanned, which is precisely the property that keeps that from becoming the next version of this bug. Recorded here rather than widened, since which profiles the playbook should cover is an operator decision. ### A separate fail-open found while working oleks/analytics-playbook#1 `route-stats <nonexistent-path> --json` exits 0 and emits an all-zeros document, making an unreadable path indistinguishable from an empty transcript. Same family of defect as this issue — a partial or failed measurement presenting as a complete one — but it belongs to `agents-routing`. Filing separately.
oleks commented 2026-09-06 12:41:36 +03:00
Author
Owner
Copy Link
Copy Source

Done — 938e09d, landed on main at 04cf5f4

Both halves. Before/after from two real playbook.xsh runs on emmett (x86_64, 8 cores), 2026-09-06, --days 30 --profile ~/.claude --profile ~/.claude-vlad, ten default plugins.

1. Widened — and it finds MORE SESSIONS, which is the check that matters

A bigger number would prove nothing; a missed root shows up as a missing session. It does:

plugin sessions, 1 root sessions, 2 roots delta
plugin-publishing 57 103 +46
decision-flow 48 56 +8
hooks-to-go 5 7 +2
anti-patterns 3 3 +0
token-budget 1 1 +0
reflection 1 1 +0
load-trend 1 1 +0
agents-routing 0 0 +0
inference-arbitrage 0 0 +0
plugin-analytics 0 0 +0
total 116 172 +56 (+48%)

On the seven that did not move — the acceptance asks me to distinguish "the second profile is genuinely empty" from "roots are being de-duplicated wrongly", not to assume the first. The discriminating evidence is within this same table: all ten scans list the identical two roots in coverage.roots, and three of them increased. If de-duplication were collapsing the second root, none could have increased. So the roots are being read, and the seven flat rows are genuine absence of ~/.claude-vlad activity for those plugins — six of which are at 0–1 sessions in total.

Baseline coverage.roots for every plugin was exactly ["/home/oleks/.claude/projects"], which is the defect this issue describes, reproduced.

2. Recorded — the half that mattered more

SUMMARY.md now leads the section with a per-plugin coverage table, and manifest.json carries the same under coverage.ia:

## Offload candidates (inference-arbitrage)

**Coverage.** Profiles passed to `offload-scan --root`: `/home/oleks/.claude`, `/home/oleks/.claude-vlad`

| plugin | roots actually scanned | sessions | attributed/candidate turns |
|---|---|---|---|
| plugin-publishing | `/home/oleks/.claude/projects`, `/home/oleks/.claude-vlad/projects` | 103 | 1590/17142 |
| decision-flow | `/home/oleks/.claude/projects`, `/home/oleks/.claude-vlad/projects` | 56 | 1939/16368 |
…

`roots actually scanned` is read from each `scan.json`'s own `coverage.roots`, not
from what was requested — a root that was asked for and not read would show up here
as a difference.

Note it reports what was read, not what was requested. Printing the requested roots would have re-created this bug one level up.

Control: the word "coverage" and the word "roots" appear zero times in the baseline SUMMARY.md. Verified by grep over the whole file.

3. Upstream boundary respected

bin/candidate-digest is untouched. It emits a bare JSON array by contract, so the summary says in as many words that the digests inherit the table's coverage and state none of their own. Nothing in inference-arbitrage was edited; kotkan/claude-plugin-inference-arbitrage#55 stays closed.

Cost

The second root roughly doubles the ia group: 176.8s → 369.1s (+192.3s). That is inherent — it is twice the transcript volume — and is the price of the measurement being true. Reported separately from oleks/analytics-playbook#2's additions so the operator can see which change bought what.

Limits

Two runs, one host, one point in time; the two profiles this playbook is invoked with. As noted in the plan, instrument report --all-profiles sees a third profile on this host (~/.claude-old, 157 records) that no group here covers. The coverage line now makes that visible instead of hiding it, which is the property that stops it becoming the next version of this bug — but widening to it is an operator decision and I did not take it.

## Done — `938e09d`, landed on `main` at `04cf5f4` Both halves. Before/after from two real `playbook.xsh` runs on emmett (x86_64, 8 cores), 2026-09-06, `--days 30 --profile ~/.claude --profile ~/.claude-vlad`, ten default plugins. ### 1. Widened — and it finds MORE SESSIONS, which is the check that matters A bigger number would prove nothing; a missed root shows up as a missing *session*. It does: | plugin | sessions, 1 root | sessions, 2 roots | delta | |---|---:|---:|---:| | plugin-publishing | 57 | **103** | **+46** | | decision-flow | 48 | **56** | **+8** | | hooks-to-go | 5 | **7** | **+2** | | anti-patterns | 3 | 3 | +0 | | token-budget | 1 | 1 | +0 | | reflection | 1 | 1 | +0 | | load-trend | 1 | 1 | +0 | | agents-routing | 0 | 0 | +0 | | inference-arbitrage | 0 | 0 | +0 | | plugin-analytics | 0 | 0 | +0 | | **total** | **116** | **172** | **+56 (+48%)** | **On the seven that did not move** — the acceptance asks me to distinguish "the second profile is genuinely empty" from "roots are being de-duplicated wrongly", not to assume the first. The discriminating evidence is *within this same table*: all ten scans list the identical two roots in `coverage.roots`, and three of them increased. If de-duplication were collapsing the second root, none could have increased. So the roots are being read, and the seven flat rows are genuine absence of `~/.claude-vlad` activity for those plugins — six of which are at 0–1 sessions in total. Baseline `coverage.roots` for every plugin was exactly `["/home/oleks/.claude/projects"]`, which is the defect this issue describes, reproduced. ### 2. Recorded — the half that mattered more `SUMMARY.md` now leads the section with a per-plugin coverage table, and `manifest.json` carries the same under `coverage.ia`: ``` ## Offload candidates (inference-arbitrage) **Coverage.** Profiles passed to `offload-scan --root`: `/home/oleks/.claude`, `/home/oleks/.claude-vlad` | plugin | roots actually scanned | sessions | attributed/candidate turns | |---|---|---|---| | plugin-publishing | `/home/oleks/.claude/projects`, `/home/oleks/.claude-vlad/projects` | 103 | 1590/17142 | | decision-flow | `/home/oleks/.claude/projects`, `/home/oleks/.claude-vlad/projects` | 56 | 1939/16368 | … `roots actually scanned` is read from each `scan.json`'s own `coverage.roots`, not from what was requested — a root that was asked for and not read would show up here as a difference. ``` Note it reports what was **read**, not what was **requested**. Printing the requested roots would have re-created this bug one level up. **Control:** the word "coverage" and the word "roots" appear **zero times** in the baseline `SUMMARY.md`. Verified by grep over the whole file. ### 3. Upstream boundary respected `bin/candidate-digest` is untouched. It emits a bare JSON array by contract, so the summary says in as many words that the digests inherit the table's coverage and state none of their own. Nothing in `inference-arbitrage` was edited; kotkan/claude-plugin-inference-arbitrage#55 stays closed. ### Cost The second root roughly doubles the `ia` group: **176.8s → 369.1s (+192.3s)**. That is inherent — it is twice the transcript volume — and is the price of the measurement being true. Reported separately from oleks/analytics-playbook#2's additions so the operator can see which change bought what. ### Limits Two runs, one host, one point in time; the two profiles this playbook is invoked with. As noted in the plan, `instrument report --all-profiles` sees a third profile on this host (`~/.claude-old`, 157 records) that no group here covers. The coverage line now makes that visible instead of hiding it, which is the property that stops it becoming the next version of this bug — but widening to it is an operator decision and I did not take it.
oleks closed this issue 2026-09-06 12:42:58 +03:00
oleks added spent time 25 minutes 2026-09-06 12:43:15 +03:00
oleks commented 2026-09-06 12:43:16 +03:00
Author
Owner
Copy Link
Copy Source

⏱ worked | 06-09-26 | session oleks/2fede70c | +25min | session-total 25min (landed at 04cf5f4 on origin/main; issue closed with evidence) | elapsed from timeline

⏱ worked | 06-09-26 | session oleks/2fede70c | +25min | session-total 25min (landed at 04cf5f4 on origin/main; issue closed with evidence) | elapsed from timeline
oleks added agent/worked:oleks/2fede70c and removed agent/wip labels 2026-09-06 12:43:16 +03:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agent/wip
agent/worked:oleks/0ba3d568
agent/worked:oleks/245e3e60
agent/worked:oleks/2fede70c
agent/worked:oleks/948115a1
agent/worked:oleks/b1b00333
bug
enhancement
epic
measurement
nixos-deploy-pending
nixos-deployed
No labels agent/worked:oleks/2fede70c bug
Milestone
No items
No Milestone
Projects
Clear projects
No projects
analytics-playbook-board
Done
Assignees
Clear assignees
archivist-agent coach-agent docs-agent (Documentation Agent) issuer-agent repo-worker steward-agent oleks
No Assignees
1 Participants
Notifications
Total Time Spent: 25 minutes
oleks
25 minutes
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: oleks/analytics-playbook#3
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?