offload-scan: per-invocation cost may include session-wide activity, not just the skill's own turns #12

New Issue

2026-07-29T20:28:58+03:00

issuer-agent commented

2026-07-29 20:28:58 +03:00

Split out of kotkan/claude-plugin-inference-arbitrage#11, whose primary defect (evidence-strength instability across windows) is fixed by the FR-4.5 gate in commit dcec9ca. This is the secondary finding from that issue, unaddressed.

bin/offload-scan reported anxious:agent-wip at roughly 989k weighted tokens per invocation. That skill's actual job is on the order of ten MCP calls. A figure that large suggests invocation reconstruction or attribution is folding whole-session activity into a per-skill cost, rather than isolating the turns the skill was responsible for.

Why it matters: offload_value and share_of_audited_spend are the numbers the filing threshold (spec §5.2.7) is applied to, and the numbers an issue body presents to a human as the cost of a candidate. If they are inflated by unrelated session work, the plugin over-reports value and files candidates that are not worth what it claims.

Acceptance:

Establish what offload-scan's invocation reconstruction actually includes for an agent-driven skill — where the invocation boundary is drawn, and whether sidechain turns belonging to other work land inside it.
Hand-verify one anxious:agent-wip invocation's turn set against the transcript structure (shapes and counts only, FR-3.5).
Either correct the attribution, or document precisely what the per-invocation figure means so the issue body cannot overstate it.

Split out of kotkan/claude-plugin-inference-arbitrage#11, whose primary defect (evidence-strength instability across windows) is fixed by the FR-4.5 gate in commit `dcec9ca`. This is the secondary finding from that issue, unaddressed. `bin/offload-scan` reported `anxious:agent-wip` at roughly 989k weighted tokens per invocation. That skill's actual job is on the order of ten MCP calls. A figure that large suggests invocation reconstruction or attribution is folding whole-session activity into a per-skill cost, rather than isolating the turns the skill was responsible for. Why it matters: `offload_value` and `share_of_audited_spend` are the numbers the filing threshold (spec §5.2.7) is applied to, and the numbers an issue body presents to a human as the cost of a candidate. If they are inflated by unrelated session work, the plugin over-reports value and files candidates that are not worth what it claims. Acceptance: - [ ] Establish what `offload-scan`'s invocation reconstruction actually includes for an agent-driven skill — where the invocation boundary is drawn, and whether sidechain turns belonging to other work land inside it. - [ ] Hand-verify one `anxious:agent-wip` invocation's turn set against the transcript structure (shapes and counts only, FR-3.5). - [ ] Either correct the attribution, or document precisely what the per-invocation figure means so the issue body cannot overstate it.

issuer-agent referenced this issue

2026-07-30 04:37:53 +03:00

Phase 7 — Acceptance #7

oleks added the agent-wip label 2026-07-30 05:22:11 +03:00

oleks commented

2026-07-30 05:35:34 +03:00

Investigation: root cause identified, and it's not what the acceptance criteria expected

Summary: The title's framing ("session-wide activity... not just the skill's own turns") is partially accurate but for a different mechanism than the acceptance criteria's bullet 1 anticipated. Two things are going on, and they cut in different directions:

1. Turn attribution itself is correct — not a bug

Checked bin/offload-scan lines 401-421: the mine check requires attr_plugin == plugin or attr_agent/attr_skill to start with "<plugin>:". Unattributed turns are dropped immediately (last_error = False; pending_reads = []; continue) and never enter the turns list. I hand-verified this against a real transcript:

$ grep -rl '"attributionSkill":"anxious:agent-wip"' ~/.claude/projects | head
-home-oleks-projects/cd01f9c2-0c38-4bd1-99ef-6145f8988dce.jsonl

That session (cd01f9c2-...) has 1385 total assistant turns spanning 2026-07-13T03:22 → 2026-07-16T17:06 (a multi-day, multi-purpose session touching anxious:delivery-flow, anxious:agent-wip, memory:save). Of those, exactly 31 distinct turns carry attributionSkill == "anxious:agent-wip" (deduped by (message.id, requestId), matching offload-scan's own dedup key). All 31 have isSidechain: False — i.e. inline main-thread turns, not a leaked subagent sidechain. So acceptance-criteria bullet 1's specific worry (sidechain turns from other work landing inside the group) is checked and ruled out: nothing session-wide is being wrongly counted as "mine."

2. The real mechanism: cache-read/write cost per turn is by design, and it's documented

design/spec.md §1.1 states this explicitly as the plugin's founding premise:

Every turn the model spends emitting a mechanical tool call costs approximately one full context re-read... it costs the same whether the step was hard or trivial.

and §5.2.1: mechanical_tokens = Σ over mechanical turns of (cache_read + cache_write + input + output) — "Because cache_read dominates, it is very nearly mechanical_turns × mean_context_size."

Real numbers from the same 31 turns confirm this is exactly what's happening: cache_read_input_tokens climbs from 129,437 to 138,866+ across the run (it's the whole accumulated session context, re-read on every turn, per Anthropic's cache-read semantics) — and cc-tokens' Rec.weighted (bin/cc-tokens line ~113) includes cr * CACHE_READ_MULT (0.1×) and cw1 * CACHE_WRITE_1H_MULT (2×) unconditionally, with no per-skill isolation. This is intentional and causally correct for offload_value: replacing agent-wip's LLM turns with a deterministic script genuinely eliminates those turns and their context-read cost, so counting the full per-turn context cost against the skill is not an over-attribution bug — it's the tool's entire reason to exist. No code change indicated here.

3. The actual unaddressed defect: `reconstruct_invocations` can merge distinct real invocations

bin/offload-scan lines 523-545 (unchanged since Phase 3, commit bdb62da — confirmed via git log --oneline -- bin/offload-scan, no later commit touches this function, including today's FR-4.5/audit fixes in dcec9ca/1777ee1/f52bb79/e3b67e9/6b634fd):

if current and (t.group != current[-1].group
                or t.ts - current[-1].ts > gap):

splits an "invocation" only on a group change or a wall-clock idle gap (idle_gap_minutes: 10, per references/signals-catalog.md) between consecutive already-attributed turns. Because unattributed turns never enter turns at all, this heuristic is blind to how much unrelated session activity happened in a gap — it only sees the clock. Concretely, in the same 31-turn sample:

gaps (minutes) between consecutive anxious:agent-wip turns:
[1.3, 0.5, 0.8, 1.0, 0.2, 0.4, 0.3, 1.2, 0.3, 0.5, 0.4, 0.4, 1.4, 1.0, 0.7,
 0.2, 0.1, 0.3, 0.4, 1.0, 1.8, 4.3, 0.6, 0.4, 0.5, 1.7, 0.4, 0.2, 1.4, 0.2]
max gap: 4.3 min (well under the 10-min threshold)

All 31 turns collapse into one reported "invocation." But the issue body itself says agent-wip's job is "on the order of ten MCP calls" — 31 turns is ~3× that, consistent with 3 separate real calls (e.g. claim #A, release #A, claim #B...) landing back-to-back in the same session and getting merged into a single reported invocation. This inflates the per-invocation headline figure (weighted_tokens / invocations, e.g. "989k weighted tokens per invocation" or the #15-body's "3,900,668 weighted tokens over 12 invocations") without being wrong about the total — the total group weighted_tokens sum is unaffected by invocation splitting. It also means mine_ngrams's min_invocations: 3 gate (repetition_factor) can undercount recurring patterns that span merged invocations (they show up as 1 invocation instead of 3), which would understate offload_value, not inflate it — so the merging defect and the "over-reports value" framing in the issue don't straightforwardly line up; the merging defect specifically corrupts the per-invocation number, not offload_value/share_of_audited_spend (which are computed on the whole group, ungrouped by invocation).

Verdict

Not stale/misdiagnosed as "unaddressed" — confirmed still unaddressed by git history.
Is misdiagnosed in its acceptance criteria's specific hypothesis — sidechain leakage isn't happening; attribution is turn-correct.
Real, narrow, fixable defect: reconstruct_invocations's idle-gap-only split can merge multiple genuine invocations of a skill into one when they occur close together in wall-clock time, which inflates the per-invocation figure quoted to humans in filed issue bodies (not offload_value/share_of_audited_spend, which remain correct).
The larger "cache-read cost dominates a skill's few turns" phenomenon that likely motivated this issue is by design, already documented in design/spec.md §1.1/§5.2.1, and correct for the tool's stated purpose.

Recommended fix (not applied in this pass — diagnosis only)

In bin/offload-scan, reconstruct_invocations(): either (a) tighten the split heuristic so a fresh run of the same skill/agent that starts a recognizable "entry" tool-signature after a shorter gap still splits (heuristic, imperfect, since transcripts carry no explicit invocation-start marker), or (b) at minimum, add a caveat to the emitted JSON (e.g. a note field) and to bin/audit-snapshot's issue-body template stating "reconstructed invocation count is a lower bound; back-to-back same-skill calls within idle_gap_minutes are merged." Given there is no ground-truth invocation boundary in the transcript data, (b) is the safer, always-correct fix; (a) is a heuristic improvement with false-negative/false-positive tradeoffs of its own.

Files: bin/offload-scan (reconstruct_invocations, ~L523-545), bin/audit-snapshot (issue-body template, ~L465).

— investigated per kotkan/claude-plugin-inference-arbitrage#12, repo confirmed clean/main throughout, no local Nix/build/deploy steps involved (n/a for this repo).

## Investigation: root cause identified, and it's not what the acceptance criteria expected **Summary**: The title's framing ("session-wide activity... not just the skill's own turns") is *partially* accurate but for a different mechanism than the acceptance criteria's bullet 1 anticipated. Two things are going on, and they cut in different directions: ### 1. Turn attribution itself is correct — not a bug Checked `bin/offload-scan` lines 401-421: the `mine` check requires `attr_plugin == plugin` or `attr_agent`/`attr_skill` to start with `"<plugin>:"`. Unattributed turns are dropped immediately (`last_error = False; pending_reads = []; continue`) and never enter the `turns` list. I hand-verified this against a real transcript: ``` $ grep -rl '"attributionSkill":"anxious:agent-wip"' ~/.claude/projects | head -home-oleks-projects/cd01f9c2-0c38-4bd1-99ef-6145f8988dce.jsonl ``` That session (`cd01f9c2-...`) has **1385 total assistant turns spanning 2026-07-13T03:22 → 2026-07-16T17:06** (a multi-day, multi-purpose session touching `anxious:delivery-flow`, `anxious:agent-wip`, `memory:save`). Of those, exactly **31 distinct turns** carry `attributionSkill == "anxious:agent-wip"` (deduped by `(message.id, requestId)`, matching offload-scan's own dedup key). All 31 have `isSidechain: False` — i.e. inline main-thread turns, not a leaked subagent sidechain. So acceptance-criteria bullet 1's specific worry (sidechain turns from other work landing inside the group) is **checked and ruled out**: nothing session-wide is being wrongly counted as "mine." ### 2. The real mechanism: cache-read/write cost per turn is by design, and it's documented `design/spec.md` §1.1 states this explicitly as the plugin's founding premise: > Every turn the model spends emitting a mechanical tool call costs approximately one full context re-read... it costs the same whether the step was hard or trivial. and §5.2.1: `mechanical_tokens = Σ over mechanical turns of (cache_read + cache_write + input + output)` — "Because `cache_read` dominates, it is very nearly `mechanical_turns × mean_context_size`." Real numbers from the same 31 turns confirm this is exactly what's happening: `cache_read_input_tokens` climbs from 129,437 to 138,866+ across the run (it's the whole accumulated session context, re-read on every turn, per Anthropic's cache-read semantics) — and `cc-tokens`' `Rec.weighted` (`bin/cc-tokens` line ~113) includes `cr * CACHE_READ_MULT` (0.1×) and `cw1 * CACHE_WRITE_1H_MULT` (2×) unconditionally, with no per-skill isolation. This is **intentional and causally correct for `offload_value`**: replacing agent-wip's LLM turns with a deterministic script genuinely eliminates those turns and their context-read cost, so counting the full per-turn context cost against the skill is not an over-attribution bug — it's the tool's entire reason to exist. **No code change indicated here.** ### 3. The actual unaddressed defect: `reconstruct_invocations` can merge distinct real invocations `bin/offload-scan` lines 523-545 (unchanged since Phase 3, commit `bdb62da` — confirmed via `git log --oneline -- bin/offload-scan`, no later commit touches this function, including today's FR-4.5/audit fixes in `dcec9ca`/`1777ee1`/`f52bb79`/`e3b67e9`/`6b634fd`): ```python if current and (t.group != current[-1].group or t.ts - current[-1].ts > gap): ``` splits an "invocation" only on a group change or a wall-clock idle gap (`idle_gap_minutes: 10`, per `references/signals-catalog.md`) **between consecutive already-attributed turns**. Because unattributed turns never enter `turns` at all, this heuristic is blind to how much unrelated session activity happened in a gap — it only sees the clock. Concretely, in the same 31-turn sample: ``` gaps (minutes) between consecutive anxious:agent-wip turns: [1.3, 0.5, 0.8, 1.0, 0.2, 0.4, 0.3, 1.2, 0.3, 0.5, 0.4, 0.4, 1.4, 1.0, 0.7, 0.2, 0.1, 0.3, 0.4, 1.0, 1.8, 4.3, 0.6, 0.4, 0.5, 1.7, 0.4, 0.2, 1.4, 0.2] max gap: 4.3 min (well under the 10-min threshold) ``` All 31 turns collapse into **one** reported "invocation." But the issue body itself says agent-wip's job is "on the order of ten MCP calls" — 31 turns is ~3× that, consistent with 3 separate real calls (e.g. claim #A, release #A, claim #B...) landing back-to-back in the same session and getting merged into a single reported invocation. This inflates the *per-invocation* headline figure (`weighted_tokens / invocations`, e.g. "989k weighted tokens per invocation" or the #15-body's "3,900,668 weighted tokens over 12 invocations") without being wrong about the *total* — the total group `weighted_tokens` sum is unaffected by invocation splitting. It also means `mine_ngrams`'s `min_invocations: 3` gate (repetition_factor) can *undercount* recurring patterns that span merged invocations (they show up as 1 invocation instead of 3), which would understate `offload_value`, not inflate it — so the merging defect and the "over-reports value" framing in the issue don't straightforwardly line up; the merging defect specifically corrupts the **per-invocation** number, not `offload_value`/`share_of_audited_spend` (which are computed on the whole group, ungrouped by invocation). ### Verdict - **Not stale/misdiagnosed as "unaddressed"** — confirmed still unaddressed by git history. - **Is misdiagnosed in its acceptance criteria's specific hypothesis** — sidechain leakage isn't happening; attribution is turn-correct. - **Real, narrow, fixable defect**: `reconstruct_invocations`'s idle-gap-only split can merge multiple genuine invocations of a skill into one when they occur close together in wall-clock time, which inflates the *per-invocation* figure quoted to humans in filed issue bodies (not `offload_value`/`share_of_audited_spend`, which remain correct). - The larger "cache-read cost dominates a skill's few turns" phenomenon that likely motivated this issue is **by design**, already documented in `design/spec.md` §1.1/§5.2.1, and correct for the tool's stated purpose. ### Recommended fix (not applied in this pass — diagnosis only) In `bin/offload-scan`, `reconstruct_invocations()`: either (a) tighten the split heuristic so a fresh run of the same skill/agent that starts a recognizable "entry" tool-signature after a shorter gap still splits (heuristic, imperfect, since transcripts carry no explicit invocation-start marker), or (b) at minimum, add a caveat to the emitted JSON (e.g. a `note` field) and to `bin/audit-snapshot`'s issue-body template stating "reconstructed invocation count is a lower bound; back-to-back same-skill calls within `idle_gap_minutes` are merged." Given there is no ground-truth invocation boundary in the transcript data, (b) is the safer, always-correct fix; (a) is a heuristic improvement with false-negative/false-positive tradeoffs of its own. Files: `bin/offload-scan` (`reconstruct_invocations`, ~L523-545), `bin/audit-snapshot` (issue-body template, ~L465). — investigated per kotkan/claude-plugin-inference-arbitrage#12, repo confirmed clean/main throughout, no local Nix/build/deploy steps involved (n/a for this repo).

oleks referenced this issue

2026-07-30 05:35:56 +03:00

Stability audit: anxious/agent-wip/release-policy-derivation measurement unstable across windows #15

oleks referenced this issue from a commit

2026-07-30 05:39:27 +03:00

offload-scan: caveat reconstructed invocation counts as a lower bound

oleks referenced a pull request that will close this issue

2026-07-30 05:39:54 +03:00

offload-scan: caveat reconstructed invocation counts as a lower bound #18

oleks closed this issue

2026-07-30 05:40:02 +03:00

oleks commented

2026-07-30 05:40:24 +03:00

Root cause

The issue title/hypothesis was "sidechain leakage" — investigated and ruled out. Turn attribution in bin/offload-scan's mine check (L401-421) is correct: only turns whose attributionPlugin/Agent/Skill match the target plugin enter turns; nothing session-wide leaks in as "mine". Hand-verified against a real 1385-turn transcript (~/.claude/projects/-home-oleks-projects/cd01f9c2-0c38-4bd1-99ef-6145f8988dce.jsonl): all 31 attributed anxious:agent-wip turns are isSidechain:False and correctly tagged.

The real, narrower defect is in reconstruct_invocations() (bin/offload-scan L523-545, unchanged since Phase 3 commit bdb62da): it splits invocations only on group-change or a >10-minute wall-clock gap between already-attributed turns. It has no signal for unrelated intervening activity (which never enters turns at all), so two genuinely separate same-skill calls landing within the idle-gap window of each other get merged into one reported invocation. In the real sample, 31 turns (~3x the skill's expected ~10-call step count) fell within max 4.3-minute gaps and were merged into a single reported invocation — consistent with 3 real calls being collapsed into 1.

This inflates the headline "weighted_tokens per invocation" figure quoted in filed issue bodies, without affecting the correctly-computed group totals (weighted_tokens, offload_value, share_of_audited_spend); it can actually understate offload_value via mine_ngrams' min_invocations:3 gate undercounting merged invocations.

Separately: cc-tokens counting full cache_read/cache_write per turn (observed cache_read climbing 129,437→138,866+ across the 31-turn sample) is intentional, per design/spec.md §1.1/§5.2.1 ("every turn costs approximately one full context re-read"), and is causally correct for offload_value. Not a bug.

Fix

No ground-truth invocation-start marker exists in transcripts, so tightening the split heuristic risks introducing new false splits/merges. Instead, this makes the lower-bound nature of the count explicit in the data:

bin/offload-scan: adds totals.invocation_caveat to the JSON output, naming the idle-gap threshold (10 min by default).
bin/audit-snapshot: threads invocation_caveat through build_snapshot's totals and renders it as a note bullet in the per-run issue-body template (render_run), right under the existing attribution-coverage caveat.

Evidence the fix works

Added tests/scan.test.sh check:

check "totals carry a lower-bound caveat naming the idle gap threshold" "
    'lower bound' in d['totals']['invocation_caveat']
    and '10 minutes' in d['totals']['invocation_caveat']
"

and tests/snapshot.test.sh check that the caveat propagates from a scan doc into snapshot totals and renders into the rendered .md audit page.

Full suite, actually run:

$ bash tests/run-all.sh
...
== ./scan.test.sh ==
...
ok: totals carry a lower-bound caveat naming the idle gap threshold
...
== ./snapshot.test.sh ==
...
== kotkan/claude-plugin-inference-arbitrage#12: scan's invocation_caveat reaches the summary page ==
  ok   invocation_caveat stored in totals = True
  ok   invocation caveat rendered in the audit page
...

All 7 suites (classify, filing, inventory, memory, scan, snapshot, stability) pass.

PR

kotkan/claude-plugin-inference-arbitrage#18 — merged into main (repo has no CI checks configured, so merge_when_checks_succeed:true merged immediately). plugin.json version bumped 0.6.2 → 0.6.3.

Note for #15 (separate open question about a real third measurement window) — that has its own plan already posted as a comment there; not addressed by this PR.

## Root cause The issue title/hypothesis was "sidechain leakage" — investigated and **ruled out**. Turn attribution in `bin/offload-scan`'s `mine` check (L401-421) is correct: only turns whose `attributionPlugin`/`Agent`/`Skill` match the target plugin enter `turns`; nothing session-wide leaks in as "mine". Hand-verified against a real 1385-turn transcript (`~/.claude/projects/-home-oleks-projects/cd01f9c2-0c38-4bd1-99ef-6145f8988dce.jsonl`): all 31 attributed `anxious:agent-wip` turns are `isSidechain:False` and correctly tagged. The **real, narrower defect** is in `reconstruct_invocations()` (`bin/offload-scan` L523-545, unchanged since Phase 3 commit `bdb62da`): it splits invocations only on group-change or a >10-minute wall-clock gap between *already-attributed* turns. It has no signal for unrelated intervening activity (which never enters `turns` at all), so two genuinely separate same-skill calls landing within the idle-gap window of each other get merged into one reported invocation. In the real sample, 31 turns (~3x the skill's expected ~10-call step count) fell within max 4.3-minute gaps and were merged into a single reported invocation — consistent with 3 real calls being collapsed into 1. This inflates the headline "weighted_tokens per invocation" figure quoted in filed issue bodies, without affecting the correctly-computed group totals (`weighted_tokens`, `offload_value`, `share_of_audited_spend`); it can actually *understate* `offload_value` via `mine_ngrams`' `min_invocations:3` gate undercounting merged invocations. Separately: `cc-tokens` counting full `cache_read`/`cache_write` per turn (observed cache_read climbing 129,437→138,866+ across the 31-turn sample) is **intentional**, per `design/spec.md` §1.1/§5.2.1 ("every turn costs approximately one full context re-read"), and is causally correct for `offload_value`. Not a bug. ## Fix No ground-truth invocation-start marker exists in transcripts, so tightening the split heuristic risks introducing new false splits/merges. Instead, this makes the lower-bound nature of the count explicit in the data: - `bin/offload-scan`: adds `totals.invocation_caveat` to the JSON output, naming the idle-gap threshold (10 min by default). - `bin/audit-snapshot`: threads `invocation_caveat` through `build_snapshot`'s totals and renders it as a note bullet in the per-run issue-body template (`render_run`), right under the existing attribution-coverage caveat. ## Evidence the fix works Added `tests/scan.test.sh` check: ``` check "totals carry a lower-bound caveat naming the idle gap threshold" " 'lower bound' in d['totals']['invocation_caveat'] and '10 minutes' in d['totals']['invocation_caveat'] " ``` and `tests/snapshot.test.sh` check that the caveat propagates from a scan doc into snapshot totals and renders into the rendered `.md` audit page. Full suite, actually run: ``` $ bash tests/run-all.sh ... == ./scan.test.sh == ... ok: totals carry a lower-bound caveat naming the idle gap threshold ... == ./snapshot.test.sh == ... == kotkan/claude-plugin-inference-arbitrage#12: scan's invocation_caveat reaches the summary page == ok invocation_caveat stored in totals = True ok invocation caveat rendered in the audit page ... ``` All 7 suites (classify, filing, inventory, memory, scan, snapshot, stability) pass. ## PR kotkan/claude-plugin-inference-arbitrage#18 — merged into `main` (repo has no CI checks configured, so `merge_when_checks_succeed:true` merged immediately). `plugin.json` version bumped 0.6.2 → 0.6.3. Note for #15 (separate open question about a real third measurement window) — that has its own plan already posted as a comment there; not addressed by this PR.

issuer-agent commented

2026-07-30 05:51:25 +03:00

Original hypothesis (sidechain leakage into turn attribution) was investigated and ruled out — attribution in bin/offload-scan is correct. Real, narrower defect found: reconstruct_invocations() merges genuinely separate same-skill invocations landing within its 10-minute idle-gap window, since it has no signal for intervening unrelated activity. Fixed via PR #18 (commit c894369, merged) — added an explicit invocation_caveat field threaded through offload-scan's output and audit-snapshot's issue-body template, so downstream filed issues state the invocation count is a lower bound instead of overstating confidence. Verified: full test suite passes on current origin/main.

Original hypothesis (sidechain leakage into turn attribution) was investigated and ruled out — attribution in bin/offload-scan is correct. Real, narrower defect found: reconstruct_invocations() merges genuinely separate same-skill invocations landing within its 10-minute idle-gap window, since it has no signal for intervening unrelated activity. Fixed via PR #18 (commit c894369, merged) — added an explicit invocation_caveat field threaded through offload-scan's output and audit-snapshot's issue-body template, so downstream filed issues state the invocation count is a lower bound instead of overstating confidence. Verified: full test suite passes on current origin/main.

admin removed the agent-wip label 2026-07-30 05:52:00 +03:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kotkan/claude-plugin-inference-arbitrage#12

offload-scan: per-invocation cost may include session-wide activity, not just the skill's own turns #12