kotkan/claude-plugin-inference-arbitrage
3
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

offload-scan: caveat reconstructed invocation counts as a lower bound #18

Merged
oleks merged 1 commits from fix/issue-12-invocation-caveat into main 2026-07-30 05:40:01 +03:00
Conversation 0 Commits 1 Files Changed 5
+55 -1
oleks commented 2026-07-30 05:39:54 +03:00
Owner
Copy Link
Copy Source

Fixes kotkan/claude-plugin-inference-arbitrage#12

Root cause

Investigated the filed hypothesis (sidechain/session-wide activity leaking into per-invocation figures) and ruled it out: turn attribution in bin/offload-scan's mine check (L401-421) is correct — only turns whose attributionPlugin/Agent/Skill match the target plugin enter turns. Hand-verified against a real 1385-turn transcript; all 31 attributed anxious:agent-wip turns are isSidechain:False and correctly tagged.

The real, narrower defect is in reconstruct_invocations() (L523-545, unchanged since Phase 3): it splits invocations only on group-change or a >10-minute wall-clock gap between already-attributed turns. It has no signal for unrelated intervening activity (which never enters turns at all), so two genuinely separate same-skill calls that happen to land within the idle-gap window of each other get merged into one reported invocation. In the real sample, 31 turns (~3x the skill's expected step count) fell within max 4.3-minute gaps and were merged into a single reported invocation, consistent with 3 real calls being collapsed into 1.

This inflates the headline "weighted_tokens per invocation" figure quoted in filed issue bodies, without affecting the correctly-computed group totals (weighted_tokens, offload_value, share_of_audited_spend).

Note: the other component that makes per-invocation figures look large — cc-tokens counting full cache_read/cache_write per turn — is intentional per design/spec.md §1.1/§5.2.1 and not a bug.

Fix

Rather than add a new splitting heuristic (no ground-truth invocation-start marker exists in transcripts, so any heuristic risks new false splits/merges), this adds an explicit caveat to the emitted data so downstream consumers don't overstate confidence in the per-invocation figure:

  • bin/offload-scan: totals.invocation_caveat in the JSON output, naming the idle-gap threshold.
  • bin/audit-snapshot: threads invocation_caveat through build_snapshot's totals and renders it as a note bullet in the per-run issue-body template (render_run).

Tests

  • tests/scan.test.sh: asserts totals.invocation_caveat mentions "lower bound" and the configured idle-gap ("10 minutes").
  • tests/snapshot.test.sh: asserts the caveat propagates from a scan doc into snapshot totals and renders into the audit page markdown.

Full suite run: for f in tests/*.test.sh; do bash "$f"; done — all suites pass (classify, filing, inventory, memory, scan, snapshot, stability).

Version bumped 0.6.2 -> 0.6.3 (patch, per plugin-version convention).

Fixes kotkan/claude-plugin-inference-arbitrage#12 ## Root cause Investigated the filed hypothesis (sidechain/session-wide activity leaking into per-invocation figures) and ruled it out: turn attribution in `bin/offload-scan`'s `mine` check (L401-421) is correct — only turns whose `attributionPlugin`/`Agent`/`Skill` match the target plugin enter `turns`. Hand-verified against a real 1385-turn transcript; all 31 attributed `anxious:agent-wip` turns are `isSidechain:False` and correctly tagged. The real, narrower defect is in `reconstruct_invocations()` (L523-545, unchanged since Phase 3): it splits invocations only on group-change or a >10-minute wall-clock gap between *already-attributed* turns. It has no signal for unrelated intervening activity (which never enters `turns` at all), so two genuinely separate same-skill calls that happen to land within the idle-gap window of each other get merged into one reported invocation. In the real sample, 31 turns (~3x the skill's expected step count) fell within max 4.3-minute gaps and were merged into a single reported invocation, consistent with 3 real calls being collapsed into 1. This inflates the headline "weighted_tokens per invocation" figure quoted in filed issue bodies, without affecting the correctly-computed group totals (`weighted_tokens`, `offload_value`, `share_of_audited_spend`). Note: the other component that makes per-invocation figures look large — `cc-tokens` counting full `cache_read`/`cache_write` per turn — is intentional per `design/spec.md` §1.1/§5.2.1 and not a bug. ## Fix Rather than add a new splitting heuristic (no ground-truth invocation-start marker exists in transcripts, so any heuristic risks new false splits/merges), this adds an explicit caveat to the emitted data so downstream consumers don't overstate confidence in the per-invocation figure: - `bin/offload-scan`: `totals.invocation_caveat` in the JSON output, naming the idle-gap threshold. - `bin/audit-snapshot`: threads `invocation_caveat` through `build_snapshot`'s totals and renders it as a note bullet in the per-run issue-body template (`render_run`). ## Tests - `tests/scan.test.sh`: asserts `totals.invocation_caveat` mentions "lower bound" and the configured idle-gap ("10 minutes"). - `tests/snapshot.test.sh`: asserts the caveat propagates from a scan doc into snapshot totals and renders into the audit page markdown. Full suite run: `for f in tests/*.test.sh; do bash "$f"; done` — all suites pass (classify, filing, inventory, memory, scan, snapshot, stability). Version bumped 0.6.2 -> 0.6.3 (patch, per plugin-version convention).
oleks added 1 commit 2026-07-30 05:39:54 +03:00
offload-scan: caveat reconstructed invocation counts as a lower bound
ci/woodpecker/push/test Pipeline failed
Details
ci/woodpecker/pr/test Pipeline failed
Details
c8943690e0 
reconstruct_invocations() only sees already-attributed turns and splits on a
wall-clock idle gap (10 min default); it has no signal for unrelated
activity between two genuinely separate same-skill calls, so back-to-back
invocations closer together than the gap collapse into one reported
invocation. This inflates the headline per-invocation figures quoted in
filed issue bodies without affecting group totals (weighted_tokens,
offload_value, share_of_audited_spend).

Sidechain-leakage was investigated as the suspected root cause and ruled
out: turn attribution (the `mine` check) is correct, nothing session-wide
enters `turns`. The real, narrower defect is invocation-count inflation
from idle-gap merging.

Adds `totals.invocation_caveat` to offload-scan's JSON output, threads it
through audit-snapshot's snapshot totals, and renders it as a note in the
issue-body template so downstream issue bodies don't overstate confidence
in the per-invocation figure.

Fixes kotkan/claude-plugin-inference-arbitrage#12
oleks scheduled this pull request to auto merge when all checks succeed 2026-07-30 05:39:59 +03:00
oleks merged commit 58b14cbde1 into main 2026-07-30 05:40:01 +03:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
activity/automate
agent-wip
area/agent-behavior
Agent behavior and delegation logic
 bug
cost
domain/agents
Agent orchestration and multi-agent systems
 efficiency
infra-misconfig
kind/capability-gap
kind/defect
A defect or bug that needs fixing
 token-offload
A step paying for inference where a deterministic script would do (inference-arbitrage audit)
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
admin anton oleks
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: kotkan/claude-plugin-inference-arbitrage#18
Delete Branch "fix/issue-12-invocation-caveat"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?