kotkan/claude-plugin-inference-arbitrage
3
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

token-offload: adhoc json reshaping (inference-arbitrage:offload-analyst) #24

New Issue
Closed
opened 2026-07-30 17:48:54 +03:00 by issuer-agent · 2 comments
issuer-agent commented 2026-07-30 17:48:54 +03:00
Copy Link
Copy Source

Measured cost. 849.5k weighted tokens over 26 invocations (2026-07-29T18:42:20Z -> 2026-07-30T14:40:05Z), 40% of this plugin's audited spend.

Evidence strength. measured — 26 invocations, 208 attributed turns.

Boundary position. llm-over-script-digest — confidence high.
Determinism tests: T1 ✓ T2 ✓ T3 ✓ T4 ✓ T5 ✓

Proposed signature.
candidate_digest(inv: PluginInventory, scan: OffloadScan) -> list[CandidateRow]

Examples.

  • verb_ratio=0.71 on a skill backed by a 606-LOC bin/ script -> flags=[] (script already exists, conjunction fails, not surfaced)
  • agent row: mtr=0.17, share_of_spend=0.40, judgment_density=0.034, dominant ngram recurrences=8 -> flags=['high-share-low-judgment-recurring-ngram'] (surfaced)
  • EDGE: agent row with the same mtr=0.17 and share_of_spend=0.40 but judgment_density=0.55 -> flags=[] (the judgment_density brake, already documented in this plugin's own skill body, suppresses the flag; this is the healthy case and must not be surfaced)

When a human would overrule this. A row crosses every mechanical threshold (share, mtr, read-amplification, recurring ngram) yet the recurring shape turns out to be this audit's own necessary two-pass discipline (read the static inventory, then the dynamic scan, then reshape both into a working set) rather than a genuinely wasteful loop. The digest can only see token/turn shape, not intent, so a human analyst still has to look at the actual command sequence before deciding candidate vs necessary-overhead. This is why the cut is position 3 (a digest that feeds analyst judgment), not position 1 (an auto-filed script).

Escalation path. The digest flags a row when it crosses the already-codified static thresholds (same constants boundary-classify already uses: MIN_INVOCATIONS, MIN_ATTRIBUTED_TURNS, MIN_SHARE_OF_SPEND) or shows a dominant recurring n-gram with judgment_density below a documented cutoff. A flagged row is not auto-graded: the analyst reads the actual ngram/tool-call shape, applies the judgment_density brake, and only rows the analyst confirms as genuine candidates proceed into a judgments.json entry for boundary-classify.

What crosses the boundary. {'row_kind': 'skill | agent', 'name': 'str', 'share_of_spend': 'float', 'mtr': 'float', 'judgment_density': 'float', 'read_amplification': 'float', 'retry_density': 'float', 'fanout_multiplier': 'float', 'dominant_ngram_recurrences': 'int | null', 'has_bin_script': 'bool', 'flags': 'list[str]'}

This is a proposal from an audit, not an accepted design. It is deliberately
falsifiable: if the overrule case above is common, the cut belongs earlier in
the pipeline than proposed, and this issue should be reshaped rather than
implemented. Closing this as wontfix is a useful outcome and is tracked as a
calibration signal.

Audit: Audits/inference-arbitrage/2026-07-30 on the kotkan/claude-plugin-inference-arbitrage wiki. Rubric v1.0.0.

<!-- ia-candidate: inference-arbitrage/offload-analyst/adhoc-json-reshaping --> **Measured cost.** 849.5k weighted tokens over 26 invocations (2026-07-29T18:42:20Z -> 2026-07-30T14:40:05Z), 40% of this plugin's audited spend. **Evidence strength.** `measured` — 26 invocations, 208 attributed turns. **Boundary position.** llm-over-script-digest — confidence high. Determinism tests: T1 ✓ T2 ✓ T3 ✓ T4 ✓ T5 ✓ **Proposed signature.** `candidate_digest(inv: PluginInventory, scan: OffloadScan) -> list[CandidateRow]` **Examples.** - verb_ratio=0.71 on a skill backed by a 606-LOC bin/ script -> flags=[] (script already exists, conjunction fails, not surfaced) - agent row: mtr=0.17, share_of_spend=0.40, judgment_density=0.034, dominant ngram recurrences=8 -> flags=['high-share-low-judgment-recurring-ngram'] (surfaced) - EDGE: agent row with the same mtr=0.17 and share_of_spend=0.40 but judgment_density=0.55 -> flags=[] (the judgment_density brake, already documented in this plugin's own skill body, suppresses the flag; this is the healthy case and must not be surfaced) **When a human would overrule this.** A row crosses every mechanical threshold (share, mtr, read-amplification, recurring ngram) yet the recurring shape turns out to be this audit's own necessary two-pass discipline (read the static inventory, then the dynamic scan, then reshape both into a working set) rather than a genuinely wasteful loop. The digest can only see token/turn shape, not intent, so a human analyst still has to look at the actual command sequence before deciding candidate vs necessary-overhead. This is why the cut is position 3 (a digest that feeds analyst judgment), not position 1 (an auto-filed script). **Escalation path.** The digest flags a row when it crosses the already-codified static thresholds (same constants boundary-classify already uses: MIN_INVOCATIONS, MIN_ATTRIBUTED_TURNS, MIN_SHARE_OF_SPEND) or shows a dominant recurring n-gram with judgment_density below a documented cutoff. A flagged row is not auto-graded: the analyst reads the actual ngram/tool-call shape, applies the judgment_density brake, and only rows the analyst confirms as genuine candidates proceed into a judgments.json entry for boundary-classify. **What crosses the boundary.** {'row_kind': 'skill | agent', 'name': 'str', 'share_of_spend': 'float', 'mtr': 'float', 'judgment_density': 'float', 'read_amplification': 'float', 'retry_density': 'float', 'fanout_multiplier': 'float', 'dominant_ngram_recurrences': 'int | null', 'has_bin_script': 'bool', 'flags': 'list[str]'} --- This is a proposal from an audit, not an accepted design. It is deliberately falsifiable: if the overrule case above is *common*, the cut belongs earlier in the pipeline than proposed, and this issue should be reshaped rather than implemented. Closing this as `wontfix` is a useful outcome and is tracked as a calibration signal. Audit: `Audits/inference-arbitrage/2026-07-30` on the kotkan/claude-plugin-inference-arbitrage wiki. Rubric v1.0.0.
oleks added the agent-wip label 2026-07-30 19:18:19 +03:00
oleks commented 2026-07-30 19:27:58 +03:00
Owner
Copy Link
Copy Source

Root cause confirmed as diagnosed. agents/offload-analyst.md §2 ("Gather candidates from three distinct sources") told the analyst to read inv.json and scan.json and reshape them into a candidate table by hand, every audit run — no bin/ script existed for that join. skills/offload-audit/SKILL.md step 3 had the same gap.

Fix: kotkan/claude-plugin-inference-arbitrage#28 (branch fix/issue-24-candidate-digest, PR open into main).

Added bin/candidate-digest inv.json scan.json --json, implementing candidate_digest(inv, scan) -> list[CandidateRow] exactly to the signature proposed in this issue. Design choices, with evidence:

  • Reused, not reinvented: imports MIN_INVOCATIONS/MIN_ATTRIBUTED_TURNS from bin/boundary-classify and MIN_SHARE_OF_SPEND from bin/ia_store.py — no new share/invocation constants.
  • judgment_density brake, actually implemented, not just checked: new JUDGMENT_DENSITY_BRAKE = 0.3 constant (no existing catalog threshold covered this axis). Verified against both worked examples from this issue's body: 
    $ ./bin/candidate-digest /tmp/t-inv.json /tmp/t-scan.json
# agent row, judgment_density=0.034, share=0.40, dominant ngram=8
"flags": ["high-share-low-judgment-recurring-ngram"]
# same mtr/share/ngram, judgment_density=0.55
"flags": []
    (full fixture in tests/candidate-digest.test.sh, examples 2 and 3/EDGE — both pass)
  • has_bin_script suppresses everything, matching the "verb_ratio=0.71 backed by a 606-LOC script → flags=[]" example: 
    == example 1: scripted skill, high mtr/share, has_bin_script suppresses everything ==
  ok   has_bin_script = True
  ok   flags = []
    Derived from plugin-inventory's command_blocks (a skill whose own documented procedure already invokes bin/...). For agent rows this is conservatively False — plugin-inventory's JSON carries no agent body/command-block text at all, only skills do, so the script genuinely cannot see this for agents. Documented inline as a real limitation, not a shortcut.
  • Position 3 preserved deliberately: flags is documented (in the script's own docstring and in both SKILL.md/offload-analyst.md) as advisory input to the analyst's own read of the actual n-gram/tool-call shape — never a verdict. bin/boundary-classify remains the only place a verdict is computed; candidate-digest has no filing gate of its own and is never called from the filing chain.

Real-data verification (not just synthetic fixtures), against a matched inv.json/scan.json pair from a prior anxious audit left in /tmp:

$ ./bin/candidate-digest /tmp/anxious-inventory.json /tmp/anxious-scan.json
[
  {"row_kind": "agent", "name": "anxious:issuer-agent", "share_of_spend": 0.431, ...,
   "judgment_density": 0.053, "dominant_ngram_recurrences": 108, "has_bin_script": false,
   "flags": ["high-share-low-judgment-recurring-ngram"]},
  ...
  {"row_kind": "agent", "name": "repo-worker", "share_of_spend": 0.053, ...,
   "dominant_ngram_recurrences": null, "flags": []},
  ...
]

Output is correctly sorted by share_of_spend, flags only fire on rows that clear every gate, has_bin_script is false everywhere (no anxious skill's command blocks reference bin/, which matches its actual layout).

Wiring: skills/offload-audit/SKILL.md step 3 and agents/offload-analyst.md §2 both now call bin/candidate-digest and explicitly state the flag semantics (advisory, brake-suppressed, never a filing trigger).

Tests: tests/candidate-digest.test.sh (new, auto-discovered by tests/run-all.sh) covers all three worked examples plus a below-MIN_SHARE_OF_SPEND case and a full CandidateRow schema check. Full suite run:

$ bash tests/run-all.sh; echo "EXIT: $?"
EXIT: 0

No FAIL/MISSING/Traceback anywhere in the log.

Issue body assessment: not stale — read correctly, matches current repo state (agents/offload-analyst.md, skills/offload-audit/SKILL.md, bin/boundary-classify, bin/ia_store.py all confirmed by direct read before implementing). No misdiagnosis found.

PR: kotkan/claude-plugin-inference-arbitrage#28 (not merged — leaving that to whoever reviews).

**Root cause confirmed as diagnosed.** `agents/offload-analyst.md` §2 ("Gather candidates from three distinct sources") told the analyst to read `inv.json` and `scan.json` and reshape them into a candidate table by hand, every audit run — no `bin/` script existed for that join. `skills/offload-audit/SKILL.md` step 3 had the same gap. **Fix: kotkan/claude-plugin-inference-arbitrage#28** (branch `fix/issue-24-candidate-digest`, PR open into `main`). Added `bin/candidate-digest inv.json scan.json --json`, implementing `candidate_digest(inv, scan) -> list[CandidateRow]` exactly to the signature proposed in this issue. Design choices, with evidence: - **Reused, not reinvented**: imports `MIN_INVOCATIONS`/`MIN_ATTRIBUTED_TURNS` from `bin/boundary-classify` and `MIN_SHARE_OF_SPEND` from `bin/ia_store.py` — no new share/invocation constants. - **judgment_density brake, actually implemented, not just checked**: new `JUDGMENT_DENSITY_BRAKE = 0.3` constant (no existing catalog threshold covered this axis). Verified against both worked examples from this issue's body: ``` $ ./bin/candidate-digest /tmp/t-inv.json /tmp/t-scan.json # agent row, judgment_density=0.034, share=0.40, dominant ngram=8 "flags": ["high-share-low-judgment-recurring-ngram"] # same mtr/share/ngram, judgment_density=0.55 "flags": [] ``` (full fixture in `tests/candidate-digest.test.sh`, examples 2 and 3/EDGE — both pass) - **has_bin_script suppresses everything**, matching the "verb_ratio=0.71 backed by a 606-LOC script → flags=[]" example: ``` == example 1: scripted skill, high mtr/share, has_bin_script suppresses everything == ok has_bin_script = True ok flags = [] ``` Derived from `plugin-inventory`'s `command_blocks` (a skill whose own documented procedure already invokes `bin/...`). For agent rows this is conservatively `False` — `plugin-inventory`'s JSON carries no agent body/command-block text at all, only skills do, so the script genuinely cannot see this for agents. Documented inline as a real limitation, not a shortcut. - **Position 3 preserved deliberately**: `flags` is documented (in the script's own docstring and in both `SKILL.md`/`offload-analyst.md`) as advisory input to the analyst's own read of the actual n-gram/tool-call shape — never a verdict. `bin/boundary-classify` remains the only place a verdict is computed; `candidate-digest` has no filing gate of its own and is never called from the filing chain. **Real-data verification** (not just synthetic fixtures), against a matched `inv.json`/`scan.json` pair from a prior `anxious` audit left in `/tmp`: ``` $ ./bin/candidate-digest /tmp/anxious-inventory.json /tmp/anxious-scan.json [ {"row_kind": "agent", "name": "anxious:issuer-agent", "share_of_spend": 0.431, ..., "judgment_density": 0.053, "dominant_ngram_recurrences": 108, "has_bin_script": false, "flags": ["high-share-low-judgment-recurring-ngram"]}, ... {"row_kind": "agent", "name": "repo-worker", "share_of_spend": 0.053, ..., "dominant_ngram_recurrences": null, "flags": []}, ... ] ``` Output is correctly sorted by `share_of_spend`, flags only fire on rows that clear every gate, `has_bin_script` is `false` everywhere (no `anxious` skill's command blocks reference `bin/`, which matches its actual layout). **Wiring**: `skills/offload-audit/SKILL.md` step 3 and `agents/offload-analyst.md` §2 both now call `bin/candidate-digest` and explicitly state the flag semantics (advisory, brake-suppressed, never a filing trigger). **Tests**: `tests/candidate-digest.test.sh` (new, auto-discovered by `tests/run-all.sh`) covers all three worked examples plus a below-`MIN_SHARE_OF_SPEND` case and a full `CandidateRow` schema check. Full suite run: ``` $ bash tests/run-all.sh; echo "EXIT: $?" EXIT: 0 ``` No `FAIL`/`MISSING`/`Traceback` anywhere in the log. **Issue body assessment**: not stale — read correctly, matches current repo state (`agents/offload-analyst.md`, `skills/offload-audit/SKILL.md`, `bin/boundary-classify`, `bin/ia_store.py` all confirmed by direct read before implementing). No misdiagnosis found. PR: kotkan/claude-plugin-inference-arbitrage#28 (not merged — leaving that to whoever reviews).
oleks removed the agent-wip label 2026-07-30 19:31:08 +03:00
oleks closed this issue 2026-07-30 19:57:32 +03:00
oleks commented 2026-07-30 19:57:46 +03:00
Owner
Copy Link
Copy Source

Fixed and merged via PR #28, commit 0482278 (merged onto main after a trivial version-field conflict with #27, resolved as 0.8.1). Closing.

Fixed and merged via PR #28, commit 0482278 (merged onto main after a trivial version-field conflict with #27, resolved as 0.8.1). Closing.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
activity/automate
agent-wip
area/agent-behavior
Agent behavior and delegation logic
 bug
cost
domain/agents
Agent orchestration and multi-agent systems
 efficiency
infra-misconfig
kind/capability-gap
kind/defect
A defect or bug that needs fixing
 token-offload
A step paying for inference where a deterministic script would do (inference-arbitrage audit)
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
admin anton oleks
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: kotkan/claude-plugin-inference-arbitrage#24
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?