claude-plugin-inference-arbitrage/design/rubric.md

rubric.md — where the boundary between script and intelligence belongs

"the problem here is to correctly understand a points where we need intelligence between script entities correctly positioned"

This is the crux of the whole plugin, so it gets its own document. It ships verbatim as references/boundary-rubric.md, and every snapshot records the rubric_version it was classified under.

The rubric has five parts, applied in order:

1. Five determinism tests — is this step a program at all?
2. Four positions — the answer is not binary, and the interesting cases are the two in the middle.
3. The positioning principle — given a pipeline, where exactly do you cut?
4. Two failure modes — named, with detection questions for each.
5. The falsifiability triple — the gate a candidate's reasoning must pass to be filed. 5b. The evidence-stability gate — the gate its evidence must pass.

---

1. The five determinism tests

A step may be offloaded to a script only if it passes all five. These are necessary conditions, not a score to be averaged — a step that fails one has failed, however well it does on the others.

T1 — Closed input domain

Can the inputs be enumerated or typed?

Paths, exit codes, JSON documents, git refs, file contents, timestamps, integers, a fixed set of enum values — closed. Free-form natural language, a user's stated intent, an arbitrary error message from an unknown tool, the contents of a web page — open.

The tell: if you can write the function's parameter list without using the type "whatever the human said," it is closed.

T2 — Stable output oracle

Is there one right answer, and can you assert it in a test?

If two competent engineers looking at the same input would produce the same output, and you can write assert f(x) == y, there is an oracle. If the answer is "it depends what we're trying to do here," there is not.

This is the strongest single test, because it is constructive. If you cannot write the assertion, you do not have a specification, and code without a specification is just inference with worse error messages. The inability to write the test is the finding.

T3 — Invariance

Same input, same output, every time — across runs, models, and days?

Then a script is strictly better: it is free, instant, and cannot drift.

The diagnostic is subtle and worth stating carefully. Ask: if this step gave a different answer on two identical inputs, would that be a bug or a feature? If it would be a bug, the step is a script and running it on inference is introducing nondeterminism into something that should not have any. If run-to-run variation is acceptable or desirable — a differently-worded explanation, a different but equally valid prioritization — that variance is the signature of judgment, and freezing it into a script destroys something real.

T4 — No semantic gap

Does the answer depend on anything not present in the inputs?

World knowledge, the user's goals, project history, what "good" means here, whether a name is apt, whether a description is honest, whether this counts as a bug — all live outside the inputs. A step that needs them cannot be a total function over its arguments, and any script pretending otherwise is encoding a guess as a fact.

T4 is the test that most often saves a step from being wrongly scripted.

T5 — Bounded branch factor

Is the decision tree enumerable?

A dozen cases, or a hundred, or a table — enumerable. "Any of the open-ended ways this could go wrong" — not. A step with an unbounded tail of special cases can still be partly scripted (handle the enumerable cases, escalate the tail), but that is position 3 below, not position 1.

Reading the results

Outcome	Position
All five pass	pure-script
T2 or T4 fails	pure-inference — stop, do not propose an offload
T1/T3/T5 fail but T2 and T4 hold	Decomposable — go to §3 and find the cut
Fails only because the inputs are unstructured	Often a digest problem, not a judgment problem — a script can structure the input even if it cannot make the decision

That last row matters more than it looks. A great many steps look like judgment only because nobody normalized the input.

---

2. The four positions

The binary "script or LLM" is the wrong frame, and forcing it is how both failure modes in §4 happen. There are four positions, and the two in the middle are where almost all the value is.

Position 1 — pure-script

Deterministic end to end. Offload wholly; the model does not participate.

Parse, glob, diff, count, aggregate, sort by a formula, validate against a schema, git plumbing, format a report from structured data.

Real instance: cc-tokens' entire JSONL pipeline. Nothing in the de-duplication of (message.id, requestId) or the application of a pricing table benefits from a model.

Position 2 — script-with-compiled-judgment

The judgment is real, but it is made once, at authoring time, by a human or a model, and then frozen into a rule, table, regex, threshold, or config. Runtime is a pure script.

Real instance: price-hunter's require/exclude/broken/confirm rules per hunt. Deciding that a hunt for a product must match the product noun and not the flavour word is judgment; applying that rule to ten thousand listings is not.

The characteristic risk is rule rot. The compiled judgment was correct for the world as it was at authoring time. When the world moves, the script keeps applying the old judgment, confidently and silently — and this is precisely why the tune-hunt skill exists. Any position-2 recommendation must therefore carry a staleness plan: how the rule is re-validated, and what signal says it has rotted. A position-2 offload without a re-validation story is not a saving, it is a deferred bug.

Position 3 — llm-over-script-digest — the dominant win

The script gathers, filters, normalizes, joins, and ranks. The model judges only the shortlist. This is the position the user is describing when they say "intelligence between script entities correctly positioned," and it is where most real candidates land.

Real instance: worktree-audit. The script enumerates worktrees and computes the provably-safe classification (including the remote-upstream subtlety that inference kept getting wrong); the model looks only at the must-reconcile pile and decides whether a given dirty tree is someone's in-progress work worth rescuing.

Prospective instance: anxious:steward. The deterministic axes of the label taxonomy are computed by script across all repos; the model is handed only the disagreements, the no-signal cases, and the weak-prior axes.

The whole design question in position 3 is where the cut goes — §3.

Position 4 — pure-inference

Irreducible. Do not touch, and say so explicitly in the report, because "this is correctly done by inference" is a finding.

Intent, naming, prose, trade-off explanation for this particular user, novel synthesis, disambiguation, adversarial or unstructured input, and — always — the authorization call on an irreversible action.

---

3. The positioning principle — where exactly to cut

Given a pipeline of steps, the boundary belongs at its narrowest waist: the point where, upstream, volume is high and every transformation is mechanical and total; at the point, the representation is small and typed; and downstream, the decisions are few and semantic.

Formally, you are choosing the cut that minimizes

tokens_crossing_the_boundary  ×  judgment_calls_downstream

subject to one hard constraint that dominates the optimization:

Semantic sufficiency. The digest crossing the boundary must contain everything the judgment rests on.

A cut that saves tokens but strips the evidence the decision depends on is a false offload, and it is worse than no offload: it doesn't slow the model down, it makes it wrong, and it hides the reason. The model downstream cannot know what the script filtered away, so it will answer confidently from a mutilated view.

The four positioning questions

Ask these in order about any proposed cut:

P1 — What crosses? Write the schema of the digest. If you cannot write it, you have not found the boundary — you have only found a place where you would like one to be.

P2 — What is lost? For each field the script drops, name the judgment that would have used it. If any judgment downstream would have used a dropped field, the cut is in the wrong place. Move it upstream.

P3 — What happens on surprise? When the script meets an input it was not designed for, does it escalate to the model, or does it silently return a wrong answer? A correct boundary always has an escalation path. A script whose only options are "handle it" and "handle it wrong" is at the wrong altitude. This is the same shape as the taxonomy's own axis-1 rule ending in "no signal — defer to human".

P4 — Who owns the irreversible action? The script may compute, propose, rank, and prepare. It must not authorize. Anything that deletes, deploys, pushes, sends, or spends stays behind a judgment gate — and this holds even when all five determinism tests pass, because determinism is not the same as safety. Blast radius is an independent axis and it overrides the rubric.

The compression heuristic

Good cuts show a large information-compression ratio with near-zero semantic loss. A script that turns 400 KB of git worktree list + git log + git status output into a 12-row typed table of (path, branch, dirty, has_remote_upstream, classification) is a good cut: three orders of magnitude of compression, and nothing a human would have used got dropped.

A script that turns the same input into "7 worktrees, 3 stale" is a bad cut at the same location — same compression, catastrophic semantic loss. The location was right; the schema was wrong. Most bad cuts are bad schemas at good locations, which is why P1 and P2 come before anything else.

---

4. The two failure modes

4a — Over-scripting (premature crystallization)

Freezing a judgment that legitimately varies. The system becomes fast, confident, and wrong, and — the real damage — nobody is watching anymore, because the step no longer produces output a human reads.

Detection questions:

• Does the correct answer depend on anything not in the inputs? (T4)
• Would the rule need to change when the world changes, and would anyone notice?
• Are you scripting the decision, or scripting the gathering for the decision? The second is almost always available and almost always safe.
• Is there a case where a competent human would look at the script's output and say "no, not this time"? If yes, that case must escalate, not be encoded.

4b — Under-scripting (inference as interpreter)

Using a model as a slow, expensive, nondeterministic interpreter for something a program does correctly. This is the status quo the plugin exists to find, and it is invisible precisely because it works — the answers are right, just paid for at ~one full context re-read per step.

Detection questions:

• Does the same tool sequence recur across invocations? (§5.2.2 of spec.md measures exactly this)
• Has the model ever got this step wrong in a way a script could not? A recorded error here is strong evidence — the @{u} bug in worktree classification is the canonical example.
• Is the model re-deriving, every invocation, a procedure that is already written down in the skill body as numbered steps?
• Would a competent engineer, shown this step, reach for a shell one-liner?

The asymmetry

These two errors are not equally bad, and the rubric is deliberately biased.

Under-scripting costs money. Over-scripting costs correctness, silently, and compounds. A missed candidate means a plugin stays somewhat expensive and the next audit will find it again. A wrong candidate means a script that gives a confident wrong answer for a year with no one watching. Hence the conservative filing threshold, and hence §5.

---

5. The falsifiability triple — the gate

Before any candidate may be filed as an issue (FR-4.2), the agent must be able to write, in the issue body itself, all three of:

(a) The function signature. Name, typed parameters, typed return. If it cannot be typed, T1 or T2 failed and this is not a script.

(b) Three input→expected-output pairs, one of which is an edge case. These are the test assertions. If they cannot be written, T2 failed — there is no oracle, and what looked like an algorithm is a judgment wearing an algorithm's clothes.

(c) The case where a human would overrule the script. The input where the script returns its defined answer and a competent human says "no, not here" — plus what the escalation path does about it.

(c) is the load-bearing element, and it is deliberately the hardest.

An agent that cannot name the overrule case has not understood the boundary; it has only noticed that a step looks repetitive. That is exactly the state in which over-scripting happens. So the rule is strict and mechanical:

No overrule case → downgrade to boundary question. Report it to the user as an open question about where the boundary belongs. Never file it.

The inverse is also informative. If the overrule case turns out to be common — if the human would overrule often — then the step is not a position-1 script at all. It is position 3, and the cut belongs earlier in the pipeline, at the gather/normalize stage rather than at the decision.

Boundary questions are a first-class output, not a failure. They are the plugin asking the user for the judgment the user is uniquely able to supply, which is precisely the behavior the rubric is trying to teach.

---

5b. The evidence-stability gate

The triple in §5 asks whether the reasoning survives scrutiny. This asks whether the evidence does.

A candidate's headline is a number and a word: measured, 12% of audited spend. Both come from a window — the range of transcripts the dynamic pass happened to read — and the window is chosen by whoever ran the audit, usually by typing a default. If the same candidate reads measured, 12% over thirty days and thin, 0% over seven, then the audit has not measured the candidate. It has measured the calendar, and reported the result as a property of the plugin.

The damage is specific, and it is §4a's damage arriving by a new route. A human reading the issue sees one window's figure, with no indication that another window exists or disagrees. They cannot discount what they are not shown. So the number reads as a fact, gets acted on as a fact, and the correction — if it ever comes — arrives only because someone independently re-ran the audit on a different range and happened to notice. That is not a control. It is luck with a review attached.

Hence the same shape of rule as §5:

Evidence that changes label across windows → downgrade to boundary question. Compare each filable candidate against the most recent prior window that recorded it. If the evidence-strength label differs, or if the two windows land on opposite sides of the filing threshold, the candidate is not filed.

Three boundaries make this a usable rule rather than a blanket refusal.

Movement is not instability. A candidate at 12% one window and 8% the next is stable: the human is told the same thing, and the difference is a trend, which the snapshot diff already reports as one. The gate fires only when the description a human receives would change — the label, or which side of the threshold the value falls on. Everything else is signal about the target, and suppressing it would throw away the trend the plugin exists to accumulate.

Absence of history is not instability. The first audit of a target has nothing to compare against, and a candidate seen for the first time has no earlier window. Those file normally, marked unchecked. Inventing an instability finding out of missing history would mean no plugin could ever be audited once, which contradicts the principle that a plugin with no measured history stays auditable on its definition alone.

A narrow window is not instability either — and this one was learned the hard way. The rule above, applied symmetrically, downgraded the very candidate it was written for. anxious/agent-wip/release-policy-derivation reads thin, 0% over 7 days and measured, 11.9% over 27 — which looks like the §5b failure until you measure two more windows: 0 → 2 → 12 → 15 invocations at 7, 14, 27 and 60 days. It runs about once every two days. The short window did not catch an unstable candidate; it failed to observe a real one, and the wider window kept raising the reading rather than diluting it. Punishing the candidate for that gets the artifact backwards: the window was the artifact.

So a weak reading only overturns a measured one when it was sampled over a comparably wide window — at least 90% of the width that produced the measured reading. Below that, the answer is neither "file" nor "boundary question" but a third, honest one: insufficient-window — cannot tell either way, come back with a wider window. Nothing is filed, nothing is self-reported, and the wider snapshot stays the standing comparison point so the next adequate run compares against it.

The parity is deliberately relative, not "at least N days": the adequate width is a fact about the candidate's invocation rate, which the auditor learns from the measurement rather than knowing in advance, so an absolute floor would be a guess about frequency dressed up as a threshold. And the carve-out is one-directional. A thin prior against a measured current is still a downgrade — there the earlier reading is the weak one — and so is a threshold crossing where both windows read measured, because a window that reached measured did observe the candidate; what moved was its share of spend, which is signal, not sampling.

The instability, when it fires, is a finding about the auditor. The target did not change; the tool described it two different ways. So it is reported against this plugin's own repo, not the target's — the same self-application discipline as §FR-8, applied to the plugin's own headline number.

---

6. Confidence scoring

boundary_confidence	Requires
high	All five determinism tests pass, position 1 or 3, falsifiability triple complete including the overrule case, escalation path defined
medium	Tests pass but the triple is partial, or position 2 without a staleness plan, or the digest schema (P1) is not yet writable
low	A repetition or cost signal exists, but T2 or T4 is in doubt

Only high is filed as an issue. medium and low are reported as boundary questions in the run summary and carried forward in snapshots, so that a later audit with more measured evidence can promote them.

Carrying them forward matters: a low candidate that recurs across four audits with growing cost is itself a signal worth surfacing, even if no single run could confidently classify it.