Table of Contents
- rubric.md — where the boundary between script and intelligence belongs
- 1. The five determinism tests
- T1 — Closed input domain
- T2 — Stable output oracle
- T3 — Invariance
- T4 — No semantic gap
- T5 — Bounded branch factor
- Reading the results
- 2. The four positions
- Position 1 — pure-script
- Position 2 — script-with-compiled-judgment
- Position 3 — llm-over-script-digest — the dominant win
- Position 4 — pure-inference
- 3. The positioning principle — where exactly to cut
- 4. The two failure modes
- 4a — Over-scripting (premature crystallization)
- 4b — Under-scripting (inference as interpreter)
- The asymmetry
- 5. The falsifiability triple — the gate
- 6. Confidence scoring
Wiki copy of design/rubric.md, v1.0.0
rubric.md — where the boundary between script and intelligence belongs
"the problem here is to correctly understand a points where we need intelligence between script entities correctly positioned"
This is the crux of the whole plugin, so it gets its own document. It ships
verbatim as references/boundary-rubric.md, and every snapshot records the
rubric_version it was classified under.
The rubric has five parts, applied in order:
- Five determinism tests — is this step a program at all?
- Four positions — the answer is not binary, and the interesting cases are the two in the middle.
- The positioning principle — given a pipeline, where exactly do you cut?
- Two failure modes — named, with detection questions for each.
- The falsifiability triple — the gate a candidate must pass to be filed.
1. The five determinism tests
A step may be offloaded to a script only if it passes all five. These are necessary conditions, not a score to be averaged — a step that fails one has failed, however well it does on the others.
T1 — Closed input domain
Can the inputs be enumerated or typed?
Paths, exit codes, JSON documents, git refs, file contents, timestamps, integers, a fixed set of enum values — closed. Free-form natural language, a user's stated intent, an arbitrary error message from an unknown tool, the contents of a web page — open.
The tell: if you can write the function's parameter list without using the type "whatever the human said," it is closed.
T2 — Stable output oracle
Is there one right answer, and can you assert it in a test?
If two competent engineers looking at the same input would produce the same
output, and you can write assert f(x) == y, there is an oracle. If the answer
is "it depends what we're trying to do here," there is not.
This is the strongest single test, because it is constructive. If you cannot write the assertion, you do not have a specification, and code without a specification is just inference with worse error messages. The inability to write the test is the finding.
T3 — Invariance
Same input, same output, every time — across runs, models, and days?
Then a script is strictly better: it is free, instant, and cannot drift.
The diagnostic is subtle and worth stating carefully. Ask: if this step gave a different answer on two identical inputs, would that be a bug or a feature? If it would be a bug, the step is a script and running it on inference is introducing nondeterminism into something that should not have any. If run-to-run variation is acceptable or desirable — a differently-worded explanation, a different but equally valid prioritization — that variance is the signature of judgment, and freezing it into a script destroys something real.
T4 — No semantic gap
Does the answer depend on anything not present in the inputs?
World knowledge, the user's goals, project history, what "good" means here, whether a name is apt, whether a description is honest, whether this counts as a bug — all live outside the inputs. A step that needs them cannot be a total function over its arguments, and any script pretending otherwise is encoding a guess as a fact.
T4 is the test that most often saves a step from being wrongly scripted.
T5 — Bounded branch factor
Is the decision tree enumerable?
A dozen cases, or a hundred, or a table — enumerable. "Any of the open-ended ways this could go wrong" — not. A step with an unbounded tail of special cases can still be partly scripted (handle the enumerable cases, escalate the tail), but that is position 3 below, not position 1.
Reading the results
| Outcome | Position |
|---|---|
| All five pass | pure-script |
| T2 or T4 fails | pure-inference — stop, do not propose an offload |
| T1/T3/T5 fail but T2 and T4 hold | Decomposable — go to §3 and find the cut |
| Fails only because the inputs are unstructured | Often a digest problem, not a judgment problem — a script can structure the input even if it cannot make the decision |
That last row matters more than it looks. A great many steps look like judgment only because nobody normalized the input.
2. The four positions
The binary "script or LLM" is the wrong frame, and forcing it is how both failure modes in §4 happen. There are four positions, and the two in the middle are where almost all the value is.
Position 1 — pure-script
Deterministic end to end. Offload wholly; the model does not participate.
Parse, glob, diff, count, aggregate, sort by a formula, validate against a schema, git plumbing, format a report from structured data.
Real instance: cc-tokens' entire JSONL pipeline. Nothing in the
de-duplication of (message.id, requestId) or the application of a pricing table
benefits from a model.
Position 2 — script-with-compiled-judgment
The judgment is real, but it is made once, at authoring time, by a human or a model, and then frozen into a rule, table, regex, threshold, or config. Runtime is a pure script.
Real instance: price-hunter's require/exclude/broken/confirm rules per
hunt. Deciding that a hunt for a product must match the product noun and not the
flavour word is judgment; applying that rule to ten thousand listings is not.
The characteristic risk is rule rot. The compiled judgment was correct for the
world as it was at authoring time. When the world moves, the script keeps applying
the old judgment, confidently and silently — and this is precisely why the
tune-hunt skill exists. Any position-2 recommendation must therefore carry a
staleness plan: how the rule is re-validated, and what signal says it has rotted.
A position-2 offload without a re-validation story is not a saving, it is a
deferred bug.
Position 3 — llm-over-script-digest — the dominant win
The script gathers, filters, normalizes, joins, and ranks. The model judges only the shortlist. This is the position the user is describing when they say "intelligence between script entities correctly positioned," and it is where most real candidates land.
Real instance: worktree-audit. The script enumerates worktrees and computes the
provably-safe classification (including the remote-upstream subtlety that
inference kept getting wrong); the model looks only at the must-reconcile pile
and decides whether a given dirty tree is someone's in-progress work worth
rescuing.
Prospective instance: anxious:steward. The deterministic axes of the label
taxonomy are computed by script across all repos; the model is handed only the
disagreements, the no-signal cases, and the weak-prior axes.
The whole design question in position 3 is where the cut goes — §3.
Position 4 — pure-inference
Irreducible. Do not touch, and say so explicitly in the report, because "this is correctly done by inference" is a finding.
Intent, naming, prose, trade-off explanation for this particular user, novel synthesis, disambiguation, adversarial or unstructured input, and — always — the authorization call on an irreversible action.
3. The positioning principle — where exactly to cut
Given a pipeline of steps, the boundary belongs at its narrowest waist: the point where, upstream, volume is high and every transformation is mechanical and total; at the point, the representation is small and typed; and downstream, the decisions are few and semantic.
Formally, you are choosing the cut that minimizes
tokens_crossing_the_boundary × judgment_calls_downstream
subject to one hard constraint that dominates the optimization:
Semantic sufficiency. The digest crossing the boundary must contain everything the judgment rests on.
A cut that saves tokens but strips the evidence the decision depends on is a false offload, and it is worse than no offload: it doesn't slow the model down, it makes it wrong, and it hides the reason. The model downstream cannot know what the script filtered away, so it will answer confidently from a mutilated view.
The four positioning questions
Ask these in order about any proposed cut:
P1 — What crosses? Write the schema of the digest. If you cannot write it, you have not found the boundary — you have only found a place where you would like one to be.
P2 — What is lost? For each field the script drops, name the judgment that would have used it. If any judgment downstream would have used a dropped field, the cut is in the wrong place. Move it upstream.
P3 — What happens on surprise? When the script meets an input it was not designed for, does it escalate to the model, or does it silently return a wrong answer? A correct boundary always has an escalation path. A script whose only options are "handle it" and "handle it wrong" is at the wrong altitude. This is the same shape as the taxonomy's own axis-1 rule ending in "no signal — defer to human".
P4 — Who owns the irreversible action? The script may compute, propose, rank, and prepare. It must not authorize. Anything that deletes, deploys, pushes, sends, or spends stays behind a judgment gate — and this holds even when all five determinism tests pass, because determinism is not the same as safety. Blast radius is an independent axis and it overrides the rubric.
The compression heuristic
Good cuts show a large information-compression ratio with near-zero semantic loss.
A script that turns 400 KB of git worktree list + git log + git status
output into a 12-row typed table of (path, branch, dirty, has_remote_upstream, classification) is a good cut: three orders of magnitude of compression, and
nothing a human would have used got dropped.
A script that turns the same input into "7 worktrees, 3 stale" is a bad cut at
the same location — same compression, catastrophic semantic loss. The location was
right; the schema was wrong. Most bad cuts are bad schemas at good locations,
which is why P1 and P2 come before anything else.
4. The two failure modes
4a — Over-scripting (premature crystallization)
Freezing a judgment that legitimately varies. The system becomes fast, confident, and wrong, and — the real damage — nobody is watching anymore, because the step no longer produces output a human reads.
Detection questions:
- Does the correct answer depend on anything not in the inputs? (T4)
- Would the rule need to change when the world changes, and would anyone notice?
- Are you scripting the decision, or scripting the gathering for the decision? The second is almost always available and almost always safe.
- Is there a case where a competent human would look at the script's output and say "no, not this time"? If yes, that case must escalate, not be encoded.
4b — Under-scripting (inference as interpreter)
Using a model as a slow, expensive, nondeterministic interpreter for something a program does correctly. This is the status quo the plugin exists to find, and it is invisible precisely because it works — the answers are right, just paid for at ~one full context re-read per step.
Detection questions:
- Does the same tool sequence recur across invocations? (§5.2.2 of
spec.mdmeasures exactly this) - Has the model ever got this step wrong in a way a script could not? A
recorded error here is strong evidence — the
@{u}bug in worktree classification is the canonical example. - Is the model re-deriving, every invocation, a procedure that is already written down in the skill body as numbered steps?
- Would a competent engineer, shown this step, reach for a shell one-liner?
The asymmetry
These two errors are not equally bad, and the rubric is deliberately biased.
Under-scripting costs money. Over-scripting costs correctness, silently, and compounds. A missed candidate means a plugin stays somewhat expensive and the next audit will find it again. A wrong candidate means a script that gives a confident wrong answer for a year with no one watching. Hence the conservative filing threshold, and hence §5.
5. The falsifiability triple — the gate
Before any candidate may be filed as an issue (FR-4.2), the agent must be able to write, in the issue body itself, all three of:
(a) The function signature. Name, typed parameters, typed return. If it cannot be typed, T1 or T2 failed and this is not a script.
(b) Three input→expected-output pairs, one of which is an edge case. These are the test assertions. If they cannot be written, T2 failed — there is no oracle, and what looked like an algorithm is a judgment wearing an algorithm's clothes.
(c) The case where a human would overrule the script. The input where the script returns its defined answer and a competent human says "no, not here" — plus what the escalation path does about it.
(c) is the load-bearing element, and it is deliberately the hardest.
An agent that cannot name the overrule case has not understood the boundary; it has only noticed that a step looks repetitive. That is exactly the state in which over-scripting happens. So the rule is strict and mechanical:
No overrule case → downgrade to boundary question. Report it to the user as an open question about where the boundary belongs. Never file it.
The inverse is also informative. If the overrule case turns out to be common — if the human would overrule often — then the step is not a position-1 script at all. It is position 3, and the cut belongs earlier in the pipeline, at the gather/normalize stage rather than at the decision.
Boundary questions are a first-class output, not a failure. They are the plugin asking the user for the judgment the user is uniquely able to supply, which is precisely the behavior the rubric is trying to teach.
6. Confidence scoring
boundary_confidence |
Requires |
|---|---|
high |
All five determinism tests pass, position 1 or 3, falsifiability triple complete including the overrule case, escalation path defined |
medium |
Tests pass but the triple is partial, or position 2 without a staleness plan, or the digest schema (P1) is not yet writable |
low |
A repetition or cost signal exists, but T2 or T4 is in doubt |
Only high is filed as an issue. medium and low are reported as boundary
questions in the run summary and carried forward in snapshots, so that a later
audit with more measured evidence can promote them.
Carrying them forward matters: a low candidate that recurs across four audits
with growing cost is itself a signal worth surfacing, even if no single run could
confidently classify it.