oleks
dcec9cabfb
Phase 7 filed anxious/agent-wip/release-policy-derivation as `measured` at 11.94% of audited spend on a 30-day window. The same candidate over 7 days came back `thin` at 0.00%. Only the date range differed. A human reading either issue body alone cannot know the other exists, which is rubric.md §4a's failure mode reached through the plugin's own headline number. That was caught by a human running a second skeptical review — luck, not a control. It is now mechanical, in the same register as FR-4.2's overrule-case gate: - bin/stability-classify matches each `file` candidate against the most recent prior snapshot recording it (FR-5.2 identity) and downgrades it to a boundary question if `measurement_strength` changed or the share crossed the 2% filing threshold between windows. Movement within a label is a trend, not an instability; no prior window is `unchecked`, not unstable (FR-2.4). - The downgrade is a finding against THIS repo, not the target's — the target did not change, the auditor described it two ways. Carries an `ia-stability` marker so a re-run comments rather than duplicating. - bin/ia_store.py factors the store and identity resolution out of audit-snapshot so both scripts hash signatures identically. A divergence there would make the gate match nothing and fail open. Documented as spec FR-4.5 and rubric §5b (shipped verbatim in references/). Wired into skills/offload-audit as step 4b, before filing-plan; steps 5-6 now consume its output rather than classified.json. S4 is retired in its old form — a human triage catching a bad candidate is not a repeatable test — and re-passed as: the gate catches the real release-policy-derivation case with no human in the loop. Asserted in tests/stability.test.sh case (f) against the verbatim two-window Phase 7 output. Closes kotkan/claude-plugin-inference-arbitrage#11 Suite: 178 assertions, exit 0.
151 lines
4.0 KiB
JSON
151 lines
4.0 KiB
JSON
{
|
|
"rubric_version": "1.0.0",
|
|
"target": "anxious (7-day window)",
|
|
"summary": {
|
|
"candidates": 5,
|
|
"high_confidence": 1,
|
|
"to_file": 1,
|
|
"boundary_questions": 2,
|
|
"no_offload": 1,
|
|
"drift_notes": 1,
|
|
"conclusion": "1 high-confidence candidate(s)"
|
|
},
|
|
"candidates": [
|
|
{
|
|
"candidate_id": "anxious/issuer-agent/consecutive-agent-fanout",
|
|
"skill": "anxious:issuer-agent",
|
|
"category": "usage",
|
|
"position": "pure-inference",
|
|
"determinism_tests": {
|
|
"T1": false,
|
|
"T2": false,
|
|
"T3": true,
|
|
"T4": false,
|
|
"T5": false
|
|
},
|
|
"falsifiability_triple": {
|
|
"signature": false,
|
|
"examples": false,
|
|
"overrule_case": false
|
|
},
|
|
"boundary_confidence": "low",
|
|
"measurement_strength": "measured",
|
|
"share_of_audited_spend": 0.75,
|
|
"offload_value": 3027155,
|
|
"verdict": "no-offload",
|
|
"reasons": [
|
|
"T2 or T4 in doubt \u2014 the oracle or the semantic gap is unresolved",
|
|
"position 4 \u2014 correctly done by inference. Reported as a finding, not a question."
|
|
]
|
|
},
|
|
{
|
|
"candidate_id": "anxious/agent-wip/release-policy-derivation",
|
|
"skill": "anxious:agent-wip",
|
|
"category": "usage",
|
|
"position": "llm-over-script-digest",
|
|
"determinism_tests": {
|
|
"T1": true,
|
|
"T2": true,
|
|
"T3": true,
|
|
"T4": true,
|
|
"T5": true
|
|
},
|
|
"falsifiability_triple": {
|
|
"signature": true,
|
|
"examples": true,
|
|
"overrule_case": true
|
|
},
|
|
"boundary_confidence": "high",
|
|
"measurement_strength": "thin",
|
|
"share_of_audited_spend": 0.0,
|
|
"offload_value": 0,
|
|
"verdict": "file",
|
|
"reasons": [
|
|
"all five determinism tests pass; triple complete; escalation path defined",
|
|
"filed on the static case \u2014 evidence is thin (0 invocations, 0 attributed turns), so the value claim is unproven and must be stated as such in the issue body"
|
|
]
|
|
},
|
|
{
|
|
"candidate_id": "anxious/steward-agent/taxonomy-axis-derivation",
|
|
"skill": "anxious:steward-agent",
|
|
"category": "usage",
|
|
"position": "llm-over-script-digest",
|
|
"determinism_tests": {
|
|
"T1": true,
|
|
"T2": true,
|
|
"T3": true,
|
|
"T4": true,
|
|
"T5": true
|
|
},
|
|
"falsifiability_triple": {
|
|
"signature": true,
|
|
"examples": false,
|
|
"overrule_case": true
|
|
},
|
|
"boundary_confidence": "medium",
|
|
"measurement_strength": "thin",
|
|
"share_of_audited_spend": 0.0,
|
|
"offload_value": 0,
|
|
"verdict": "boundary-question",
|
|
"reasons": [
|
|
"falsifiability triple incomplete: missing examples",
|
|
"confidence medium \u2014 reported, not filed"
|
|
]
|
|
},
|
|
{
|
|
"candidate_id": "anxious/pr-rot/unmeasured-script-verb-dominance",
|
|
"skill": "anxious:pr-rot",
|
|
"category": "static",
|
|
"position": "llm-over-script-digest",
|
|
"determinism_tests": {
|
|
"T1": true,
|
|
"T2": true,
|
|
"T3": true,
|
|
"T4": true,
|
|
"T5": true
|
|
},
|
|
"falsifiability_triple": {
|
|
"signature": true,
|
|
"examples": false,
|
|
"overrule_case": false
|
|
},
|
|
"boundary_confidence": "medium",
|
|
"measurement_strength": "thin",
|
|
"share_of_audited_spend": 0.0,
|
|
"offload_value": 0,
|
|
"verdict": "boundary-question",
|
|
"reasons": [
|
|
"falsifiability triple incomplete: missing examples, overrule_case",
|
|
"HARD GATE (FR-4.2): no overrule case \u2014 downgraded to a boundary question, never filed"
|
|
]
|
|
},
|
|
{
|
|
"candidate_id": "anxious/kanban-flow/hook-prose-drift",
|
|
"skill": "anxious:kanban-flow",
|
|
"category": "hook-prose-drift",
|
|
"position": "pure-inference",
|
|
"determinism_tests": {
|
|
"T1": false,
|
|
"T2": false,
|
|
"T3": true,
|
|
"T4": false,
|
|
"T5": true
|
|
},
|
|
"falsifiability_triple": {
|
|
"signature": false,
|
|
"examples": false,
|
|
"overrule_case": false
|
|
},
|
|
"boundary_confidence": "low",
|
|
"measurement_strength": "thin",
|
|
"share_of_audited_spend": 0.0,
|
|
"offload_value": 0,
|
|
"verdict": "drift-note",
|
|
"reasons": [
|
|
"T2 or T4 in doubt \u2014 the oracle or the semantic gap is unresolved",
|
|
"hook/prose drift \u2014 a restated configuration contract, not an un-scripted algorithm. The fix is to point at the hook file, not to write a script."
|
|
]
|
|
}
|
|
]
|
|
}
|