oleks
dcec9cabfb
Phase 7 filed anxious/agent-wip/release-policy-derivation as `measured` at 11.94% of audited spend on a 30-day window. The same candidate over 7 days came back `thin` at 0.00%. Only the date range differed. A human reading either issue body alone cannot know the other exists, which is rubric.md §4a's failure mode reached through the plugin's own headline number. That was caught by a human running a second skeptical review — luck, not a control. It is now mechanical, in the same register as FR-4.2's overrule-case gate: - bin/stability-classify matches each `file` candidate against the most recent prior snapshot recording it (FR-5.2 identity) and downgrades it to a boundary question if `measurement_strength` changed or the share crossed the 2% filing threshold between windows. Movement within a label is a trend, not an instability; no prior window is `unchecked`, not unstable (FR-2.4). - The downgrade is a finding against THIS repo, not the target's — the target did not change, the auditor described it two ways. Carries an `ia-stability` marker so a re-run comments rather than duplicating. - bin/ia_store.py factors the store and identity resolution out of audit-snapshot so both scripts hash signatures identically. A divergence there would make the gate match nothing and fail open. Documented as spec FR-4.5 and rubric §5b (shipped verbatim in references/). Wired into skills/offload-audit as step 4b, before filing-plan; steps 5-6 now consume its output rather than classified.json. S4 is retired in its old form — a human triage catching a bad candidate is not a repeatable test — and re-passed as: the gate catches the real release-policy-derivation case with no human in the loop. Asserted in tests/stability.test.sh case (f) against the verbatim two-window Phase 7 output. Closes kotkan/claude-plugin-inference-arbitrage#11 Suite: 178 assertions, exit 0.
710 lines
31 KiB
Python
Executable File
710 lines
31 KiB
Python
Executable File
#!/usr/bin/env python3
|
|
"""Persists one audit run as a snapshot record and diffs runs over time
|
|
(spec FR-5, plan §5-6).
|
|
|
|
It takes already-computed data — `plugin-inventory`, `offload-scan`,
|
|
`boundary-classify` — and does nothing but record, match, and subtract. Every
|
|
step here passes all five determinism tests, so none of it belongs in a model
|
|
(FR-8.1).
|
|
|
|
STORAGE AND THE PUSH BOUNDARY
|
|
-----------------------------
|
|
plan §5 says the store is this plugin's own Gitea wiki, "cloned to a local
|
|
cache and pushed via cluster:gitea-agent (this plugin never writes Gitea
|
|
directly)" — which is in tension with itself, because a git push to the wiki
|
|
repo IS a write to Gitea.
|
|
|
|
Resolved as: **this script is offline and filesystem-only.** It reads and
|
|
writes a local store directory laid out exactly like the wiki tree:
|
|
|
|
<store>/Data/<target>/snapshots.jsonl
|
|
<store>/Audits/<target>/<YYYY-MM-DD>.md
|
|
<store>/Audits/<target>/Latest.md
|
|
|
|
It never opens a socket — no git push, no Gitea API — which keeps plan §7's
|
|
"offline, only the two output paths touch the network" true, and keeps the
|
|
wiki credentials entirely outside this plugin. Publishing is a separate,
|
|
explicit action by the caller (the `offload-trend` / `offload-audit` skill, or
|
|
a human): `audit-snapshot pages` emits exactly which wiki pages to write with
|
|
their full content, and the caller performs the writes with the Gitea wiki
|
|
tools. `audit-snapshot import` is the other half of that bridge — it seeds the
|
|
local store from a `snapshots.jsonl` the caller fetched off the wiki, so a
|
|
machine with a cold cache still diffs against real history.
|
|
|
|
The append-only JSONL lives on the wiki as a page whose name is the file path;
|
|
Gitea wiki pages are files in a git repo, so the wiki IS the git-backed store
|
|
plan §5 asks for, and `git log` on it is the audit trail. Appending is
|
|
read-current-content, add one line, write back — which is why `pages` emits
|
|
the whole file, not a delta.
|
|
"""
|
|
|
|
import argparse
|
|
import json
|
|
import os
|
|
import sys
|
|
from datetime import datetime, timezone
|
|
|
|
from ia_store import (DEFAULT_STORE, MIN_INVOCATIONS, MIN_SHARE_OF_SPEND,
|
|
audits_dir, build_index, candidate_signature, find_match,
|
|
load_snapshots, scan_rows, signature_hash, snapshots_path,
|
|
store_root, tiered)
|
|
|
|
# Relative change in the volume-normalized metrics below which a candidate is
|
|
# `persisting` rather than grown/shrunk. Sampling noise across two windows of
|
|
# different length is easily a few percent; 10% is the smallest band that does
|
|
# not turn every run into a trend.
|
|
NOISE_BAND = 0.10
|
|
|
|
STATUSES = ("new", "persisting", "grown", "shrunk", "resolved",
|
|
"signature-drifted", "stale", "claimed-fixed-unconfirmed",
|
|
# the step ran in one window and not the other: no continuous
|
|
# series, so no direction (FR-5.4)
|
|
"unmeasured")
|
|
|
|
|
|
# --------------------------------------------------------------------------
|
|
# store
|
|
# --------------------------------------------------------------------------
|
|
|
|
def append_snapshot(store, target, snap):
|
|
path = snapshots_path(store, target)
|
|
os.makedirs(os.path.dirname(path), exist_ok=True)
|
|
with open(path, "a") as fh:
|
|
fh.write(json.dumps(snap, sort_keys=True) + "\n")
|
|
return path
|
|
|
|
|
|
def write_text(path, text):
|
|
os.makedirs(os.path.dirname(path), exist_ok=True)
|
|
with open(path, "w") as fh:
|
|
fh.write(text)
|
|
return path
|
|
|
|
|
|
# --------------------------------------------------------------------------
|
|
# write
|
|
# --------------------------------------------------------------------------
|
|
|
|
def tool_version():
|
|
manifest = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
|
|
".claude-plugin", "plugin.json")
|
|
try:
|
|
with open(manifest) as fh:
|
|
return json.load(fh).get("version")
|
|
except OSError:
|
|
return None
|
|
|
|
|
|
def measurements_by_signature(scan):
|
|
"""Every recurring signature the scan saw, keyed by hash — including ones
|
|
that no longer classify as candidates. FR-5.5 needs exactly this: a later
|
|
window's measured cost for a signature whose candidate is gone."""
|
|
total = scan.get("totals", {}).get("weighted_tokens") or 0
|
|
out = {}
|
|
for _name, row in scan_rows(scan):
|
|
for ng in row.get("ngrams", []):
|
|
h = signature_hash(ng["sig"])
|
|
invocations = ng.get("invocations", 0)
|
|
waste = ng.get("waste", 0)
|
|
out[h] = {
|
|
"invocations": invocations,
|
|
"offload_value": waste,
|
|
"share_of_plugin": round(waste / total, 4) if total else 0.0,
|
|
"cost_per_invocation": int(waste / invocations) if invocations else 0,
|
|
}
|
|
return out
|
|
|
|
|
|
def snapshot_candidate(cand, scan, explicit, issue_map):
|
|
sig, source = candidate_signature(cand, scan, explicit)
|
|
row = next((r for n, r in scan_rows(scan) if n == cand.get("skill")), {})
|
|
invocations = row.get("invocations", 0)
|
|
value = cand.get("offload_value", 0)
|
|
return {
|
|
"candidate_id": cand.get("candidate_id"),
|
|
"signature": sig,
|
|
"signature_source": source,
|
|
"signature_hash": signature_hash(sig),
|
|
"skill": cand.get("skill"),
|
|
"position": cand.get("position"),
|
|
"boundary_confidence": cand.get("boundary_confidence"),
|
|
"determinism_tests": cand.get("determinism_tests"),
|
|
"falsifiability": cand.get("falsifiability_triple"),
|
|
"measurement": {
|
|
"invocations": invocations,
|
|
"offload_waste": row.get("offload_waste", 0),
|
|
"offload_value": value,
|
|
"share_of_plugin": cand.get("share_of_audited_spend", 0.0),
|
|
# None, never 0, when the step did not run: a window in which a
|
|
# candidate was never invoked has no cost-per-invocation to report,
|
|
# and folding that to 0 makes "did not run" indistinguishable from
|
|
# "improved to zero" — the quiet-window artefact FR-5.4 exists to
|
|
# prevent, one level up from absolute tokens.
|
|
"cost_per_invocation": int(value / invocations) if invocations else None,
|
|
},
|
|
"measurement_strength": cand.get("measurement_strength"),
|
|
"digest_schema": cand.get("digest_schema"),
|
|
"escalation_path": cand.get("escalation_path"),
|
|
"issue": issue_map.get(cand.get("candidate_id")) or cand.get("issue"),
|
|
"verdict": cand.get("verdict"),
|
|
}
|
|
|
|
|
|
def build_snapshot(target, inventory, scan, classified, issue_map, explicit, run_id, repo,
|
|
extra_notes=()):
|
|
plugin = inventory.get("plugin", {})
|
|
scan_total = scan.get("totals", {})
|
|
|
|
rows = classified.get("candidates", [])
|
|
filed = [c for c in rows if c.get("verdict") == "file"]
|
|
questions = [c for c in rows if c.get("verdict") == "boundary-question"]
|
|
# Derived notes first, caller-supplied after. The caller's are outcomes of
|
|
# the filing pass — `filing: unavailable` and why (FR-6.4) — which this
|
|
# script cannot see and which have to reach the wiki page, not just the
|
|
# user's terminal.
|
|
notes = [
|
|
f"{c['candidate_id']}: {c['verdict']} — {c['reasons'][-1] if c.get('reasons') else ''}"
|
|
for c in rows if c.get("verdict") in ("no-offload", "drift-note")
|
|
] + list(extra_notes)
|
|
|
|
return {
|
|
"run_id": run_id,
|
|
"rubric_version": classified.get("rubric_version"),
|
|
"tool_versions": {
|
|
"inference-arbitrage": tool_version(),
|
|
"cc-tokens": scan.get("tool_versions", {}).get("cc_tokens_path"),
|
|
},
|
|
"target": {
|
|
"name": target,
|
|
"version": plugin.get("version"),
|
|
"source": plugin.get("source"),
|
|
"repo": repo,
|
|
# Not in plan §6, but FR-5.3's `stale` is uncomputable without it:
|
|
# "the skill no longer exists" needs the skill list of the run it
|
|
# no longer exists in.
|
|
"skills": sorted({s.get("name") for s in inventory.get("skills", []) if s.get("name")}
|
|
| {a.get("name") for a in inventory.get("agents", []) if a.get("name")}),
|
|
},
|
|
"window": scan.get("window", {}),
|
|
"coverage": {
|
|
"ratio": scan.get("coverage", {}).get("ratio"),
|
|
"attributed_turns": scan.get("coverage", {}).get("attributed_turns"),
|
|
},
|
|
"totals": {
|
|
"weighted_tokens": scan_total.get("weighted_tokens", 0),
|
|
"invocations": scan_total.get("invocations", 0),
|
|
"share_mechanical": scan_total.get("mechanical_share", 0.0),
|
|
},
|
|
"candidates": [snapshot_candidate(c, scan, explicit, issue_map) for c in filed],
|
|
"boundary_questions": [snapshot_candidate(c, scan, explicit, issue_map) for c in questions],
|
|
"measurements_by_signature": measurements_by_signature(scan),
|
|
"notes": notes,
|
|
}
|
|
|
|
|
|
# --------------------------------------------------------------------------
|
|
# diff (FR-5.3, FR-5.4, FR-5.5)
|
|
# --------------------------------------------------------------------------
|
|
|
|
def rel(before, after):
|
|
"""None when the base is zero — a ratio against nothing is not a ratio.
|
|
Callers fall back to the sign of the absolute delta in that case."""
|
|
if not before:
|
|
return None
|
|
return round((after - before) / before, 4)
|
|
|
|
|
|
def metric_delta(prev_m, curr_m, key):
|
|
# A missing measurement dict means the candidate was absent from that
|
|
# snapshot entirely (new, or departed) and 0 is the right base. An explicit
|
|
# None means the candidate was present but not invoked — that is an
|
|
# absence of measurement and must not be folded to a number.
|
|
a = (prev_m or {}).get(key, 0)
|
|
b = (curr_m or {}).get(key, 0)
|
|
if a is None or b is None:
|
|
# At least one window has no measurement for this metric. There is no
|
|
# delta to take against an absence, and inventing one is how a window
|
|
# the step sat out becomes a reported improvement.
|
|
return {"from": a, "to": b, "delta": None, "rel": None, "unmeasured": True}
|
|
return {"from": a, "to": b, "delta": b - a, "rel": rel(a, b)}
|
|
|
|
|
|
def direction(primary, secondary):
|
|
"""Volume-normalized verdict (FR-5.4). cost-per-invocation leads: it is the
|
|
metric a quiet week cannot move. share-of-spend only breaks the tie, and
|
|
absolute tokens never vote."""
|
|
if primary.get("unmeasured"):
|
|
# The step was not invoked in one of the two windows, so the series is
|
|
# not continuous and no direction can be read from it.
|
|
return "unmeasured"
|
|
for m in (primary, secondary):
|
|
if m.get("unmeasured"):
|
|
continue
|
|
r = m["rel"]
|
|
if r is None:
|
|
# Base of zero: the ratio is undefined, so the sign of the move is
|
|
# all there is.
|
|
if m["delta"] > 0:
|
|
return "grown"
|
|
continue
|
|
if r > NOISE_BAND:
|
|
return "grown"
|
|
if r < -NOISE_BAND:
|
|
return "shrunk"
|
|
return "persisting"
|
|
|
|
|
|
def remeasured(prev_cand, curr_snap):
|
|
return (curr_snap.get("measurements_by_signature") or {}).get(
|
|
prev_cand.get("signature_hash"))
|
|
|
|
|
|
def departed_status(prev_cand, curr_snap, issue_state):
|
|
"""A candidate that is no longer on the list. `resolved` is the strict
|
|
FR-5.5 conjunction — issue closed AND remeasured below threshold with
|
|
enough invocations. Everything else that merely stopped being reported is
|
|
`claimed-fixed-unconfirmed`, with the missing leg named: committed is not
|
|
deployed, and closed is not fixed."""
|
|
skills = set((curr_snap.get("target") or {}).get("skills") or [])
|
|
if skills and prev_cand.get("skill") not in skills:
|
|
return "stale", (f"skill {prev_cand.get('skill')} is absent from "
|
|
f"{curr_snap['target'].get('name')} "
|
|
f"{curr_snap['target'].get('version')} — the candidate no longer exists")
|
|
|
|
issue = prev_cand.get("issue")
|
|
state = issue_state.get(issue) if issue else None
|
|
m = remeasured(prev_cand, curr_snap)
|
|
below = bool(m and m["invocations"] >= MIN_INVOCATIONS
|
|
and m["share_of_plugin"] < MIN_SHARE_OF_SPEND)
|
|
|
|
if state == "closed" and below:
|
|
return "resolved", (f"{issue} closed and remeasured at "
|
|
f"{m['share_of_plugin']:.2%} of spend over "
|
|
f"{m['invocations']} invocations")
|
|
if state == "closed":
|
|
if m:
|
|
return "claimed-fixed-unconfirmed", (
|
|
f"{issue} closed but remeasurement is {m['invocations']} invocation(s) at "
|
|
f"{m['share_of_plugin']:.2%} — needs >= {MIN_INVOCATIONS} invocations below "
|
|
f"{MIN_SHARE_OF_SPEND:.0%}")
|
|
return "claimed-fixed-unconfirmed", (
|
|
f"{issue} closed but the signature was not remeasured in this window")
|
|
if issue:
|
|
return "claimed-fixed-unconfirmed", (
|
|
f"dropped off the candidate list while {issue} is still {state or 'unknown'}")
|
|
return "claimed-fixed-unconfirmed", (
|
|
"dropped off the candidate list with no linked issue and no remeasurement below threshold")
|
|
|
|
|
|
def row_of(cand, tier, status, reason, matched_by, prev_cand=None):
|
|
prev_m = (prev_cand or {}).get("measurement")
|
|
curr_m = cand.get("measurement") if cand else None
|
|
return {
|
|
"candidate_id": (cand or prev_cand).get("candidate_id"),
|
|
"skill": (cand or prev_cand).get("skill"),
|
|
"tier": tier,
|
|
"status": status,
|
|
"matched_by": matched_by,
|
|
"issue": (cand or prev_cand).get("issue"),
|
|
"primary": {
|
|
"cost_per_invocation": metric_delta(prev_m, curr_m, "cost_per_invocation"),
|
|
"share_of_plugin": metric_delta(prev_m, curr_m, "share_of_plugin"),
|
|
},
|
|
"context_only": {
|
|
"offload_value": metric_delta(prev_m, curr_m, "offload_value"),
|
|
"invocations": metric_delta(prev_m, curr_m, "invocations"),
|
|
},
|
|
"reason": reason,
|
|
}
|
|
|
|
|
|
def diff_snapshots(prev, curr, issue_state):
|
|
prev_all = list(tiered(prev))
|
|
idx = build_index([c for c, _ in prev_all])
|
|
prev_tier = {c["candidate_id"]: t for c, t in prev_all}
|
|
seen = set()
|
|
rows = []
|
|
|
|
for cand, tier in tiered(curr):
|
|
match, by = find_match(cand, idx)
|
|
if not match:
|
|
rows.append(row_of(cand, tier, "new", "not present in the previous snapshot", None))
|
|
continue
|
|
seen.add(match["candidate_id"])
|
|
if match.get("signature_hash") != cand.get("signature_hash"):
|
|
reason = (f"tool-sequence signature changed "
|
|
f"{match.get('signature_hash')} -> {cand.get('signature_hash')} "
|
|
f"(matched by {by}); the step was refactored, so the cost series is not continuous")
|
|
rows.append(row_of(cand, tier, "signature-drifted", reason, by, match))
|
|
continue
|
|
row = row_of(cand, tier, "persisting", None, by, match)
|
|
row["status"] = direction(row["primary"]["cost_per_invocation"],
|
|
row["primary"]["share_of_plugin"])
|
|
row["reason"] = (f"cost/invocation {row['primary']['cost_per_invocation']['rel']}, "
|
|
f"share {row['primary']['share_of_plugin']['rel']} "
|
|
f"(band +/-{NOISE_BAND:.0%})")
|
|
rows.append(row)
|
|
|
|
for cand, tier in prev_all:
|
|
if cand["candidate_id"] in seen:
|
|
continue
|
|
status, reason = departed_status(cand, curr, issue_state)
|
|
rows.append(row_of(None, tier, status, reason, None, cand))
|
|
|
|
summary = {s: sum(1 for r in rows if r["status"] == s) for s in STATUSES}
|
|
unaddressed = [
|
|
r for r in rows
|
|
if r["tier"] == "candidate" and r["status"] in ("persisting", "grown", "signature-drifted")
|
|
]
|
|
promotions = [
|
|
r for r in rows
|
|
if r["tier"] == "boundary-question" and r["status"] == "grown"
|
|
and prev_tier.get(r["candidate_id"]) == "boundary-question"
|
|
]
|
|
|
|
prev_vol = prev.get("totals", {}).get("weighted_tokens", 0)
|
|
curr_vol = curr.get("totals", {}).get("weighted_tokens", 0)
|
|
|
|
return {
|
|
"target": curr.get("target", {}).get("name"),
|
|
"from": {"run_id": prev.get("run_id"), "window": prev.get("window"),
|
|
"rubric_version": prev.get("rubric_version"),
|
|
"target_version": prev.get("target", {}).get("version")},
|
|
"to": {"run_id": curr.get("run_id"), "window": curr.get("window"),
|
|
"rubric_version": curr.get("rubric_version"),
|
|
"target_version": curr.get("target", {}).get("version")},
|
|
"rubric_comparable": prev.get("rubric_version") == curr.get("rubric_version"),
|
|
"volume": {
|
|
"weighted_tokens": {"from": prev_vol, "to": curr_vol, "rel": rel(prev_vol, curr_vol)},
|
|
"note": ("absolute totals are context only — a quieter window lowers every absolute "
|
|
"number without anything improving. Per-candidate verdicts come from "
|
|
"cost-per-invocation and share-of-spend."),
|
|
},
|
|
"candidates": rows,
|
|
"summary": summary,
|
|
"unaddressed": [r["candidate_id"] for r in unaddressed],
|
|
"unaddressed_issues": sorted({r["issue"] for r in unaddressed if r["issue"]}),
|
|
"promotions": [r["candidate_id"] for r in promotions],
|
|
}
|
|
|
|
|
|
# --------------------------------------------------------------------------
|
|
# rendering
|
|
# --------------------------------------------------------------------------
|
|
|
|
def fmt_rel(r):
|
|
if r is None:
|
|
return "n/a"
|
|
return f"{r:+.0%}"
|
|
|
|
|
|
def fmt_cpi(v):
|
|
"""`not run`, never 0 — the step had no invocations in that window."""
|
|
return "not run" if v is None else f"{v:,}"
|
|
|
|
|
|
def candidate_table(rows):
|
|
if not rows:
|
|
return "— none —\n"
|
|
out = ["| Candidate | Status | Cost/invocation | Share of spend | Issue |",
|
|
"|---|---|---|---|---|"]
|
|
for r in rows:
|
|
cpi = r["primary"]["cost_per_invocation"]
|
|
share = r["primary"]["share_of_plugin"]
|
|
out.append(
|
|
f"| `{r['candidate_id']}` | {r['status']} | "
|
|
f"{fmt_cpi(cpi['from'])} → {fmt_cpi(cpi['to'])} ({fmt_rel(cpi['rel'])}) | "
|
|
f"{share['from']:.2%} → {share['to']:.2%} ({fmt_rel(share['rel'])}) | "
|
|
f"{r['issue'] or '—'} |")
|
|
return "\n".join(out) + "\n"
|
|
|
|
|
|
def render_run(snap, diffdoc):
|
|
t = snap["target"]
|
|
w = snap.get("window", {})
|
|
lines = [
|
|
f"# {t['name']} — audit {snap['run_id']}",
|
|
"",
|
|
f"- Target: `{t['name']}` {t.get('version') or '?'} ({t.get('source') or '?'})"
|
|
+ (f", repo `{t['repo']}`" if t.get("repo") else ""),
|
|
f"- Window: {w.get('since')} → {w.get('until')}",
|
|
f"- Attribution coverage: {(snap['coverage'].get('ratio') or 0):.1%} "
|
|
f"({snap['coverage'].get('attributed_turns')} attributed turns) — "
|
|
"hook-driven and unattributed agent turns are excluded, so every figure below is a lower bound.",
|
|
f"- Mechanical share of audited spend: {(snap['totals'].get('share_mechanical') or 0):.1%}",
|
|
f"- Rubric v{snap.get('rubric_version')}, inference-arbitrage "
|
|
f"v{snap.get('tool_versions', {}).get('inference-arbitrage')}",
|
|
"",
|
|
"## Candidates",
|
|
"",
|
|
]
|
|
for c in snap.get("candidates", []):
|
|
m = c["measurement"]
|
|
lines += [
|
|
f"### `{c['candidate_id']}`",
|
|
"",
|
|
f"- Skill: `{c['skill']}` — position {c['position']}, confidence {c['boundary_confidence']}",
|
|
f"- {m['offload_value']:,} weighted tokens of offload value over {m['invocations']} "
|
|
f"invocations ({fmt_cpi(m['cost_per_invocation'])}/invocation, "
|
|
f"{m['share_of_plugin']:.2%} of audited spend)",
|
|
f"- Evidence: {c.get('measurement_strength')}; signature `{c['signature_hash']}` "
|
|
f"({c.get('signature_source')})",
|
|
f"- Issue: {c.get('issue') or 'not yet filed'}",
|
|
"",
|
|
]
|
|
if not snap.get("candidates"):
|
|
lines += ["Nothing to offload here.", ""]
|
|
|
|
if snap.get("boundary_questions"):
|
|
lines += ["## Boundary questions", "",
|
|
"Reported, not filed — the rubric could not reach `high` confidence with an "
|
|
"overrule case.", ""]
|
|
for c in snap["boundary_questions"]:
|
|
lines.append(f"- `{c['candidate_id']}` (`{c['skill']}`) — {c['boundary_confidence']}, "
|
|
f"{c['measurement']['share_of_plugin']:.2%} of spend")
|
|
lines.append("")
|
|
|
|
if snap.get("notes"):
|
|
lines += ["## Notes", ""] + [f"- {n}" for n in snap["notes"]] + [""]
|
|
|
|
if diffdoc:
|
|
lines += ["## Change since " + str(diffdoc["from"]["run_id"]), "",
|
|
candidate_table(diffdoc["candidates"]), ""]
|
|
if not diffdoc["rubric_comparable"]:
|
|
lines.append(f"**Rubric version changed** "
|
|
f"({diffdoc['from']['rubric_version']} → {diffdoc['to']['rubric_version']}): "
|
|
"classifications either side of the change were not graded by the same "
|
|
"table and must not be compared naively.\n")
|
|
return "\n".join(lines).rstrip() + "\n"
|
|
|
|
|
|
def render_latest(snap, diffdoc, history):
|
|
t = snap["target"]
|
|
lines = [
|
|
f"# {t['name']} — latest audit",
|
|
"",
|
|
f"Regenerated from `Data/{t['name']}/snapshots.jsonl` — do not edit by hand.",
|
|
"",
|
|
f"- Latest run: `{snap['run_id']}` against {t['name']} {t.get('version') or '?'}",
|
|
f"- Runs recorded: {len(history)} (first `{history[0]['run_id']}`)",
|
|
f"- Open candidates: {len(snap.get('candidates', []))}; "
|
|
f"boundary questions: {len(snap.get('boundary_questions', []))}",
|
|
"",
|
|
]
|
|
if diffdoc:
|
|
s = diffdoc["summary"]
|
|
lines += [
|
|
"## Trend",
|
|
"",
|
|
f"Against `{diffdoc['from']['run_id']}`. Audited volume moved "
|
|
f"{fmt_rel(diffdoc['volume']['weighted_tokens']['rel'])} — context only; the verdicts "
|
|
"below are per-invocation and share-of-spend.",
|
|
"",
|
|
"| Status | Count |", "|---|---|",
|
|
] + [f"| {k} | {v} |" for k, v in s.items() if v] + [""]
|
|
if diffdoc["unaddressed"]:
|
|
refs = ", ".join(diffdoc["unaddressed_issues"]) or "none filed"
|
|
lines += [f"**{len(diffdoc['unaddressed'])} prior recommendation(s) still unaddressed** "
|
|
f"— {refs}", ""]
|
|
if diffdoc["promotions"]:
|
|
lines += ["**Boundary questions recurring with growing cost** (candidates for promotion): "
|
|
+ ", ".join(f"`{c}`" for c in diffdoc["promotions"]), ""]
|
|
lines += [candidate_table(diffdoc["candidates"]), ""]
|
|
|
|
lines += ["## Open candidates", "",
|
|
"| Candidate | Skill | Share of spend | Cost/invocation | Issue |",
|
|
"|---|---|---|---|---|"]
|
|
for c in snap.get("candidates", []):
|
|
m = c["measurement"]
|
|
lines.append(f"| `{c['candidate_id']}` | `{c['skill']}` | {m['share_of_plugin']:.2%} | "
|
|
f"{fmt_cpi(m['cost_per_invocation'])} | {c.get('issue') or '—'} |")
|
|
if not snap.get("candidates"):
|
|
lines.append("| — none — | | | | |")
|
|
lines += ["", "## Run history", "", "| Run | Target version | Coverage | Candidates |",
|
|
"|---|---|---|---|"]
|
|
for h in history[-10:]:
|
|
lines.append(f"| `{h['run_id']}` | {h['target'].get('version') or '?'} | "
|
|
f"{(h.get('coverage', {}).get('ratio') or 0):.1%} | "
|
|
f"{len(h.get('candidates', []))} |")
|
|
return "\n".join(lines).rstrip() + "\n"
|
|
|
|
|
|
def regenerate(store, target, history):
|
|
"""Dated page appended, Latest.md regenerated (plan §5)."""
|
|
snap = history[-1]
|
|
diffdoc = diff_snapshots(history[-2], snap, {}) if len(history) > 1 else None
|
|
day = str(snap["run_id"])[:10]
|
|
return [
|
|
write_text(os.path.join(audits_dir(store, target), f"{day}.md"), render_run(snap, diffdoc)),
|
|
write_text(os.path.join(audits_dir(store, target), "Latest.md"),
|
|
render_latest(snap, diffdoc, history)),
|
|
]
|
|
|
|
|
|
# --------------------------------------------------------------------------
|
|
# commands
|
|
# --------------------------------------------------------------------------
|
|
|
|
def load_json(path):
|
|
with open(path) as fh:
|
|
return json.load(fh)
|
|
|
|
|
|
def cmd_write(args):
|
|
inventory = load_json(args.from_files[0])
|
|
scan = load_json(args.from_files[1])
|
|
classified = load_json(args.classified)
|
|
issue_map = load_json(args.issue_map) if args.issue_map else {}
|
|
explicit = load_json(args.signatures) if args.signatures else {}
|
|
|
|
run_id = args.run_id or datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
|
history = load_snapshots(args.store, args.target)
|
|
if any(s.get("run_id") == run_id for s in history):
|
|
sys.exit(f"error: run_id {run_id} already recorded for {args.target}")
|
|
|
|
snap = build_snapshot(args.target, inventory, scan, classified,
|
|
issue_map, explicit, run_id, args.repo, args.note)
|
|
path = append_snapshot(args.store, args.target, snap)
|
|
history.append(snap)
|
|
pages = regenerate(args.store, args.target, history)
|
|
return {"run_id": run_id, "snapshots": path, "pages": pages,
|
|
"candidates": len(snap["candidates"]),
|
|
"boundary_questions": len(snap["boundary_questions"])}
|
|
|
|
|
|
def cmd_list(args):
|
|
history = load_snapshots(args.store, args.target)
|
|
return {
|
|
"target": args.target,
|
|
"runs": [{
|
|
"run_id": s.get("run_id"),
|
|
"target_version": s.get("target", {}).get("version"),
|
|
"rubric_version": s.get("rubric_version"),
|
|
"window": s.get("window", {}),
|
|
"coverage": s.get("coverage", {}).get("ratio"),
|
|
"weighted_tokens": s.get("totals", {}).get("weighted_tokens"),
|
|
"candidates": len(s.get("candidates", [])),
|
|
"boundary_questions": len(s.get("boundary_questions", [])),
|
|
} for s in history],
|
|
}
|
|
|
|
|
|
def cmd_diff(args):
|
|
history = load_snapshots(args.store, args.target)
|
|
if len(history) < 2:
|
|
sys.exit(f"error: {args.target} has {len(history)} snapshot(s); diff needs 2")
|
|
curr = history[-1]
|
|
if args.against:
|
|
prev = next((s for s in history if s.get("run_id") == args.against), None)
|
|
if prev is None:
|
|
sys.exit(f"error: no snapshot {args.against} for {args.target}")
|
|
else:
|
|
prev = history[-2]
|
|
issue_state = load_json(args.issue_state) if args.issue_state else {}
|
|
return diff_snapshots(prev, curr, issue_state)
|
|
|
|
|
|
def cmd_pages(args):
|
|
"""What to publish to the wiki, and with what content. The caller performs
|
|
the writes — see the module docstring on why this script never does."""
|
|
history = load_snapshots(args.store, args.target)
|
|
if not history:
|
|
sys.exit(f"error: no snapshots recorded for {args.target}")
|
|
snap = history[-1]
|
|
day = str(snap["run_id"])[:10]
|
|
root = store_root(args.store)
|
|
pages = [{
|
|
"page": f"Data/{args.target}/snapshots.jsonl",
|
|
"title": f"Data/{args.target}/snapshots.jsonl",
|
|
"append_only": True,
|
|
"local": snapshots_path(args.store, args.target),
|
|
}, {
|
|
"page": f"Audits/{args.target}/{day}",
|
|
"title": f"Audits/{args.target}/{day}",
|
|
"append_only": False,
|
|
"local": os.path.join(audits_dir(args.store, args.target), f"{day}.md"),
|
|
}, {
|
|
"page": f"Audits/{args.target}/Latest",
|
|
"title": f"Audits/{args.target}/Latest",
|
|
"append_only": False,
|
|
"local": os.path.join(audits_dir(args.store, args.target), "Latest.md"),
|
|
}]
|
|
for p in pages:
|
|
with open(p["local"]) as fh:
|
|
p["content"] = fh.read()
|
|
p["local"] = os.path.relpath(p["local"], root)
|
|
return {"target": args.target, "store": root, "run_id": snap["run_id"], "pages": pages}
|
|
|
|
|
|
def cmd_import(args):
|
|
"""Seed the local store from a snapshots.jsonl fetched off the wiki, so a
|
|
cold cache still diffs against real history."""
|
|
with open(args.snapshots) as fh:
|
|
incoming = [json.loads(line) for line in fh if line.strip()]
|
|
# The only place a snapshot arrives from outside this script: a wiki page a
|
|
# human can edit. Validate here rather than defensively everywhere after.
|
|
for i, snap in enumerate(incoming, 1):
|
|
missing = [k for k in ("run_id", "target", "window", "coverage", "totals",
|
|
"candidates", "boundary_questions") if k not in snap]
|
|
if missing:
|
|
sys.exit(f"error: {args.snapshots}:{i}: not a snapshot record, missing {missing}")
|
|
existing = {s.get("run_id") for s in load_snapshots(args.store, args.target)}
|
|
added = [s for s in incoming if s.get("run_id") not in existing]
|
|
for snap in added:
|
|
append_snapshot(args.store, args.target, snap)
|
|
history = load_snapshots(args.store, args.target)
|
|
pages = regenerate(args.store, args.target, history) if history else []
|
|
return {"target": args.target, "imported": len(added), "skipped": len(incoming) - len(added),
|
|
"runs": len(history), "pages": pages}
|
|
|
|
|
|
def main():
|
|
ap = argparse.ArgumentParser(prog="audit-snapshot")
|
|
ap.add_argument("--store", default=DEFAULT_STORE,
|
|
help=f"local mirror of the wiki store (default {DEFAULT_STORE})")
|
|
ap.add_argument("--json", action="store_true", help="compact single-line JSON")
|
|
sub = ap.add_subparsers(dest="cmd", required=True)
|
|
|
|
w = sub.add_parser("write", help="append one snapshot and regenerate the audit pages")
|
|
w.add_argument("--target", required=True)
|
|
w.add_argument("--from", dest="from_files", nargs=2, required=True,
|
|
metavar=("INVENTORY", "SCAN"))
|
|
w.add_argument("--classified", required=True, help="bin/boundary-classify output")
|
|
w.add_argument("--issue-map", help='JSON {candidate_id: "owner/repo#N"}')
|
|
w.add_argument("--signatures", help="JSON {candidate_id: [tool signature, ...]}")
|
|
w.add_argument("--repo", help="target's repo, owner/repo")
|
|
w.add_argument("--run-id", help="override the run id (tests; default is now, UTC)")
|
|
w.add_argument("--note", action="append", default=[], metavar="TEXT",
|
|
help="add a note to the snapshot and its summary page; repeatable. "
|
|
"Carries filing outcomes the audit knows and this script does not, "
|
|
"notably FR-6.4's `filing: unavailable` and its reason")
|
|
w.set_defaults(func=cmd_write)
|
|
|
|
ls = sub.add_parser("list", help="list recorded runs")
|
|
ls.add_argument("--target", required=True)
|
|
ls.set_defaults(func=cmd_list)
|
|
|
|
d = sub.add_parser("diff", help="diff the latest run against a prior one")
|
|
d.add_argument("--target", required=True)
|
|
d.add_argument("--against", help="run_id to compare against (default: previous)")
|
|
d.add_argument("--issue-state", help='JSON {"owner/repo#N": "open"|"closed"}')
|
|
d.set_defaults(func=cmd_diff)
|
|
|
|
p = sub.add_parser("pages", help="emit the wiki pages to publish, with content")
|
|
p.add_argument("--target", required=True)
|
|
p.set_defaults(func=cmd_pages)
|
|
|
|
i = sub.add_parser("import", help="seed the store from a wiki-fetched snapshots.jsonl")
|
|
i.add_argument("--target", required=True)
|
|
i.add_argument("--snapshots", required=True)
|
|
i.set_defaults(func=cmd_import)
|
|
|
|
args = ap.parse_args()
|
|
result = args.func(args)
|
|
json.dump(result, sys.stdout, indent=None if args.json else 2)
|
|
sys.stdout.write("\n")
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|