This plugin runs thruwire/foreman (MIT) over our own agents. Upstream supervises an OpenAI Codex worker with TypeSafe's paid Jev model. Both of those are foreign here, so both are replaced while the architecture is kept intact:

Part	Upstream	Here
Worker	`codex app-server` / `codex exec`	`claude -p` as a streaming subprocess
Judge	TypeSafe Jev (`TYPESAFE_API_KEY`, paid)	`free-worker.sh` — the free-model gateway
Policy	deterministic Python, thresholds	unchanged, upstream's
Live steering	`turn/steer` into an active turn	not available → `steer()` returns `False`, policy degrades to stop/retry

A supervised run therefore costs no paid tokens for supervision — the worker is the only paid part, and it is the part that does the work.

What it actually watches

Every 20–45 seconds (debounced, lifecycle events bypass the delay) the supervisor builds one bounded snapshot — never a repository dump:

the original job, factory status, iteration, elapsed time, attempts;
active worker, worker history, exit status, output tails (compacted from stream-json);
git status, a bounded diff, changed files;
commits and diff since the run's base commit (added here — see below);
the repository's CLAUDE.md (or AGENTS.md) instructions;
the previous assessment and the previous intervention.

A free model answers ten independent yes/no questions as probabilities:

implementation_complete  tests_sufficient  requirements_satisfied  needs_verification  ready_to_finish
meaningful_progress  worker_stuck  work_off_track  agents_md_drift  needs_human

The model only estimates. A deterministic policy picks one legal action: CONTINUE · START_WORKER · START_VERIFIER · STOP_WORKER · RETRY_WORKER · FINISH · ESCALATE.

Two additions, both measured — not assumed

1. Commits belong in the observation. Upstream watches the working tree. Our agents commit their work, which empties git diff — the judge goes blind at exactly the moment the worker does the right thing. The observation now carries git log and the diff since the run's base commit.

2. Warnings stay muted until there is evidence. A small judge escalated a healthy run at 25 seconds, before the worker had touched a single file. Warning dimensions (stuck / off track / drift / needs human) are claims about work that happened, so they are held until a changed file, a commit or a finished worker exists — or until warm-up ends. This is upstream's own instruction to the model ("insufficient evidence → answer no"), applied deterministically on our side.

Calibration: what the runs actually show

Upstream is explicit that its thresholds are not calibrated and that the JSONL timeline exists so runs can be labelled afterwards and thresholds replayed. bin/foreman-calibrate.py does that:

foreman-calibrate.py /path/to/repo-a /path/to/repo-b --true-run <id> --false-run <id>

Measured over the first four runs on this machine:

Question	Answer
Highest false warning	`agents_md_drift` 0.78 / `needs_human` 0.65 — in a run that finished cleanly
Lowest true warning that acted	`needs_human` 0.80
Can one threshold separate them?	No — they overlap. Moving 0.80 up or down cannot fix this
Replay at 0.80 without warm-up	4 false fires vs 1 true
Replay with confirmation (two highs in a row)	0 false fires — and 0 true: a short run's true warning lands on its last assessment

So the shipped defaults are: warm-up on (it removes the early false spikes), thresholds left at upstream's 0.80, and the confirmation rule available behind --confirm-warnings for long runs where one cycle of delay is cheap.

Proof it works

Run	Repository	Outcome	Numbers
1	duration-parser lab	`FINISHED`	worker implemented and committed the function, 6 tests green; 4 judge calls, 8710 free tokens, 0 rejected answers
2	dependency-ban lab	`ESCALATED`	the job demanded a third-party library, the repository's `CLAUDE.md` bans dependencies, the worker used the standard library instead — judge: off track 70 %, needs human 80 % → worker stopped, human called

Run 2 is the point of the whole thing: the job and the repository's rules contradicted each other, the worker silently picked one, and the supervisor caught the divergence instead of letting a confidently-wrong result land.

Install

claude plugin install foreman-supervisor@kotkan-fleet

Upstream foreman is a separate Python package and is not vendored:

git clone https://github.com/thruwire/foreman.git ~/pro/foreman
cd ~/pro/foreman && python3 -m venv .venv && .venv/bin/python -m pip install '.[dev]'

Run the supervisor with that virtualenv's interpreter:

~/pro/foreman/.venv/bin/python bin/foreman-claude.py \
    --repo /path/to/project \
    --job "Implement X and cover it with tests."

Useful flags: --judge-once (score the repository as it is right now, one free call, no worker), --print-prompt (show what the judge would be asked, no call at all), --model, --warmup-seconds, --confirm-warnings, --judge-attempts.

Requirements

Python 3.11+ and the upstream foreman package in a virtualenv;
claude on PATH (the worker);
free-worker.sh from the contour (the judge) — any gateway that answers a prompt on stdout works;
no TypeSafe key, no Codex CLI.

What this does not do

It does not steer a running worker: claude -p has no in-flight input channel, so a warning goes straight to stop/retry. If a steerable transport appears, steer() is the only method to change.
It does not sandbox anything. The worker runs with the permissions you give it.
It does not prove that FINISH means correct. It bounds how long wrong work can run unattended.
It does not yet wrap background workers started elsewhere (worker.sh, night runners) — that is tracked as kotkan/dreamcore-mp#179.

Upstream, read from source

docs/upstream-review.md holds the review of thruwire/foreman itself: how the runtime, policy and observation are built, what the author's own issues say (#4 thresholds uncalibrated, #6 README drift, #7/#10 transient failures escalating a whole run, #8/#9 generic backends), and the two upstream defects found while running it here.

Licence

MIT, matching upstream.

README.md Unescape Escape

foreman-supervisor