Estimate the best remote-builder configuration — how many builder nodes × how many cores per node — for a given Nix build, before you provision anything.

It reads the to-build derivation DAG from Nix, applies a build-time cost model, and computes the parallel-computing quantities that bound each dimension, then sweeps (nodes × cores) through a list-scheduler to recommend a shape.

nix-estimate .#packages.aarch64-linux.mempalace-image-arm64 --system aarch64-linux

Why

More builder nodes only help while the dependency graph is wide (many independent derivations ready at once). Once a build narrows to a long pole — one big derivation like onnxruntime, llvm, or ghc — no number of nodes helps, because Nix builds a single derivation on a single machine. That long pole is moved only by cores on one node. nix-estimator makes this concrete for your build instead of leaving it to intuition.

The model

The build is a DAG of derivations. Two classic quantities govern it:

quantity	meaning	bounds
work `T₁`	Σ of every build time	wall-clock on 1 core / 1 node
span `T∞`	longest dependent chain	floor no parallelism beats
width	peak simultaneously-ready builds	node-count ceiling

Brent's bound T_p ≈ max(T₁/p, T∞) says it all: node-parallelism is capped by width and by T₁/T∞ (average available parallelism); core-parallelism is set by the heaviest single derivation's internal -j scaling. nix-estimator reports all of these plus a (nodes × cores) → makespan grid from a critical-path list-scheduler, and picks the diminishing-returns knee.

The cost model

Nix does not predict per-derivation build time, so costmodel.py approximates it: a coarse heuristic table (heavy: onnxruntime/llvm/ghc/rustc/…; default: ~1 min; fixed-output fetches: network-bound) plus an Amdahl core_scaling per derivation. Pass --history <json> ({name: minutes} mined from real nom / nix build logs) for accuracy — history always wins over the heuristic.

Usage

# human report (default)
nix-estimate .#foo --system aarch64-linux

# machine-readable
nix-estimate .#foo --json

# sweep custom grids, use measured build times
nix-estimate .#foo --nodes 1,2,4,8 --cores 8,16,32 --history builds.json

# ignore the cache — estimate a from-scratch build of the whole closure
nix-estimate .#foo --cold

# drop derivations the upstream binary cache can already substitute (issue #22).
# Most useful with --cold: collapses the closure (incl. the ~310 stdenv
# bootstrap drvs) to the genuinely unsubstitutable remainder. Optional URL arg.
nix-estimate .#foo --cold --assume-cache
nix-estimate .#foo --assume-cache https://my-cache.example.org

# override the DEFAULT cost bucket (min/derivation) — the single biggest driver
# of the estimate on OS-sized closures (issue #23). Calibrate it with `calibrate`.
nix-estimate .#foo --default-min 0.6

# model Nix per-node concurrency + a RAM budget (issues #13/#15)
nix-estimate .#foo --max-jobs 4 --node-ram-gb 16

# print a per-node Gantt of the recommended shape's schedule (issue #16)
nix-estimate .#foo --timeline

# emit EphemeralBuilder CRs for the recommendation (issue #14) — one document
# per recommended node (no replica field: one EphemeralBuilder == one VM).
# Each `ramGb` is sized from the modeled peak concurrent RAM on that node
# (issue #19); pass --node-ram-gb to override. YAML goes to stdout, the human
# report to stderr, so this pipes cleanly
nix-estimate .#foo --provision | kubectl apply -f -
nix-estimate .#foo --provision runner --provision-name my-builder

Mining a history file (issue #12)

--history is only as good as the timings you feed it. The mine subcommand runs real builds and extracts per-derivation minutes from Nix's --log-format internal-json stream:

# build once, print {name: minutes} JSON to stdout
nix-estimate mine .#foo

# build 3× and merge (median, to shrug off noise) into a file
nix-estimate mine .#foo .#bar --repeat 3 -o builds.json
nix-estimate .#foo --history builds.json

Calibrating `--default-min` (issue #23)

The DEFAULT bucket (~1 min) dominates the headline estimate on OS-sized closures — most library derivations match neither the heavy table nor the trivial/fixed-output shapes. The calibrate subcommand classifies your closure into buckets (ignoring history), then reports the measured median minutes each bucket carries in a mined history, versus the current hardcoded constant:

nix-estimate calibrate .#foo --history builds.json --system aarch64-linux

It prints a per-bucket table (measured median vs constant, sample counts) and a one-line suggestion like re-run with --default-min 0.62. Feed that back via --default-min on the normal estimate path.

The plain nix-estimate <attr> [flags] invocation is unchanged — mine and calibrate are the only subcommands, each selected by its leading literal token.

Output: closure size, to-build count, work/span/avg/peak, the critical path (the long pole), a makespan grid, and a RECOMMENDATION (nodes × cores ≈ minutes, vs. one-big-node).

Requires nix on PATH. Pure stdlib otherwise. nix develop for a dev shell.

nix develop      # python + pytest + nix
pytest           # unit tests for the scheduler (toy DAGs, no Nix needed)

Layout

nix_estimator/
  graph.py      DAG + to-build set via `nix derivation show -r` + `--dry-run`
  costmodel.py  heuristic/history build-time + Amdahl core-scaling
  schedule.py   critical path, peak concurrency, p-machine list scheduler (pure)
  estimate.py   orchestration + node×core sweep + knee recommendation
  miner.py      mine {name: minutes} from internal-json build logs (pure parser)
  provision.py  render the recommendation as an EphemeralBuilder CR (YAML)
  cli.py        `nix-estimate` entrypoint + report + `mine` subcommand
tests/          scheduler unit tests on toy graphs

Prior art (and where this sits)

The node×core provisioning recommendation for Nix is an unfilled gap. The ingredients exist separately; nix-estimator assembles them:

Hubble (INRIA, L. Courtès) — the closest analytical core: a SimGrid simulator of scheduling strategies on the Nix/Hydra DAG with a critical-path tool and speedup plots. But it evaluates scheduling algorithms as a (dormant, ~2010-era) research artifact — it does not output a fleet-size recommendation.
nixbuild.net — solves sizing operationally, auto-assigning CPU/RAM per derivation from history, but as a closed hosted autoscaler with no DAG-level analysis you can run yourself.
DAG extractors — nom, nix-tree, nix-visualize — topology only, no timing/scheduling.
Build-time cost models — Uber's CI at Scale (NGBoost per-target Bazel build-time prediction) — methodology to borrow for a learned cost model.
Theory — list scheduling, Brent's theorem, HEFT. nix-estimator's novelty is applying it to the Nix derivation DAG for provisioning, not the theory.

Status

v0.1 — heuristic cost model, working scheduler + recommendation. Now also: history mining from internal-json build logs (mine subcommand, issue #12), --max-jobs per-node concurrency and --node-ram-gb as a second constraint (issues #13/#15), a --timeline per-node Gantt (issue #16), and a --provision mode that emits an EphemeralBuilder spec for the chosen shape (issue #14). Roadmap: a learned cost model, and calibrating the estimates against mined makespans.

MIT.

README.md Unescape Escape

nix-estimator

Why