nix-estimator

Estimate the best remote-builder configuration — how many builder nodes × how many cores per node — for a given Nix build, before you provision anything.

It reads the to-build derivation DAG from Nix, applies a build-time cost model, and computes the parallel-computing quantities that bound each dimension, then sweeps (nodes × cores) through a list-scheduler to recommend a shape.

nix-estimate .#packages.aarch64-linux.mempalace-image-arm64 --system aarch64-linux

Why

More builder nodes only help while the dependency graph is wide (many independent derivations ready at once). Once a build narrows to a long pole — one big derivation like onnxruntime, llvm, or ghc — no number of nodes helps, because Nix builds a single derivation on a single machine. That long pole is moved only by cores on one node. nix-estimator makes this concrete for your build instead of leaving it to intuition.

The model

The build is a DAG of derivations. Two classic quantities govern it:

quantity	meaning	bounds
work T₁	Σ of every build time	wall-clock on 1 core / 1 node
span T∞	longest dependent chain	floor no parallelism beats
width	peak simultaneously-ready builds	node-count ceiling

Brent's bound T_p ≈ max(T₁/p, T∞) says it all: node-parallelism is capped by width and by T₁/T∞ (average available parallelism); core-parallelism is set by the heaviest single derivation's internal -j scaling. nix-estimator reports all of these plus a (nodes × cores) → makespan grid from a critical-path list-scheduler, and picks the diminishing-returns knee.

The cost model

Nix does not predict per-derivation build time, so costmodel.py approximates it: a coarse heuristic table (heavy: onnxruntime/llvm/ghc/rustc/…; default: ~1 min; fixed-output fetches: network-bound) plus an Amdahl core_scaling per derivation. Pass --history <json> ({name: minutes} mined from real nom / nix build logs) for accuracy — history always wins over the heuristic.

Usage

# human report (default)
nix-estimate .#foo --system aarch64-linux

# machine-readable
nix-estimate .#foo --json

# sweep custom grids, use measured build times
nix-estimate .#foo --nodes 1,2,4,8 --cores 8,16,32 --history builds.json

# ignore the cache — estimate a from-scratch build of the whole closure
nix-estimate .#foo --cold

Output: closure size, to-build count, work/span/avg/peak, the critical path (the long pole), a makespan grid, and a RECOMMENDATION (nodes × cores ≈ minutes, vs. one-big-node).

Requires nix on PATH. Pure stdlib otherwise. nix develop for a dev shell.

nix develop      # python + pytest + nix
pytest           # unit tests for the scheduler (toy DAGs, no Nix needed)

Layout

nix_estimator/
  graph.py      DAG + to-build set via `nix derivation show -r` + `--dry-run`
  costmodel.py  heuristic/history build-time + Amdahl core-scaling
  schedule.py   critical path, peak concurrency, p-machine list scheduler (pure)
  estimate.py   orchestration + node×core sweep + knee recommendation
  cli.py        `nix-estimate` entrypoint + report
tests/          scheduler unit tests on toy graphs

Prior art (and where this sits)

The node×core provisioning recommendation for Nix is an unfilled gap. The ingredients exist separately; nix-estimator assembles them:

• Hubble (INRIA, L. Courtès) — the closest analytical core: a SimGrid simulator of scheduling strategies on the Nix/Hydra DAG with a critical-path tool and speedup plots. But it evaluates scheduling algorithms as a (dormant, ~2010-era) research artifact — it does not output a fleet-size recommendation.
• nixbuild.net — solves sizing operationally, auto-assigning CPU/RAM per derivation from history, but as a closed hosted autoscaler with no DAG-level analysis you can run yourself.
• DAG extractors — nom, nix-tree, nix-visualize — topology only, no timing/scheduling.
• Build-time cost models — Uber's CI at Scale (NGBoost per-target Bazel build-time prediction) — methodology to borrow for a learned cost model.
• Theory — list scheduling, Brent's theorem, HEFT. nix-estimator's novelty is applying it to the Nix derivation DAG for provisioning, not the theory.

Status

v0.1 — heuristic cost model, working scheduler + recommendation. Roadmap: mine real per-derivation timings from nom/nix logs into a history file (biggest accuracy win), RAM-per-core as a second constraint, and a --provision mode that emits an EphemeralBuilder spec for the chosen shape.

MIT.