Mine per-derivation build times from nom / nix logs into a --history file #3

New Issue

2026-07-07T20:13:07+03:00

oleks commented

2026-07-07 20:13:07 +03:00

Why (now concrete)

The emmett trial made the heuristic table's ceiling obvious: it only knows ~35 heavy-package substrings; everything else collapses to a flat 1 min. Any deep dependency chain then fabricates a long span (see #2). Real timings are the single biggest accuracy lever, and the plumbing already exists — --history {name: minutes} is wired and history wins over the heuristic. Only the miner is missing.

Proposal

Add a nix-estimate-mine subcommand / helper that parses real build logs into {name: minutes}:

Source A: nom --json / nix build --log-format internal-json event stream — has per-drv start/stop, exact and cheap to parse.
Source B (fallback): timestamped nix build -v logs, diffing build-start/finish lines.
Key by store_name() (the same key cost() looks up) so mined files drop straight into --history.
Aggregate repeated builds (median), and record the core count the build ran at so scale_to_cores can normalize to the 8-core baseline.

Acceptance

Feed a captured emmett/aarch64 build log; produce a history JSON; re-run the estimator with --history and confirm the span is driven by real heavy derivations, not glue chains.

Roadmap item #1. Complements #2 (which fixes the glue-chain artifact in the absence of history).

## Why (now concrete) The emmett trial made the heuristic table's ceiling obvious: it only knows ~35 heavy-package substrings; everything else collapses to a flat 1 min. Any deep dependency chain then fabricates a long span (see #2). Real timings are the single biggest accuracy lever, and the plumbing already exists — `--history {name: minutes}` is wired and history wins over the heuristic. Only the **miner** is missing. ## Proposal Add a `nix-estimate-mine` subcommand / helper that parses real build logs into `{name: minutes}`: - Source A: `nom --json` / `nix build --log-format internal-json` event stream — has per-drv start/stop, exact and cheap to parse. - Source B (fallback): timestamped `nix build -v` logs, diffing build-start/finish lines. - Key by `store_name()` (the same key `cost()` looks up) so mined files drop straight into `--history`. - Aggregate repeated builds (median), and record the core count the build ran at so `scale_to_cores` can normalize to the 8-core baseline. ## Acceptance Feed a captured emmett/aarch64 build log; produce a history JSON; re-run the estimator with `--history` and confirm the span is driven by real heavy derivations, not glue chains. Roadmap item #1. Complements #2 (which fixes the glue-chain artifact in the *absence* of history).

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: oleks/nix-estimator#3