diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..fdeb277 --- /dev/null +++ b/.gitignore @@ -0,0 +1,3 @@ +.parity-stage/ +result +result-* diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..18e1257 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,31 @@ +# Changelog + + + +All notable changes to parity-lib are documented here. This project follows +semantic versioning; the version is a conceptual tag (no git tag is created). + +## v0.1.0 + +Initial release (cluster #192/#193/#194, emmett#44). + +- `lib.mkParityBuilders pkgs` plus per-builder wrappers exposing the six + archetype publish-app builders: + - `mkPyPiWheelPublish` — single-arch Gitea PyPI wheel. + - `mkS390xNpmPublish` — single-arch Gitea npm native addon. + - `mkGenericBinaryPublish` — single-arch Gitea generic-registry binary. + - `mkGoBinaryPublish` — alias of `mkGenericBinaryPublish` (explicit archetype). + - `mkNix2ContainerPublish` — multi-arch OCI image with `publish-index` and + `:latest` digest copy. + - `mkHelmPublish` — Helm chart to an OCI registry. +- Each builder returns flake apps following the corrected parity standard: + `stage-` (build-parity, no registry), `publish-` (dry-run by + default), `publish-index` (build-free, fail-closed multi-arch assembly via + regctl), `publish` (all local arches + index + `:latest` last), and + `push-staged` (replay `./.parity-stage`). +- Shared shell library `ci/parity-lib.sh` (token resolution with + `$REGISTRY_TOKEN` + `pass` fallback and never printed, dev-tag guard, version + derivation, the dry-run gate, registry preflight, stage-dir helpers). +- `packages.pipeline-doctor` / `apps.pipeline-doctor` (cluster #193): static + parity-contract checker that prints local-equivalent commands. +- `flake.lock` fully pinned; nixpkgs follows the shared `fleet-pins` `nixpkgs-ci`. diff --git a/README.md b/README.md index 46bc694..8da7e85 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,182 @@ # parity-lib -Shared versioned flake-module for local pipeline parity (emmett#44, cluster #192): mkPyPiWheelPublish / mkS390xNpmPublish / mkGenericBinaryPublish / mkNix2ContainerPublish + pipeline-doctor. Single source of truth so per-repo publish apps cannot drift. \ No newline at end of file + + +Shared per-archetype **publish-app builders** for the ~51 "parity" repos +(cluster #192/#193/#194, emmett#44). Instead of each repo hand-inlining its own +`stage` / `publish` / `push-staged` shell, every repo consumes ONE versioned +source of truth: this flake's `lib.mk*Publish` builders generate the flake apps +that implement the corrected parity standard. + +## The corrected parity standard (emmett#44) + +Parity has two halves, and only one of them can be cluster-independent: + +- **BUILD/STAGE parity is cluster-independent.** `nix run .#stage-` + cross-builds the artifact into a local on-disk store (`./.parity-stage/` + by convention) and makes **NO registry contact**, so it runs identically on + emmett even when armer / the registry is down. +- **PUBLISH parity shares fate with the cluster** (the only registry is + co-located with it). The publish apps name the blocker (`registry-down`) up + front and never half-push. + +Every push entrypoint **DRY-RUNS by default** — it stages and prints what it +WOULD push. You must pass `--publish` (or set `PUBLISH=1`) to mutate the +registry, so an accidental local run can never push. + +## App shape each builder emits + +| app | parity half | registry | mutates? | +| --- | --- | --- | --- | +| `stage-` | BUILD | none | no | +| `publish-` | BUILD + PUBLISH | one arch | only with `--publish` | +| `publish-index` | PUBLISH | multi-arch assembly (regctl) | only with `--publish` | +| `publish` | BUILD + PUBLISH | all local arches + index + `:latest` | only with `--publish` | +| `push-staged` | PUBLISH | replay `.parity-stage` | only with `--publish` | + +- `publish-index` is **build-free**: it assembles `:` from the + per-arch digest-pinned tags via `regctl index create`, and is **fail-closed** — + if a required arch was not pushed this run, it refuses to assemble a partial + index. +- `publish` runs all locally-buildable arches, then the index, then copies + `:` → `:latest` as the **LAST** (single, idempotent) mutation. +- `push-staged` replays artifacts from `./.parity-stage` to the registry, for + when the cluster was down at build time. + +Single-arch archetypes (PyPI wheel, npm addon, generic/Go binary, Helm chart) +have no multi-arch index, so they expose `stage-` / `publish-` / +`publish` / `push-staged` only. The nix2container (OCI) builder is the one that +yields the full `publish-index` / `:latest` set. + +## Shared building blocks (also exposed) + +All builders source one shell library (`ci/parity-lib.sh`, materialized in the +Nix store) so behavior cannot drift between repos: + +- **Token resolution** — `$REGISTRY_TOKEN` → `pass infra/gitea/...` fallback → + named hard-fail. The token is **never printed** and scripts run under + `set -euo pipefail` only (never `set -x`). +- **Dev-tag guard** — refuses a real (`:latest`/release) publish unless + `$VERSION` is set or `$CI_COMMIT_TAG` is a `v*` tag. +- **Version derivation** — `$VERSION` → strip leading `v` + trailing `-N` from + `$CI_COMMIT_TAG` → the flake's pinned default. Identical for CI and local. + +## API — `lib.*` + +`lib` is system-independent. Two ways to consume it: + +```nix +# (a) one call, all builders, with your own pkgs: +parity.lib.mkParityBuilders pkgs # -> { mkPyPiWheelPublish, ... } + +# (b) per-builder wrapper that takes pkgs as the first argument: +parity.lib.mkPyPiWheelPublish pkgs { pname = "..."; version = "..."; ... } +``` + +Exposed attrs: + +- `lib.mkParityBuilders` — `pkgs -> { the six builders + mkApp + shellLib }` +- `lib.mkPyPiWheelPublish` — `pkgs -> args -> { apps }` +- `lib.mkS390xNpmPublish` — `pkgs -> args -> { apps }` +- `lib.mkGenericBinaryPublish` — `pkgs -> args -> { apps }` +- `lib.mkNix2ContainerPublish` — `pkgs -> args -> { apps }` +- `lib.mkGoBinaryPublish` — `pkgs -> args -> { apps }` (alias of generic-binary) +- `lib.mkHelmPublish` — `pkgs -> args -> { apps }` + +### Builder arguments + +```nix +mkPyPiWheelPublish { + pname = "asyncpg"; + version = "0.31.0"; # default; overridden by $VERSION / $CI_COMMIT_TAG + wheel = self.packages.x86_64-linux.default; # drv/dir to glob *.whl, or a .whl path + arch = "s390x"; # default +} + +mkS390xNpmPublish { + pname = "@rollup/rollup-linux-s390x-gnu"; + version = "4.0.0"; + nodeFile = self.packages.x86_64-linux.addon; # the *.node + nodeFileName = "rollup.linux-s390x-gnu.node"; + packageJson = ''{ "name": "@rollup/...", "version": "$VERSION", ... }''; +} + +mkGenericBinaryPublish { # mkGoBinaryPublish is an alias + pname = "geesefs"; + version = "0.43.5"; + binary = "${self.packages.x86_64-linux.default}/bin/geesefs"; + assetName = "geesefs-linux-s390x"; + arch = "s390x"; +} + +mkNix2ContainerPublish { + imageName = "git.oleks.space/oleks/nix-ci"; + version = "1.2.3"; + images = { # only arches buildable on THIS host + amd64 = { copyTo = self.packages.x86_64-linux.nix-ci.copyTo; }; + }; + arches = [ "amd64" "arm64" ]; # the full set the index must cover (fail-closed) +} + +mkHelmPublish { + pname = "firecrawl"; + version = "0.1.3"; + chartSrc = ./charts/firecrawl; + ociRepo = "oci://git.oleks.space/oleks"; # default +} +``` + +Common optional args on every builder: `registryHost` (`git.oleks.space`), +`registryOwner` (`oleks`), `passEntry` +(`infra/gitea/personal_access_token_packages_rw`). + +### Consuming it in a parity repo's flake + +```nix +{ + inputs.parity.url = "git+https://git.oleks.space/oleks/parity-lib"; + outputs = { self, nixpkgs, parity, flake-utils, ... }: + flake-utils.lib.eachDefaultSystem (system: + let pkgs = import nixpkgs { inherit system; }; + in { + apps = parity.lib.mkGenericBinaryPublish pkgs { + pname = "geesefs"; + version = "0.43.5"; + binary = "${self.packages.${system}.default}/bin/geesefs"; + assetName = "geesefs-linux-s390x"; + }; + }); +} +``` + +`.woodpecker.yaml` stays thin — it runs the identical app, so CI and local +cannot drift: + +```yaml +commands: + - PUBLISH=1 nix run .#publish +``` + +## pipeline-doctor (cluster #193) + +`nix run .#pipeline-doctor -- ` asserts the parity contract on a +repo: archetype declared, consumes parity-lib, token = +`$REGISTRY_TOKEN` + `pass` fallback and never printed, dev-tag guard present, a +`--dry-run` default exists, apps carry `meta.description`, and an enumerable +`publish-*` / `stage-*` naming. It prints the local-equivalent commands. It is +read-only (no token, no registry contact) and exits non-zero if any required +check fails. + +## Verifying changes locally + +```bash +nix eval .#lib --apply builtins.attrNames --json +nix flake show +nix build .#pipeline-doctor +shellcheck ci/*.sh +statix check . +markdownlint-cli2 README.md +``` + +The git.oleks.space server runs a pre-receive linter; the commands above mirror +it. diff --git a/ci/parity-lib.sh b/ci/parity-lib.sh new file mode 100644 index 0000000..fe1e052 --- /dev/null +++ b/ci/parity-lib.sh @@ -0,0 +1,149 @@ +# shellcheck shell=bash +# shellcheck disable=SC2034 # vars/functions here are consumed by sourcing apps +# +# parity-lib shared shell helpers (cluster #192, emmett#44). +# +# This file is the single source of truth for the cross-cutting concerns every +# parity publish app shares: token resolution, version derivation, the dev-tag +# guard, argument parsing (the dry-run gate), registry preflight and the +# on-disk stage directory convention. Each archetype builder in lib/builders.nix +# generates a small writeShellApplication that sources THIS file and then does +# only its archetype-specific build + push. +# +# TOKEN HYGIENE: nothing here ever echoes the token. Scripts that source this +# MUST run under `set -euo pipefail` only and MUST NOT enable `set -x`. + +# --------------------------------------------------------------------------- +# Conventions (overridable by the sourcing app before calling the helpers). +# --------------------------------------------------------------------------- +PARITY_PASS_ENTRY="${PARITY_PASS_ENTRY:-infra/gitea/personal_access_token_packages_rw}" +PARITY_REGISTRY_HOST="${PARITY_REGISTRY_HOST:-git.oleks.space}" +PARITY_REGISTRY_OWNER="${PARITY_REGISTRY_OWNER:-oleks}" +# Local on-disk stage. BUILD-parity artifacts land here; push-staged replays them. +PARITY_STAGE_DIR="${PARITY_STAGE_DIR:-${PWD}/.parity-stage}" + +# --------------------------------------------------------------------------- +# Token resolution: $REGISTRY_TOKEN -> `pass` fallback -> named hard fail. +# Prints the token on stdout so a caller can `tok="$(parity_resolve_token)"`. +# The CALLER must capture it and never echo it. This function itself is silent. +# --------------------------------------------------------------------------- +parity_resolve_token() { + if [ -n "${REGISTRY_TOKEN:-}" ]; then + printf '%s' "$REGISTRY_TOKEN" + return 0 + fi + if command -v pass >/dev/null 2>&1; then + local t + if t="$(pass show "$PARITY_PASS_ENTRY" 2>/dev/null || true)" && [ -n "$t" ]; then + printf '%s' "$t" + return 0 + fi + fi + echo "BLOCKER(empty-token): set \$REGISTRY_TOKEN (CI from_secret) or store '$PARITY_PASS_ENTRY' in pass; refusing to publish without credentials." >&2 + return 1 +} + +# --------------------------------------------------------------------------- +# Version derivation, identical for CI and local. +# Precedence: $VERSION -> strip-v + strip-trailing-increment of $CI_COMMIT_TAG +# -> the default passed as $1 (the flake's pinned version). +# Tag shape accepted: v optionally with a - build increment suffix. +# Prints the derived version on stdout. +# --------------------------------------------------------------------------- +parity_derive_version() { + local default_version="${1:-}" + if [ -n "${VERSION:-}" ]; then + printf '%s' "$VERSION" + return 0 + fi + if [ -n "${CI_COMMIT_TAG:-}" ]; then + printf '%s' "$CI_COMMIT_TAG" | sed 's/^v//; s/-[0-9]*$//' + return 0 + fi + if [ -n "$default_version" ]; then + printf '%s' "$default_version" + return 0 + fi + echo "BLOCKER(no-version): no \$VERSION, no \$CI_COMMIT_TAG and no default version." >&2 + return 1 +} + +# --------------------------------------------------------------------------- +# Dev-tag guard. Refuse a real (registry-mutating, :latest/release) publish +# unless an explicit $VERSION is set OR $CI_COMMIT_TAG looks like a v* tag. +# This stops an accidental local `--publish` from clobbering the registry with +# the flake's default development version. +# --------------------------------------------------------------------------- +parity_devtag_guard() { + if [ -n "${VERSION:-}" ]; then + return 0 + fi + if printf '%s' "${CI_COMMIT_TAG:-}" | grep -Eq '^v[0-9]'; then + return 0 + fi + echo "BLOCKER(dev-tag): refusing to publish without an explicit version." >&2 + echo " Set VERSION= or push a v* tag (CI_COMMIT_TAG) before publishing." >&2 + return 1 +} + +# --------------------------------------------------------------------------- +# Argument gate. Sets the global PARITY_PUBLISH (0 = dry-run default, 1 = push). +# Honors $PUBLISH=1, --publish and --dry-run. --help prints usage and exits 0. +# Usage: parity_parse_args "" "$@" +# --------------------------------------------------------------------------- +parity_parse_args() { + local desc="$1" + shift + PARITY_PUBLISH="${PUBLISH:-0}" + local a + for a in "$@"; do + case "$a" in + --publish) PARITY_PUBLISH=1 ;; + --dry-run) PARITY_PUBLISH=0 ;; + -h | --help) + echo "$desc" + echo "" + echo "Dry-run by default: builds/stages and prints what WOULD be pushed." + echo " --publish | PUBLISH=1 actually mutate the registry" + echo " --dry-run force dry-run" + echo "Env: VERSION= override version; REGISTRY_TOKEN registry token" + echo " (fallback: pass $PARITY_PASS_ENTRY)." + exit 0 + ;; + *) + echo "error: unknown argument '$a' (try --help)" >&2 + exit 2 + ;; + esac + done +} + +# --------------------------------------------------------------------------- +# Registry preflight: name the blocker if the registry is unreachable BEFORE +# attempting any push (CI shares fate with the cluster the registry lives on). +# --------------------------------------------------------------------------- +parity_registry_preflight() { + if ! curl -fsS -o /dev/null --max-time 10 "https://${PARITY_REGISTRY_HOST}/api/v1/version" 2>/dev/null; then + echo "BLOCKER(registry-down): https://${PARITY_REGISTRY_HOST} unreachable — re-run when the registry is back; staged artifacts in ${PARITY_STAGE_DIR} are unchanged." >&2 + return 1 + fi +} + +# --------------------------------------------------------------------------- +# Stage directory helpers. The convention: each staged artifact is written to +# ${PARITY_STAGE_DIR}// alongside a one-line meta file recording the +# version + intended registry coordinates, so push-staged can replay later. +# --------------------------------------------------------------------------- +parity_stage_path() { + # $1 = arch + printf '%s/%s' "$PARITY_STAGE_DIR" "$1" +} + +parity_stage_reset() { + # $1 = arch ; wipes + recreates that arch's stage dir, prints the path. + local d + d="$(parity_stage_path "$1")" + rm -rf "$d" + mkdir -p "$d" + printf '%s' "$d" +} diff --git a/ci/pipeline-doctor.sh b/ci/pipeline-doctor.sh new file mode 100755 index 0000000..b1508b4 --- /dev/null +++ b/ci/pipeline-doctor.sh @@ -0,0 +1,131 @@ +#!/usr/bin/env bash +# pipeline-doctor (cluster #193): assert the parity contract for a repo. +# +# Given a repo path, it statically checks that the repo follows the corrected +# emmett#44 parity standard, and prints the local-equivalent commands a dev can +# run. It is read-only: it never touches the registry and never needs a token. +# +# Usage: pipeline-doctor [] (default: .) +# Exit: 0 = all required checks pass; 1 = one or more required checks failed. +set -euo pipefail + +REPO="${1:-.}" +REPO="$(cd "$REPO" && pwd)" +FLAKE="$REPO/flake.nix" + +fail=0 +pass_n=0 +note() { printf ' %s %s\n' "$1" "$2"; } +ok() { + pass_n=$((pass_n + 1)) + note "PASS" "$1" +} +bad() { + fail=$((fail + 1)) + note "FAIL" "$1" +} +warn() { note "WARN" "$1"; } + +echo "pipeline-doctor: $REPO" + +if [ ! -f "$FLAKE" ]; then + echo "FAIL: no flake.nix at $REPO" >&2 + exit 1 +fi + +flake_txt="$(cat "$FLAKE")" + +# 1. archetype declared (a publish/stage app whose name encodes the arch, or an +# explicit archetype marker comment). +if printf '%s' "$flake_txt" | grep -Eq 'archetype|stage-[a-z0-9_]+|publish-[a-z0-9_]+'; then + ok "archetype declared (stage-*/publish-* app or archetype marker present)" +else + bad "no archetype: expected a stage-/publish- app or 'archetype' marker" +fi + +# 2. consumes parity-lib (the shared module). +if printf '%s' "$flake_txt" | grep -Eq 'parity-lib|parity\.lib|mk(PyPiWheel|S390xNpm|GenericBinary|Nix2Container|GoBinary|Helm)Publish'; then + ok "consumes parity-lib (input or one of the mk*Publish builders)" +else + bad "does not consume parity-lib: inline the input and use a mk*Publish builder" +fi + +# 3. token = $REGISTRY_TOKEN with pass fallback, and never printed. +token_src="$flake_txt" +if [ -d "$REPO/ci" ]; then + token_src="$token_src +$(cat "$REPO"/ci/*.sh 2>/dev/null || true)" +fi +if printf '%s' "$token_src" | grep -Eq 'REGISTRY_TOKEN' && + printf '%s' "$token_src" | grep -Eq 'parity_resolve_token|pass (show )?infra/gitea/personal_access_token_packages_rw'; then + ok "token = \$REGISTRY_TOKEN with pass fallback" +else + bad "token contract missing: need \$REGISTRY_TOKEN + pass fallback (parity_resolve_token)" +fi +# token never printed: flag an obvious leak. A redacting pipe (sed s/$tok/.../) +# is fine, so only flag a bare echo/printf of the token NOT piped to a redactor, +# or an enabled `set -x` (which would trace the token). The leak pattern uses a +# literal '$' built via a variable so this lib stays single-quote clean (SC2016). +dollar='[$]' +# Anchor on echo/printf used as a COMMAND (line-leading, after `;`, `|`, `&&`, +# `(` or `then/do/else`) so the doctor doesn't flag the regex literals it +# carries inside string assignments. +leak_pat="(^|[;|&(]|then |do |else )[[:space:]]*(echo|printf)[^|]*${dollar}(REGISTRY_TOKEN|TOKEN|tok|token)([^A-Za-z0-9_]|$)" +leak=0 +if printf '%s' "$token_src" | grep -Eq "^[[:space:]]*set -x([[:space:]]|$)"; then leak=1; fi +# Exclude: redactions, stdin-feeds, the doctor's own $token_src var, an escaped +# '\$' (a token NAME in a message, not its value), and the sanctioned +# `printf '%s' "$TOKEN"` capture idiom (the resolver returns the token on stdout +# for a caller to capture — that is the one blessed place a token is emitted). +if printf '%s' "$token_src" | + grep -E "$leak_pat" | + grep -vE "REDACTED|sed |stdin|token_src" | + grep -vqE 'printf .%s. "[$](REGISTRY_TOKEN|TOKEN|tok|token)"|\\[$]'; then + leak=1 +fi +if [ "$leak" -eq 0 ]; then + ok "no obvious token leak (token never bare-echoed; no set -x)" +else + bad "possible token leak: a token var is echo/printf'd un-redacted, or set -x is enabled" +fi + +# 4. dev-tag guard present. +if printf '%s' "$token_src" | grep -Eq 'parity_devtag_guard|refusing to publish without an explicit'; then + ok "dev-tag guard present" +else + bad "no dev-tag guard: a default-version --publish could clobber the registry" +fi + +# 5. a --dry-run default exists. +if printf '%s' "$token_src" | grep -Eq 'parity_parse_args|DRY-RUN|dry-run|PUBLISH:-0|PUBLISH=\$\{PUBLISH:-0\}'; then + ok "dry-run default exists (publish mutates only on --publish/PUBLISH=1)" +else + bad "no dry-run default: publish must NOT mutate the registry by default" +fi + +# 6. apps carry meta.description. +if printf '%s' "$flake_txt" | grep -Eq 'meta\.description|meta = \{'; then + ok "apps carry meta.description" +else + bad "apps missing meta.description" +fi + +# 7. enumerable publish-* naming. +pubs="$(printf '%s' "$flake_txt" | grep -oE '(stage|publish)(-[a-z0-9_]+)?' | sort -u || true)" +if [ -n "$pubs" ]; then + ok "enumerable publish-* / stage-* app naming:" + printf '%s\n' "$pubs" | sed 's/^/ /' +else + bad "no enumerable publish-*/stage-* apps found" +fi + +echo "" +echo "Local-equivalent commands (what CI runs is identical):" +echo " nix run .#stage- # BUILD-parity: stage to .parity-stage, no registry" +echo " nix run .#publish- # stage + push that arch (DRY-RUN; add --publish)" +echo " nix run .#publish-index # build-free multi-arch assembly from pushed digests" +echo " nix run .#publish # all local arches + index; :latest = digest copy of :TAG" +echo " nix run .#push-staged # replay .parity-stage to the registry (cluster-was-down)" +echo "" +echo "Summary: $pass_n passed, $fail failed." +[ "$fail" -eq 0 ] diff --git a/flake.lock b/flake.lock new file mode 100644 index 0000000..efa8062 --- /dev/null +++ b/flake.lock @@ -0,0 +1,119 @@ +{ + "nodes": { + "flake-utils": { + "inputs": { + "systems": "systems" + }, + "locked": { + "lastModified": 1731533236, + "narHash": "sha256-l0KFg5HjrsfsO/JpG+r7fRrqm12kzFHyUHqHCVpMMbI=", + "owner": "numtide", + "repo": "flake-utils", + "rev": "11707dc2f618dd54ca8739b309ec4fc024de578b", + "type": "github" + }, + "original": { + "owner": "numtide", + "repo": "flake-utils", + "type": "github" + } + }, + "fleet": { + "inputs": { + "nixpkgs": "nixpkgs", + "nixpkgs-armer": [ + "fleet", + "nixpkgs" + ], + "nixpkgs-bim": [ + "fleet", + "nixpkgs" + ], + "nixpkgs-ci": [ + "fleet", + "nixpkgs" + ], + "nixpkgs-emmett": [ + "fleet", + "nixpkgs" + ], + "nixpkgs-howard": [ + "fleet", + "nixpkgs" + ], + "nixpkgs-mermaid": [ + "fleet", + "nixpkgs" + ], + "nixpkgs-mermaid-gpu": [ + "fleet", + "nixpkgs" + ], + "nixpkgs-micron": [ + "fleet", + "nixpkgs" + ], + "nixpkgs-projects": [ + "fleet", + "nixpkgs" + ] + }, + "locked": { + "lastModified": 1779533061, + "narHash": "sha256-orWNYXtYURhEj3X4+xGMAhaEcKRvwXqTtJ8x2jV/M+Q=", + "ref": "refs/heads/main", + "rev": "b818e345ec4470e4b3e335bd2f864183c512116d", + "revCount": 13, + "type": "git", + "url": "https://git.oleks.space/oleks/fleet-pins" + }, + "original": { + "type": "git", + "url": "https://git.oleks.space/oleks/fleet-pins" + } + }, + "nixpkgs": { + "locked": { + "lastModified": 1777268161, + "narHash": "sha256-bxrdOn8SCOv8tN4JbTF/TXq7kjo9ag4M+C8yzzIRYbE=", + "owner": "NixOS", + "repo": "nixpkgs", + "rev": "1c3fe55ad329cbcb28471bb30f05c9827f724c76", + "type": "github" + }, + "original": { + "owner": "NixOS", + "repo": "nixpkgs", + "rev": "1c3fe55ad329cbcb28471bb30f05c9827f724c76", + "type": "github" + } + }, + "root": { + "inputs": { + "flake-utils": "flake-utils", + "fleet": "fleet", + "nixpkgs": [ + "fleet", + "nixpkgs-ci" + ] + } + }, + "systems": { + "locked": { + "lastModified": 1681028828, + "narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=", + "owner": "nix-systems", + "repo": "default", + "rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e", + "type": "github" + }, + "original": { + "owner": "nix-systems", + "repo": "default", + "type": "github" + } + } + }, + "root": "root", + "version": 7 +} diff --git a/flake.nix b/flake.nix new file mode 100644 index 0000000..365bf13 --- /dev/null +++ b/flake.nix @@ -0,0 +1,88 @@ +{ + description = "parity-lib — shared per-archetype publish-app builders for the ~51 parity repos (cluster #192/#193/#194, emmett#44)"; + + inputs = { + # Shared fleet pin so the ~51 consumers stay binary-cache aligned. + fleet.url = "git+https://git.oleks.space/oleks/fleet-pins"; + nixpkgs.follows = "fleet/nixpkgs-ci"; + flake-utils.url = "github:numtide/flake-utils"; + }; + + outputs = + { + self, + nixpkgs, + flake-utils, + ... + }: + { + # --------------------------------------------------------------------- + # lib.* — system-INDEPENDENT entry points. A consumer calls + # parity.lib.mkParityBuilders { pkgs = ; } + # and gets the six mk*Publish builders back, OR uses the per-builder + # convenience wrappers below which take pkgs as the first argument. + # --------------------------------------------------------------------- + lib = + let + builders = pkgs: import ./lib/builders.nix { inherit pkgs; }; + wrap = name: pkgs: args: (builders pkgs).${name} args; + in + { + mkParityBuilders = builders; + mkPyPiWheelPublish = wrap "mkPyPiWheelPublish"; + mkS390xNpmPublish = wrap "mkS390xNpmPublish"; + mkGenericBinaryPublish = wrap "mkGenericBinaryPublish"; + mkNix2ContainerPublish = wrap "mkNix2ContainerPublish"; + mkGoBinaryPublish = wrap "mkGoBinaryPublish"; + mkHelmPublish = wrap "mkHelmPublish"; + }; + } + // flake-utils.lib.eachDefaultSystem ( + system: + let + pkgs = import nixpkgs { inherit system; }; + inherit (pkgs) lib; + + pipelineDoctor = pkgs.writeShellApplication { + name = "pipeline-doctor"; + runtimeInputs = with pkgs; [ + coreutils + gnugrep + gnused + findutils + jq + ]; + text = ''exec bash ${./ci/pipeline-doctor.sh} "$@"''; + }; + + # Smoke check: instantiate every builder with stub args so the apps eval. + builders = import ./lib/builders.nix { inherit pkgs; }; + smoke = pkgs.runCommand "parity-lib-smoke" { } '' + : "${ + builtins.toString ( + lib.attrNames (builders.mkPyPiWheelPublish { + pname = "demo"; + version = "0.0.0"; + wheel = pkgs.hello; + }) + ) + }" + touch $out + ''; + in + { + packages = { + pipeline-doctor = pipelineDoctor; + default = pipelineDoctor; + }; + + apps.pipeline-doctor = { + type = "app"; + program = lib.getExe pipelineDoctor; + meta.description = "Assert the parity contract for a repo and print local-equivalent commands (cluster #193)."; + }; + + checks.smoke = smoke; + } + ); +} diff --git a/lib/builders.nix b/lib/builders.nix new file mode 100644 index 0000000..3ba5f29 --- /dev/null +++ b/lib/builders.nix @@ -0,0 +1,742 @@ +# Per-archetype publish-app BUILDERS (cluster #192/#194, emmett#44). +# +# `mkParityBuilders { pkgs }` returns the six mk*Publish builders plus the +# shared building blocks. Each mk*Publish takes a small attrset and returns an +# attrset of flake apps following the corrected parity standard: +# +# stage- BUILD-parity, cluster-independent: writes the artifact to the +# on-disk stage (./.parity-stage/), NO registry contact. +# publish- stage + push that arch. DRY-RUN by default (--publish to push). +# publish-index build-free multi-arch assembly from digest-pinned refs +# (regctl), fail-closed if a required arch was not pushed. +# publish all locally-buildable arches + publish-index; ':latest' is a +# digest copy of ':TAG' done LAST as the idempotent mutation. +# push-staged replay artifacts from ./.parity-stage to the registry. +# +# Not every archetype yields every app: a single-arch wheel/binary/npm addon +# has no multi-arch index, so it exposes stage/publish/push-staged only. The OCI +# (nix2container) builder is the one that yields the full index/latest set. +{ pkgs }: +let + inherit (pkgs) lib; + + shellLib = ../ci/parity-lib.sh; + + # Common preamble: source the shared shell helpers from the store and apply + # any per-app coordinate overrides BEFORE the helpers are used. + preamble = + { + registryHost ? "git.oleks.space", + registryOwner ? "oleks", + passEntry ? "infra/gitea/personal_access_token_packages_rw", + }: + '' + set -euo pipefail + # shellcheck source=/dev/null + . ${shellLib} + # Coordinate overrides are read by the sourced lib; export so shellcheck + # treats them as used (SC2034) and child processes inherit them. + export PARITY_REGISTRY_HOST=${lib.escapeShellArg registryHost} + export PARITY_REGISTRY_OWNER=${lib.escapeShellArg registryOwner} + export PARITY_PASS_ENTRY=${lib.escapeShellArg passEntry} + export PARITY_STAGE_DIR="''${PARITY_STAGE_DIR:-''${PWD}/.parity-stage}" + ''; + + mkApp = drv: desc: { + type = "app"; + program = lib.getExe drv; + meta.description = desc; + }; + + baseInputs = with pkgs; [ + coreutils + curl + findutils + gnugrep + gnused + pass + ]; + + # ========================================================================= + # PyPI wheel (single-arch). Stage a built wheel into .parity-stage, then + # POST it to the Gitea PyPI registry (idempotent: 409 = already present). + # Args: { pname, version|versionFn, wheel (drv|path producing *.whl), + # arch ? "s390x", registryHost?, registryOwner?, passEntry? } + # ========================================================================= + mkPyPiWheelPublish = + { + pname, + version, + wheel, + arch ? "s390x", + registryHost ? "git.oleks.space", + registryOwner ? "oleks", + passEntry ? "infra/gitea/personal_access_token_packages_rw", + extractCmd ? null, + }: + let + head = preamble { inherit registryHost registryOwner passEntry; }; + # The wheel input may be a directory (Nix store output) we glob for *.whl, + # or an explicit single .whl path. + wheelRef = "${wheel}"; + stageText = '' + ${head} + VERSION="$(parity_derive_version ${lib.escapeShellArg version})" + echo "→ staging ${pname} $VERSION wheel (${arch}, no registry contact)" + d="$(parity_stage_reset ${lib.escapeShellArg arch})" + src=${lib.escapeShellArg wheelRef} + if [ -d "$src" ]; then + # shellcheck disable=SC2044 + for w in $(find "$src" -name '*.whl'); do cp "$w" "$d/"; done + else + cp "$src" "$d/" + fi + whl="$(find "$d" -name '*.whl' | head -1)" + if [ -z "$whl" ]; then echo "BLOCKER(no-wheel): no *.whl under $src" >&2; exit 1; fi + printf 'pypi %s %s\n' "${pname}" "$VERSION" >"$d/.parity-meta" + sha256sum "$whl" + echo " staged: $whl" + ''; + stage = pkgs.writeShellApplication { + name = "stage-${arch}"; + runtimeInputs = baseInputs ++ [ pkgs.nix ]; + text = stageText; + }; + publish = pkgs.writeShellApplication { + name = "publish-${arch}"; + runtimeInputs = baseInputs ++ [ pkgs.nix ]; + text = '' + ${head} + parity_parse_args "Build + publish the ${pname} ${arch} wheel to the Gitea PyPI registry" "$@" + VERSION="$(parity_derive_version ${lib.escapeShellArg version})" + ${lib.getExe stage} + d="$(parity_stage_path ${lib.escapeShellArg arch})" + whl="$(find "$d" -name '*.whl' | head -1)" + if [ "$PARITY_PUBLISH" != "1" ]; then + echo "[dry-run] would POST $(basename "$whl") -> https://$PARITY_REGISTRY_HOST/api/packages/$PARITY_REGISTRY_OWNER/pypi (name=${pname} version=$VERSION)" + echo "[dry-run] re-run with --publish / PUBLISH=1 to upload." + exit 0 + fi + parity_devtag_guard + tok="$(parity_resolve_token)" + parity_registry_preflight + sha="$(sha256sum "$whl" | cut -d' ' -f1)" + http="$(curl -s -o /dev/null -w '%{http_code}' -X POST \ + -H "Authorization: token $tok" \ + -F ':action=file_upload' -F 'protocol_version=1' \ + -F "sha256_digest=$sha" -F "name=${pname}" -F "version=$VERSION" \ + -F 'filetype=bdist_wheel' -F "content=@$whl" \ + "https://$PARITY_REGISTRY_HOST/api/packages/$PARITY_REGISTRY_OWNER/pypi")" + case "$http" in + 200 | 201 | 409) echo "Published ${pname} $VERSION (HTTP $http)" ;; + *) echo "BLOCKER(upload-failed): POST returned HTTP $http" >&2; exit 1 ;; + esac + ''; + }; + pushStaged = mkPushStagedGeneric { + inherit + pname + arch + registryHost + registryOwner + passEntry + ; + kind = "pypi"; + }; + in + { + "stage-${arch}" = mkApp stage "Stage the ${pname} ${arch} wheel into .parity-stage (no registry contact)."; + "publish-${arch}" = mkApp publish "Stage + publish the ${pname} ${arch} wheel (dry-run by default; --publish to push)."; + "publish" = mkApp publish "Publish all locally-buildable arches (${arch} only for this PyPI repo)."; + "push-staged" = pushStaged; + }; + + # ========================================================================= + # s390x npm native addon (single-arch). Stage a ready-to-publish package dir + # (the .node + a generated package.json) then `npm publish`. + # Args: { pname (npm full name, e.g. @rollup/rollup-linux-s390x-gnu), + # version, nodeFile (drv|path to the *.node), nodeFileName, + # packageJson (string, with $version interpolated by the app), + # arch ? "s390x", registry* } + # ========================================================================= + mkS390xNpmPublish = + { + pname, + version, + nodeFile, + nodeFileName, + packageJson, + arch ? "s390x", + registryHost ? "git.oleks.space", + registryOwner ? "oleks", + passEntry ? "infra/gitea/personal_access_token_packages_rw", + }: + let + head = preamble { inherit registryHost registryOwner passEntry; }; + stage = pkgs.writeShellApplication { + name = "stage-${arch}"; + runtimeInputs = baseInputs ++ [ pkgs.nodejs ]; + text = '' + ${head} + VERSION="$(parity_derive_version ${lib.escapeShellArg version})" + echo "→ staging ${pname}@$VERSION (${arch}, no registry contact)" + d="$(parity_stage_reset ${lib.escapeShellArg arch})" + install -m 0644 ${lib.escapeShellArg "${nodeFile}"} "$d/${nodeFileName}" + cat >"$d/package.json" <"$d/.parity-meta" + echo " staged into $d:" + ls -lh "$d" + ''; + }; + publish = pkgs.writeShellApplication { + name = "publish-${arch}"; + runtimeInputs = baseInputs ++ [ pkgs.nodejs ]; + text = '' + ${head} + parity_parse_args "Build + publish the ${pname} ${arch} npm addon to the Gitea npm registry" "$@" + VERSION="$(parity_derive_version ${lib.escapeShellArg version})" + ${lib.getExe stage} + d="$(parity_stage_path ${lib.escapeShellArg arch})" + if [ "$PARITY_PUBLISH" != "1" ]; then + echo "[dry-run] would 'npm publish' $d -> https://$PARITY_REGISTRY_HOST/api/packages/$PARITY_REGISTRY_OWNER/npm/" + echo "[dry-run] re-run with --publish / PUBLISH=1 to push." + exit 0 + fi + parity_devtag_guard + tok="$(parity_resolve_token)" + parity_registry_preflight + reg="https://$PARITY_REGISTRY_HOST/api/packages/$PARITY_REGISTRY_OWNER/npm/" + ( cd "$d" + set +e + out="$(npm publish --registry="$reg" \ + "--//$PARITY_REGISTRY_HOST/api/packages/$PARITY_REGISTRY_OWNER/npm/:_authToken=$tok" 2>&1)" + rc=$? + set -e + printf '%s\n' "$out" | sed "s/$tok/***REDACTED***/g" + if [ "$rc" -ne 0 ]; then + if printf '%s' "$out" | grep -qiE 'already exists|409|conflict'; then + echo "Version already published — idempotent success."; exit 0 + fi + exit "$rc" + fi ) + echo "Published ${pname}@$VERSION" + ''; + }; + in + { + "stage-${arch}" = mkApp stage "Stage the ${pname} ${arch} npm addon into .parity-stage (no registry contact)."; + "publish-${arch}" = mkApp publish "Stage + publish the ${pname} ${arch} npm addon (dry-run by default; --publish to push)."; + "publish" = mkApp publish "Publish all locally-buildable arches (${arch} only for this npm repo)."; + "push-staged" = mkPushStagedGeneric { + inherit + pname + arch + registryHost + registryOwner + passEntry + ; + kind = "npm"; + }; + }; + + # ========================================================================= + # Generic binary (single-arch). Stage a built binary, PUT it to the Gitea + # generic registry (idempotent: 201 created / 409 exists). + # Args: { pname, version, binary (drv|path), assetName, arch ? "s390x", + # registry* } + # ========================================================================= + mkGenericBinaryPublish = + { + pname, + version, + binary, + assetName, + arch ? "s390x", + registryHost ? "git.oleks.space", + registryOwner ? "oleks", + passEntry ? "infra/gitea/personal_access_token_packages_rw", + }: + let + head = preamble { inherit registryHost registryOwner passEntry; }; + stage = pkgs.writeShellApplication { + name = "stage-${arch}"; + runtimeInputs = baseInputs ++ [ + pkgs.nix + pkgs.file + ]; + text = '' + ${head} + VERSION="$(parity_derive_version ${lib.escapeShellArg version})" + echo "→ staging ${pname} $VERSION (${arch}, no registry contact)" + d="$(parity_stage_reset ${lib.escapeShellArg arch})" + cp ${lib.escapeShellArg "${binary}"} "$d/${assetName}" + printf 'generic %s %s %s\n' "${pname}" "$VERSION" "${assetName}" >"$d/.parity-meta" + file "$d/${assetName}" 2>/dev/null || true + echo " staged: $d/${assetName}" + ''; + }; + publish = pkgs.writeShellApplication { + name = "publish-${arch}"; + runtimeInputs = baseInputs ++ [ + pkgs.nix + pkgs.file + ]; + text = '' + ${head} + parity_parse_args "Build + publish the ${pname} ${arch} binary to the Gitea generic registry" "$@" + VERSION="$(parity_derive_version ${lib.escapeShellArg version})" + ${lib.getExe stage} + d="$(parity_stage_path ${lib.escapeShellArg arch})" + if [ "$PARITY_PUBLISH" != "1" ]; then + echo "[dry-run] would PUT $d/${assetName} -> https://$PARITY_REGISTRY_HOST/api/packages/$PARITY_REGISTRY_OWNER/generic/${pname}/$VERSION/${assetName}" + echo "[dry-run] re-run with --publish / PUBLISH=1 to upload." + exit 0 + fi + parity_devtag_guard + tok="$(parity_resolve_token)" + parity_registry_preflight + http="$(curl -s -o /dev/null -w '%{http_code}' -X PUT \ + -H "Authorization: token $tok" --upload-file "$d/${assetName}" \ + "https://$PARITY_REGISTRY_HOST/api/packages/$PARITY_REGISTRY_OWNER/generic/${pname}/$VERSION/${assetName}")" + case "$http" in + 201 | 409) echo "Published ${assetName} $VERSION (HTTP $http)" ;; + *) echo "BLOCKER(upload-failed): PUT returned HTTP $http" >&2; exit 1 ;; + esac + ''; + }; + in + { + "stage-${arch}" = mkApp stage "Stage the ${pname} ${arch} binary into .parity-stage (no registry contact)."; + "publish-${arch}" = mkApp publish "Stage + publish the ${pname} ${arch} binary (dry-run by default; --publish to push)."; + "publish" = mkApp publish "Publish all locally-buildable arches (${arch} only for this generic-binary repo)."; + "push-staged" = mkPushStagedGeneric { + inherit + pname + arch + assetName + registryHost + registryOwner + passEntry + ; + kind = "generic"; + inherit version; + }; + }; + + # ========================================================================= + # Go binary (single-arch generic). Thin alias over the generic-binary builder + # — a cross-compiled Go binary is published to the generic registry exactly + # like any other binary. Kept as a distinct named builder so the archetype is + # explicit in consumer flakes and pipeline-doctor. + # ========================================================================= + mkGoBinaryPublish = mkGenericBinaryPublish; + + # ========================================================================= + # Helm chart (single artifact). Stage `helm package` output (.tgz) then + # `helm push` to the OCI registry (idempotent: a duplicate version errors, + # which we treat as success). + # Args: { pname (chart name), version, chartSrc (drv|path to chart dir), + # ociRepo ? "oci://git.oleks.space/oleks", registry* } + # ========================================================================= + mkHelmPublish = + { + pname, + version, + chartSrc, + ociRepo ? "oci://git.oleks.space/oleks", + registryHost ? "git.oleks.space", + registryOwner ? "oleks", + passEntry ? "infra/gitea/personal_access_token_packages_rw", + }: + let + head = preamble { inherit registryHost registryOwner passEntry; }; + stage = pkgs.writeShellApplication { + name = "stage-chart"; + runtimeInputs = baseInputs ++ [ pkgs.kubernetes-helm ]; + text = '' + ${head} + VERSION="$(parity_derive_version ${lib.escapeShellArg version})" + echo "→ staging helm chart ${pname} $VERSION (no registry contact)" + d="$(parity_stage_reset chart)" + helm package ${lib.escapeShellArg "${chartSrc}"} --version "$VERSION" --app-version "$VERSION" --destination "$d" + printf 'helm %s %s\n' "${pname}" "$VERSION" >"$d/.parity-meta" + echo " staged:"; ls -lh "$d"/*.tgz + ''; + }; + publish = pkgs.writeShellApplication { + name = "publish-chart"; + runtimeInputs = baseInputs ++ [ pkgs.kubernetes-helm ]; + text = '' + ${head} + parity_parse_args "Package + publish the ${pname} helm chart to ${ociRepo}" "$@" + VERSION="$(parity_derive_version ${lib.escapeShellArg version})" + ${lib.getExe stage} + d="$(parity_stage_path chart)" + tgz="$(find "$d" -name '*.tgz' | head -1)" + if [ "$PARITY_PUBLISH" != "1" ]; then + echo "[dry-run] would 'helm push $(basename "$tgz")' -> ${ociRepo}" + echo "[dry-run] re-run with --publish / PUBLISH=1 to push." + exit 0 + fi + parity_devtag_guard + tok="$(parity_resolve_token)" + parity_registry_preflight + echo "$tok" | helm registry login "$PARITY_REGISTRY_HOST" -u "$PARITY_REGISTRY_OWNER" --password-stdin >/dev/null 2>&1 + set +e + out="$(helm push "$tgz" ${lib.escapeShellArg ociRepo} 2>&1)" + rc=$? + set -e + printf '%s\n' "$out" | sed "s/$tok/***REDACTED***/g" + if [ "$rc" -ne 0 ]; then + if printf '%s' "$out" | grep -qiE 'already exists|409|conflict'; then + echo "Chart version already published — idempotent success."; exit 0 + fi + exit "$rc" + fi + echo "Published chart ${pname} $VERSION" + ''; + }; + in + { + "stage-chart" = mkApp stage "Package the ${pname} helm chart into .parity-stage (no registry contact)."; + "publish-chart" = mkApp publish "Package + publish the ${pname} helm chart (dry-run by default; --publish to push)."; + "publish" = mkApp publish "Publish the ${pname} helm chart."; + "push-staged" = mkPushStagedHelm { + inherit + pname + ociRepo + registryHost + registryOwner + passEntry + ; + }; + }; + + # ========================================================================= + # nix2container OCI image (MULTI-ARCH). This is the archetype that yields the + # full stage/publish-/publish-index/publish/push-staged set. + # + # Args: { + # imageName, # e.g. git.oleks.space/oleks/nix-ci + # version|tagFn, # the immutable :TAG + # images, # { = { copyTo = ; }; } + # # only arches buildable on THIS host should appear + # arches ? attrNames images, # the full set the index must cover (fail-closed) + # registry* } + # + # Per-arch publish copies the arch image to :-. + # publish-index assembles : from the per-arch digest-pinned + # tags via `regctl index create`, failing closed if any required arch tag is + # missing. publish runs all local arches, then the index, then copies + # : -> :latest as the LAST idempotent mutation. + # ========================================================================= + mkNix2ContainerPublish = + { + imageName, + version, + images, + arches ? lib.attrNames images, + registryHost ? "git.oleks.space", + registryOwner ? "oleks", + passEntry ? "infra/gitea/personal_access_token_packages_rw", + }: + let + head = preamble { inherit registryHost registryOwner passEntry; }; + localArches = lib.attrNames images; + + # per-arch stage: realise the image's copy-to closure locally (no push). + mkArchStage = + arch: + pkgs.writeShellApplication { + name = "stage-${arch}"; + runtimeInputs = baseInputs ++ [ pkgs.nix ]; + text = '' + ${head} + VERSION="$(parity_derive_version ${lib.escapeShellArg version})" + echo "→ staging ${imageName}:$VERSION-${arch} (build closure only, no push)" + d="$(parity_stage_reset ${lib.escapeShellArg arch})" + # Realise the nix2container copyTo wrapper (its closure is the image). + out=${lib.escapeShellArg "${images.${arch}.copyTo}"} + printf 'oci %s %s %s\n' ${lib.escapeShellArg imageName} "$VERSION" "${arch}" >"$d/.parity-meta" + printf '%s\n' "$out" >"$d/copy-to-path" + echo " staged copyTo: $out" + ''; + }; + + mkArchPublish = + arch: + pkgs.writeShellApplication { + name = "publish-${arch}"; + runtimeInputs = baseInputs ++ [ pkgs.nix ]; + text = '' + ${head} + parity_parse_args "Build + push ${imageName}:-${arch}" "$@" + VERSION="$(parity_derive_version ${lib.escapeShellArg version})" + ${lib.getExe (mkArchStage arch)} + if [ "$PARITY_PUBLISH" != "1" ]; then + echo "[dry-run] would copy ${imageName}:$VERSION-${arch} (digest-pinned per-arch tag)" + echo "[dry-run] re-run with --publish / PUBLISH=1 to push." + exit 0 + fi + parity_devtag_guard + tok="$(parity_resolve_token)" + parity_registry_preflight + creds="$PARITY_REGISTRY_OWNER:$tok" + ${lib.escapeShellArg "${images.${arch}.copyTo}"}/bin/copy-to \ + --dest-creds "$creds" \ + "docker://${imageName}:$VERSION-${arch}" + echo "Pushed ${imageName}:$VERSION-${arch}" + ''; + }; + + archStageApps = lib.listToAttrs ( + map (a: { + name = "stage-${a}"; + value = mkApp (mkArchStage a) "Stage the ${imageName} ${a} image closure into .parity-stage (no push)."; + }) localArches + ); + archPublishApps = lib.listToAttrs ( + map (a: { + name = "publish-${a}"; + value = mkApp (mkArchPublish a) "Build + push ${imageName}:-${a} (dry-run by default; --publish to push)."; + }) localArches + ); + + # build-free multi-arch index from the per-arch digest-pinned tags. + indexApp = pkgs.writeShellApplication { + name = "publish-index"; + runtimeInputs = baseInputs ++ [ pkgs.regclient ]; + text = '' + ${head} + parity_parse_args "Assemble the multi-arch index ${imageName}: from pushed per-arch tags" "$@" + VERSION="$(parity_derive_version ${lib.escapeShellArg version})" + req=(${lib.concatStringsSep " " (map lib.escapeShellArg arches)}) + if [ "$PARITY_PUBLISH" != "1" ]; then + echo "[dry-run] would 'regctl index create ${imageName}:$VERSION' from:" + for a in "''${req[@]}"; do echo " ${imageName}:$VERSION-$a"; done + echo "[dry-run] re-run with --publish / PUBLISH=1 to assemble." + exit 0 + fi + parity_devtag_guard + tok="$(parity_resolve_token)" + parity_registry_preflight + regctl registry login "$PARITY_REGISTRY_HOST" -u "$PARITY_REGISTRY_OWNER" --pass-stdin <<<"$tok" >/dev/null 2>&1 + # Fail-closed: every required arch tag must exist before assembly. + refs=() + for a in "''${req[@]}"; do + ref="${imageName}:$VERSION-$a" + if ! regctl manifest head "$ref" >/dev/null 2>&1; then + echo "BLOCKER(missing-arch): required arch tag $ref was not pushed this run; refusing to assemble a partial index." >&2 + exit 1 + fi + refs+=("$ref") + done + regctl index create "${imageName}:$VERSION" "''${refs[@]/#/--ref=}" + echo "Assembled index ${imageName}:$VERSION" + ''; + }; + + # publish: all locally-buildable arches -> index -> :latest (last). + publishAll = pkgs.writeShellApplication { + name = "publish"; + runtimeInputs = baseInputs ++ [ + pkgs.nix + pkgs.regclient + ]; + text = '' + ${head} + parity_parse_args "Publish all local arches of ${imageName} + the multi-arch index; :latest = digest copy of :TAG" "$@" + VERSION="$(parity_derive_version ${lib.escapeShellArg version})" + fwd=() + [ "$PARITY_PUBLISH" = "1" ] && fwd=(--publish) + ${lib.concatMapStringsSep "\n" (a: " ${lib.getExe (mkArchPublish a)} \"\${fwd[@]}\"") localArches} + ${lib.getExe indexApp} "''${fwd[@]}" + if [ "$PARITY_PUBLISH" != "1" ]; then + echo "[dry-run] would copy ${imageName}:$VERSION -> ${imageName}:latest (digest copy, LAST mutation)" + exit 0 + fi + tok="$(parity_resolve_token)" + regctl registry login "$PARITY_REGISTRY_HOST" -u "$PARITY_REGISTRY_OWNER" --pass-stdin <<<"$tok" >/dev/null 2>&1 + # ':latest' is a pure digest copy of the just-assembled ':TAG', done + # LAST so it is the single idempotent mutation that flips the channel. + regctl image copy "${imageName}:$VERSION" "${imageName}:latest" + echo "Published ${imageName}:$VERSION and flipped :latest" + ''; + }; + + pushStaged = pkgs.writeShellApplication { + name = "push-staged"; + runtimeInputs = baseInputs ++ [ pkgs.nix ]; + text = '' + ${head} + parity_parse_args "Replay staged ${imageName} arch closures from .parity-stage to the registry" "$@" + VERSION="$(parity_derive_version ${lib.escapeShellArg version})" + if [ "$PARITY_PUBLISH" != "1" ]; then + echo "[dry-run] would replay staged arch images under $PARITY_STAGE_DIR to ${imageName}:$VERSION-" + echo "[dry-run] re-run with --publish / PUBLISH=1 to push." + exit 0 + fi + parity_devtag_guard + tok="$(parity_resolve_token)" + parity_registry_preflight + creds="$PARITY_REGISTRY_OWNER:$tok" + shopt -s nullglob + for d in "$PARITY_STAGE_DIR"/*/; do + [ -f "$d/copy-to-path" ] || continue + arch="$(basename "$d")" + copyto="$(cat "$d/copy-to-path")" + "$copyto/bin/copy-to" --dest-creds "$creds" "docker://${imageName}:$VERSION-$arch" + echo "Replayed ${imageName}:$VERSION-$arch" + done + ''; + }; + in + archStageApps + // archPublishApps + // { + "publish-index" = mkApp indexApp "Assemble the multi-arch index ${imageName}: from pushed per-arch tags (fail-closed)."; + "publish" = mkApp publishAll "Publish all local arches + index; :latest = digest copy of :TAG (last mutation)."; + "push-staged" = mkApp pushStaged "Replay staged ${imageName} arch closures from .parity-stage to the registry."; + }; + + # ========================================================================= + # Shared push-staged for single-artifact archetypes (pypi/npm/generic): replay + # the staged file from .parity-stage to the registry without rebuilding. + # ========================================================================= + mkPushStagedGeneric = + { + pname, + arch, + kind, + version ? "", + assetName ? "", + registryHost ? "git.oleks.space", + registryOwner ? "oleks", + passEntry ? "infra/gitea/personal_access_token_packages_rw", + }: + let + head = preamble { inherit registryHost registryOwner passEntry; }; + # The kind is known at Nix-eval time, so the kind-specific replay body is + # selected here (not via a runtime `case` on a constant, which trips + # shellcheck SC2194). + kindBody = { + pypi = '' + whl="$(find "$d" -name '*.whl' | head -1)" + sha="$(sha256sum "$whl" | cut -d' ' -f1)" + http="$(curl -s -o /dev/null -w '%{http_code}' -X POST \ + -H "Authorization: token $tok" -F ':action=file_upload' \ + -F 'protocol_version=1' -F "sha256_digest=$sha" \ + -F "name=${pname}" -F "version=$VERSION" -F 'filetype=bdist_wheel' \ + -F "content=@$whl" \ + "https://$PARITY_REGISTRY_HOST/api/packages/$PARITY_REGISTRY_OWNER/pypi")" + case "$http" in + 200 | 201 | 409) echo "Replayed ${pname} $VERSION (HTTP $http)" ;; + *) echo "BLOCKER(upload-failed): HTTP $http" >&2; exit 1 ;; + esac + ''; + generic = '' + http="$(curl -s -o /dev/null -w '%{http_code}' -X PUT \ + -H "Authorization: token $tok" --upload-file "$d/${assetName}" \ + "https://$PARITY_REGISTRY_HOST/api/packages/$PARITY_REGISTRY_OWNER/generic/${pname}/$VERSION/${assetName}")" + case "$http" in + 201 | 409) echo "Replayed ${assetName} $VERSION (HTTP $http)" ;; + *) echo "BLOCKER(upload-failed): HTTP $http" >&2; exit 1 ;; + esac + ''; + npm = '' + reg="https://$PARITY_REGISTRY_HOST/api/packages/$PARITY_REGISTRY_OWNER/npm/" + ( + cd "$d" + set +e + out="$(npm publish --registry="$reg" "--//$PARITY_REGISTRY_HOST/api/packages/$PARITY_REGISTRY_OWNER/npm/:_authToken=$tok" 2>&1)" + rc=$? + set -e + printf '%s\n' "$out" | sed "s/$tok/***REDACTED***/g" + if [ "$rc" -ne 0 ]; then + if printf '%s' "$out" | grep -qiE 'already exists|409|conflict'; then + echo "Already published — idempotent." + exit 0 + fi + exit "$rc" + fi + ) + ''; + }; + app = pkgs.writeShellApplication { + name = "push-staged"; + runtimeInputs = baseInputs ++ lib.optional (kind == "npm") pkgs.nodejs; + text = '' + ${head} + parity_parse_args "Replay the staged ${pname} ${arch} artifact (${kind}) from .parity-stage to the registry" "$@" + d="$(parity_stage_path ${lib.escapeShellArg arch})" + if [ ! -d "$d" ]; then + echo "BLOCKER(no-stage): nothing staged at $d — run 'nix run .#stage-${arch}' first." >&2 + exit 1 + fi + VERSION="$(parity_derive_version ${lib.escapeShellArg version})" + if [ "$PARITY_PUBLISH" != "1" ]; then + echo "[dry-run] would replay $d (${kind}) -> https://$PARITY_REGISTRY_HOST (${pname} $VERSION)" + echo "[dry-run] re-run with --publish / PUBLISH=1 to push." + exit 0 + fi + parity_devtag_guard + tok="$(parity_resolve_token)" + parity_registry_preflight + ${kindBody.${kind}} + ''; + }; + in + mkApp app "Replay the staged ${pname} ${arch} artifact (${kind}) from .parity-stage (dry-run by default; --publish to push)."; + + mkPushStagedHelm = + { + pname, + ociRepo, + registryHost ? "git.oleks.space", + registryOwner ? "oleks", + passEntry ? "infra/gitea/personal_access_token_packages_rw", + }: + let + head = preamble { inherit registryHost registryOwner passEntry; }; + app = pkgs.writeShellApplication { + name = "push-staged"; + runtimeInputs = baseInputs ++ [ pkgs.kubernetes-helm ]; + text = '' + ${head} + parity_parse_args "Replay the staged ${pname} helm chart from .parity-stage to ${ociRepo}" "$@" + d="$(parity_stage_path chart)" + tgz="$(find "$d" -name '*.tgz' 2>/dev/null | head -1 || true)" + if [ -z "$tgz" ]; then + echo "BLOCKER(no-stage): no staged chart under $d — run 'nix run .#stage-chart' first." >&2 + exit 1 + fi + if [ "$PARITY_PUBLISH" != "1" ]; then + echo "[dry-run] would 'helm push $(basename "$tgz")' -> ${ociRepo}" + exit 0 + fi + parity_devtag_guard + tok="$(parity_resolve_token)" + parity_registry_preflight + echo "$tok" | helm registry login "$PARITY_REGISTRY_HOST" -u "$PARITY_REGISTRY_OWNER" --password-stdin >/dev/null 2>&1 + helm push "$tgz" ${lib.escapeShellArg ociRepo} + echo "Replayed chart $(basename "$tgz")" + ''; + }; + in + mkApp app "Replay the staged ${pname} helm chart from .parity-stage (dry-run by default; --publish to push)."; + +in +{ + inherit + mkPyPiWheelPublish + mkS390xNpmPublish + mkGenericBinaryPublish + mkNix2ContainerPublish + mkGoBinaryPublish + mkHelmPublish + ; + # Shared building blocks, exposed for advanced/bespoke consumers. + inherit mkApp shellLib; +}