Add shared clone step, constants doc, and buildx-quartet generator (#2, #3, #4)

docs/constants.md

@@ -0,0 +1,46 @@
+# Centralized registry/host constants (oleks/ci-scripts#4)
+
+Woodpecker has no cross-repo variable store and no `include:` mechanism, so
+these constants can't be defined once and referenced by name inside raw YAML
+pipelines. Two real mechanisms exist instead — use whichever fits the value:
+
+1. **Woodpecker secret or org secret** — for anything that's actually a
+   credential, or that we want to be able to rotate without touching every
+   repo's pipeline file. Prefer this whenever a value isn't public.
+2. **Manifest field, interpolated by the oleks/ci-scripts generator**
+   (`generator/render.py`, oleks/ci-scripts#2) — for values that are public
+   infrastructure addresses baked into rendered YAML. Put the value in the
+   per-repo manifest once; the generator stamps it into every rendered file.
+   This is the practical stand-in for a "constants file" Woodpecker can't
+   natively `include:`.
+
+There is no third option (a shared "constants.yaml" `include:`d by every
+pipeline) — Woodpecker pipelines are standalone YAML with no include
+directive, so the generator is the only place central substitution happens.
+
+## Inventory
+
+| Constant | Value(s) | Where it's inlined today | Mechanism |
+|---|---|---|---|
+| Git server / netrc machine | `git.oleks.space` | every `clone:` block | manifest field `git_host` (rarely changes; secret would be overkill) |
+| Gitea clone token | — | every `clone:` block, `GITEA_CLONE_TOKEN` env | **secret** `gitea_clone_token` (org secret, already is one) |
+| Registry push token | — | matrix/manifest build steps | **secret** `registry_token` (org secret) |
+| Attic cache token | — | split (attic-closure) build steps | **secret** `attic_token` (org secret) |
+| Build image | `git.oleks.space/oleks/nix-ci:latest` / `:latest-arm64` | every build step's `image:` | manifest field `image` + arch suffix rule (generator appends `-arm64` for arm64 legs) |
+| BuildKit address | `tcp://buildkit-${ARCH}.infra.svc.cluster.local:1234`, `tcp://buildkit-rootless-${ARCH}.infra.svc.cluster.local:1234` | matrix-shape build/manifest steps | manifest field `buildkit_prefix` (`buildkit` or `buildkit-rootless`), arch interpolated by the generator |
+| Resolver nameserver | `169.254.20.10` | s390x cross-build pipelines (building/s390x, out of this worker's scope) | manifest field `nameserver` where the shape needs it; coordinate with fleet-worker (owns building/s390x + building/arm64) |
+| Kubernetes node selector overrides | e.g. `kubernetes.io/hostname: howard2404` (csi-s3, pinned to the only s390x/arm64-capable node) | matrix-shape `backend_options.kubernetes.nodeSelector` | manifest field `node_selector` (map, optional) |
+
+## Convention going forward
+
+- If a value is secret-shaped (token, password, key) → it MUST be a
+  Woodpecker secret, never a manifest field or literal in generated YAML.
+- If a value is public infrastructure (hostname, image ref, in-cluster
+  service DNS name) that's the same across most repos → put it in the
+  per-repo manifest consumed by `generator/render.py` (oleks/ci-scripts#2),
+  not hand-typed into `.woodpecker/*.yaml`.
+- If a value is genuinely repo-specific (a one-off node pin, a nonstandard
+  arch list) → it's still a manifest field, just not shared across repos.
+  The manifest is the single place per-repo variance lives; the rendered
+  YAML is never hand-edited (see the "GENERATED" header the generator
+  stamps on every output file).

generator/render.py

@@ -0,0 +1,186 @@
+#!/usr/bin/env python3
+"""Render Woodpecker buildx-quartet pipelines from a per-repo manifest.
+
+oleks/ci-scripts#2 — one generator, three supported shapes (`kind:` in the
+manifest), because the repos in scope aren't actually one shape:
+
+  matrix  - single build.yaml with a `matrix: {ARCH: [...]}` + manifest.yaml
+            combining the per-arch tags (xonsh-image, csi-s3, ...).
+  split   - separate per-arch files (amd64.yaml, arm64.yaml, ...), no
+            manifest step — used for attic-closure "nix run .#publish-*"
+            publishes rather than docker buildx (woodpecker-peek, ...).
+  single  - one container.yaml, one native-arch build (+ optional extra
+            components like a grpc image), no matrix, no manifest
+            (the-eye, emdash.kotkanagrilli.fi).
+
+Usage:
+    python3 generator/render.py manifests/<repo>.yaml --out /path/to/repo/.woodpecker
+
+Every rendered file gets a "GENERATED by oleks/ci-scripts — do not hand-edit"
+header. Re-running is idempotent: same manifest in, byte-identical files out.
+"""
+import argparse
+import pathlib
+import sys
+
+import yaml
+
+HERE = pathlib.Path(__file__).parent
+TEMPLATES = HERE / "templates"
+
+GENERATED_HEADER = (
+    "# GENERATED by oleks/ci-scripts — do not hand-edit.\n"
+    "# Source manifest: {manifest_path}\n"
+    "# Regenerate: python3 generator/render.py {manifest_path} --out <repo>/.woodpecker\n"
+    "\n"
+)
+
+
+def load_template(*parts):
+    return (TEMPLATES / pathlib.Path(*parts)).read_text()
+
+
+def render_matrix(manifest: dict) -> dict:
+    arches = manifest["arches"]
+    image = manifest["image"]
+    script = manifest.get("script", "ci/local.sh")
+    buildkit_prefix = manifest.get("buildkit_prefix", "buildkit")
+    node_selector = manifest.get("node_selector")
+    timeout = manifest.get("timeout")
+    manifest_arches_env = manifest.get("manifest_arches_env")
+
+    arch_list = "\n".join(f"    - {a}" for a in arches)
+    timeout_block = f"\ntimeout: {timeout}\n" if timeout else ""
+    if node_selector:
+        lines = "\n".join(f"          {k}: {v}" for k, v in node_selector.items())
+        node_selector_block = f"        nodeSelector:\n{lines}\n"
+        node_selector_top_block = (
+            "    backend_options:\n      kubernetes:\n        nodeSelector:\n" + lines + "\n"
+        )
+    else:
+        node_selector_block = ""
+        node_selector_top_block = ""
+
+    manifest_extra_env = ""
+    if manifest_arches_env:
+        manifest_extra_env = f'      MANIFEST_ARCHES: "{manifest_arches_env}"\n'
+    if node_selector:
+        manifest_extra_env = (
+            f'      BUILDKIT_ADDR: "tcp://{buildkit_prefix}-{arches[0]}.infra.svc.cluster.local:1234"\n'
+            + manifest_extra_env
+        )
+
+    build = load_template("matrix", "build.yaml.tmpl").format(
+        arch_list=arch_list,
+        timeout_block=timeout_block,
+        script=script,
+        image=image,
+        buildkit_prefix=buildkit_prefix,
+        node_selector_block=node_selector_block,
+    )
+    mnf = load_template("matrix", "manifest.yaml.tmpl").format(
+        script=script,
+        image=image,
+        manifest_extra_env=manifest_extra_env,
+        node_selector_top_block=node_selector_top_block,
+    )
+    return {"build.yaml": build, "manifest.yaml": mnf}
+
+
+def render_split(manifest: dict) -> dict:
+    repo = manifest["repo"]
+    git_host = manifest.get("git_host", "git.oleks.space")
+    setup_script = manifest.get("setup_script", "ci/setup.sh")
+    out = {}
+    tmpl = load_template("split", "PERARCH.yaml.tmpl")
+    for arch, leg in manifest["legs"].items():
+        header = leg["header"].rstrip("\n") + "\n"
+        entrypoint_comment = "".join(
+            f"      # {l}\n" if l else "      #\n"
+            for l in leg["entrypoint_comment"].rstrip("\n").splitlines()
+        )
+        out[f"{arch}.yaml"] = tmpl.format(
+            repo=repo,
+            nix_system=leg["nix_system"],
+            arch=arch,
+            header=header,
+            entrypoint_comment=entrypoint_comment,
+            git_host=git_host,
+            image=leg["image"],
+            setup_script=setup_script,
+            app=leg["app"],
+        )
+    return out
+
+
+def render_single(manifest: dict) -> dict:
+    repo = manifest["repo"]
+    arch = manifest["arch"]
+    git_host = manifest.get("git_host", "git.oleks.space")
+    branches = "[" + ", ".join(manifest["branches"]) + "]"
+    buildkit_addr = f"tcp://buildkit-{arch}.infra.svc.cluster.local:1234"
+
+    step_tmpl = load_template("single", "COMPONENT_STEP.tmpl")
+    steps = []
+    for c in manifest["components"]:
+        command = f"./ci/local.sh --arch {arch}"
+        if c.get("component"):
+            command += f" --component {c['component']}"
+        steps.append(
+            step_tmpl.format(
+                step_name=c["name"],
+                image=manifest["image"],
+                buildkit_addr=buildkit_addr,
+                command=command,
+                arch=arch,
+                memory=c.get("memory", "4Gi"),
+            )
+        )
+    container = load_template("single", "container.yaml.tmpl").format(
+        repo=repo,
+        arch=arch,
+        branches=branches,
+        git_host=git_host,
+        component_steps="\n".join(steps),
+    )
+    return {"container.yaml": container}
+
+
+RENDERERS = {"matrix": render_matrix, "split": render_split, "single": render_single}
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("manifest", type=pathlib.Path)
+    ap.add_argument("--out", type=pathlib.Path, required=True, help="target .woodpecker/ dir")
+    ap.add_argument("--check", action="store_true", help="don't write, just diff against --out")
+    args = ap.parse_args()
+
+    manifest = yaml.safe_load(args.manifest.read_text())
+    kind = manifest["kind"]
+    if kind not in RENDERERS:
+        sys.exit(f"unknown kind: {kind}")
+
+    files = RENDERERS[kind](manifest)
+    header = GENERATED_HEADER.format(manifest_path=args.manifest)
+
+    args.out.mkdir(parents=True, exist_ok=True)
+    changed = False
+    for name, body in files.items():
+        target = args.out / name
+        content = header + body
+        if args.check:
+            existing = target.read_text() if target.exists() else ""
+            if existing != content:
+                changed = True
+                print(f"DIFF: {target}")
+        else:
+            target.write_text(content)
+            print(f"wrote {target}")
+
+    if args.check and changed:
+        sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()

generator/templates/matrix/build.yaml.tmpl

@@ -0,0 +1,33 @@
+matrix:
+  ARCH:
+{arch_list}
+
+when:
+  - event: tag
+    ref: "refs/tags/v*"
+{timeout_block}
+labels:
+  arch: ${{ARCH}}
+
+# Thin wrapper (cluster #196, emmett#44): CI runs the SAME {script} a
+# developer runs, so CI and local can't drift. CI-specific bits are env
+# overrides only: BUILDKIT_ADDR -> the in-cluster NATIVE per-arch buildkit
+# (no QEMU), PUBLISH=1 -> actually push. Output is unchanged: a per-arch
+# :<version>-<arch> image, later merged into :<version>+:latest by manifest.yaml.
+steps:
+  - name: build-and-push
+    image: {image}:latest
+    environment:
+      REGISTRY_TOKEN:
+        from_secret: registry_token
+      BUILDKIT_ADDR: "tcp://{buildkit_prefix}-${{ARCH}}.infra.svc.cluster.local:1234"
+      PUBLISH: "1"
+    commands:
+      - echo "▸ arch=$(uname -m)"
+      - {script} --arch "${{ARCH}}"
+    backend_options:
+      kubernetes:
+{node_selector_block}        labels:
+          commit-tag: "${{CI_COMMIT_TAG}}"
+          pipeline-number: "${{CI_PIPELINE_NUMBER}}"
+          build-arch: "${{ARCH}}"

generator/templates/matrix/manifest.yaml.tmpl

@@ -0,0 +1,21 @@
+when:
+  - event: tag
+    ref: "refs/tags/v*"
+
+depends_on:
+  - build
+
+# Thin wrapper (cluster #196, emmett#44): the SAME {script} composes the
+# per-arch tags into the multi-arch :<version> + :latest manifest list.
+# PUBLISH=1 is the only CI-specific override (local default is dry-run).
+steps:
+  - name: manifest
+    image: {image}:latest
+    environment:
+      REGISTRY_TOKEN:
+        from_secret: registry_token
+{manifest_extra_env}      PUBLISH: "1"
+    commands:
+      - echo "▸ arch=$(uname -m)"
+      - {script} --manifest
+{node_selector_top_block}

generator/templates/single/COMPONENT_STEP.tmpl

@@ -0,0 +1,26 @@
+  - name: {step_name}
+    image: {image}
+    environment:
+      REGISTRY_TOKEN:
+        from_secret: registry_token
+      BUILDKIT_ADDR: {buildkit_addr}
+      PUBLISH: "1"
+      BRANCH: "${{CI_COMMIT_BRANCH}}"
+      VERSION: "0.1.${{CI_PIPELINE_NUMBER}}"
+    commands:
+      - echo "▸ arch=$(uname -m)"
+      - {command}
+
+    backend_options:
+      kubernetes:
+        nodeSelector:
+          kubernetes.io/arch: {arch}
+        resources:
+          requests:
+            memory: {memory}
+          limits:
+            memory: {memory}
+        labels:
+          commit-branch: "${{CI_COMMIT_BRANCH}}"
+          commit-sha: "${{CI_COMMIT_SHA}}"
+          pipeline-number: "${{CI_PIPELINE_NUMBER}}"

generator/templates/single/container.yaml.tmpl

@@ -0,0 +1,25 @@
+# Build + push the {repo} container (cluster #196, emmett#44).
+#
+# Keep-Dockerfile OCI image build; native single-arch buildkit, no matrix,
+# no manifest step. See {repo}/ci/local.sh for the actual build logic.
+
+labels:
+  arch: {arch}
+
+when:
+  - event: push
+    branch: {branches}
+
+clone:
+  - name: clone
+    image: woodpeckerci/plugin-git
+    environment:
+      CI_NETRC_MACHINE: {git_host}
+      CI_NETRC_USERNAME: oleks
+      CI_NETRC_PASSWORD:
+        from_secret: gitea_clone_token
+      PLUGIN_TAGS: "false"
+      PLUGIN_DEPTH: "1"
+
+steps:
+{component_steps}

generator/templates/split/PERARCH.yaml.tmpl

@@ -0,0 +1,39 @@
+{header}
+when:
+  - event: push
+    branch: main
+  - event: tag
+
+clone:
+  - name: clone
+    image: woodpeckerci/plugin-git
+    environment:
+      CI_NETRC_MACHINE: {git_host}
+      CI_NETRC_USERNAME: oleks
+      CI_NETRC_PASSWORD:
+        from_secret: gitea_clone_token
+      PLUGIN_TAGS: "false"
+      PLUGIN_DEPTH: "1"
+
+steps:
+  - name: build-{arch}
+    image: {image}
+    environment:
+      ATTIC_TOKEN:
+        from_secret: attic_token
+      GITEA_CLONE_TOKEN:
+        from_secret: gitea_clone_token
+    backend_options:
+      kubernetes:
+        nodeSelector:
+          kubernetes.io/arch: {arch}
+        labels:
+          commit-tag: "${{CI_COMMIT_TAG}}"
+          commit-branch: "${{CI_COMMIT_BRANCH}}"
+          pipeline-number: "${{CI_PIPELINE_NUMBER}}"
+    commands:
+      - echo "▸ arch=$(uname -m)"
+      # nix.conf substituters + {git_host} netrc (to fetch the parity-lib
+      # flake input). Same bootstrap as flake-hub/ci/setup.sh.
+      - sh {setup_script}
+{entrypoint_comment}      - PUBLISH=1 nix run .#{app} -- --publish

manifests/csi-s3.yaml

@@ -0,0 +1,10 @@
+repo: csi-s3
+kind: matrix
+image: git.oleks.space/oleks/nix-ci
+script: ci/local.sh
+arches: [amd64, arm64]
+buildkit_prefix: buildkit-rootless
+timeout: 240
+node_selector:
+  kubernetes.io/hostname: howard2404
+manifest_arches_env: "amd64 arm64"

manifests/the-eye.yaml

@@ -0,0 +1,9 @@
+repo: the-eye
+kind: single
+image: git.oleks.space/oleks/nix-ci:latest-arm64
+arch: arm64
+branches: [develop, staging, production]
+components:
+  - name: build-and-push
+  - name: build-and-push-grpc
+    component: grpc

manifests/woodpecker-peek.yaml

@@ -0,0 +1,31 @@
+repo: woodpecker-peek
+kind: split
+git_host: git.oleks.space
+setup_script: ci/setup.sh
+legs:
+  amd64:
+    nix_system: x86_64-linux
+    image: git.oleks.space/oleks/nix-ci:latest
+    app: publish-x86_64-linux
+    header: |
+      # Build woodpecker-peek for x86_64-linux and push the closure to the
+      # attic binary cache. Mirrors ~/projects/flake-hub/.woodpecker/amd64.yaml.
+    entrypoint_comment: |
+      Same entrypoint as a local `nix run .#publish-x86_64-linux -- --publish`.
+      All build/login/push logic lives in parity-lib's mkAtticClosurePublish;
+      PUBLISH=1 makes the shared app actually push (local runs dry-run).
+  arm64:
+    nix_system: aarch64-linux
+    image: git.oleks.space/oleks/nix-ci:latest-arm64
+    app: publish-aarch64-linux
+    header: |
+      # Build woodpecker-peek for aarch64-linux and push the closure to attic.
+      #
+      # NODE-BOUND LEG (emmett#44, attic-closure): this binary is cgo/fyne +
+      # OpenGL + X11 with no cross path, so aarch64-linux CANNOT be built on
+      # emmett (linux/amd64) and is intentionally NOT a local-parity flake app.
+      # It must run on an aarch64 node — hence the arch nodeSelector below.
+      # Mirrors ~/projects/flake-hub/.woodpecker/arm64.yaml.
+    entrypoint_comment: |
+      Identical shared parity-lib entrypoint as amd64, only the arch differs.
+      All build/login/push logic lives in parity-lib's mkAtticClosurePublish.

manifests/xonsh-image.yaml

@@ -0,0 +1,6 @@
+repo: xonsh-image
+kind: matrix
+image: git.oleks.space/oleks/nix-ci
+script: ci/local.sh
+arches: [amd64, arm64]
+buildkit_prefix: buildkit

templates/clone-step.yaml

@@ -0,0 +1,25 @@
+# Canonical Woodpecker clone step (oleks/ci-scripts#3).
+#
+# Copy this `clone:` block verbatim into any pipeline that needs an explicit
+# clone override (private repo over HTTPS via a Gitea PAT, shallow clone, or
+# tags disabled). Most simple pipelines don't need this at all — Woodpecker's
+# built-in implicit clone already authenticates the same way when the repo is
+# private and the server has forge credentials wired up. Reach for this block
+# when you need to tune PLUGIN_TAGS/PLUGIN_DEPTH or you're on a runner that
+# doesn't have implicit forge auth (e.g. a Kubernetes backend crossing
+# namespaces).
+#
+# Required secret: `gitea_clone_token` — a Gitea PAT with repo read scope,
+# set as an org secret so every repo can reference it without redefining it
+# (see docs/constants.md, oleks/ci-scripts#4).
+
+clone:
+  - name: clone
+    image: woodpeckerci/plugin-git
+    environment:
+      CI_NETRC_MACHINE: git.oleks.space
+      CI_NETRC_USERNAME: oleks
+      CI_NETRC_PASSWORD:
+        from_secret: gitea_clone_token
+      PLUGIN_TAGS: "false"   # set "true" if the build needs tag history (e.g. nix-shell git describe)
+      PLUGIN_DEPTH: "1"      # shallow clone; drop this key entirely for full history