From 4ab389d1eb357353459cde4405f5643a85c5a272 Mon Sep 17 00:00:00 2001 From: repo-worker Date: Sat, 4 Jul 2026 19:17:21 +0300 Subject: [PATCH] Add shared clone step, constants doc, and buildx-quartet generator (#2, #3, #4) --- docs/constants.md | 46 +++++ generator/render.py | 186 ++++++++++++++++++ generator/templates/matrix/build.yaml.tmpl | 33 ++++ generator/templates/matrix/manifest.yaml.tmpl | 21 ++ .../templates/single/COMPONENT_STEP.tmpl | 26 +++ .../templates/single/container.yaml.tmpl | 25 +++ generator/templates/split/PERARCH.yaml.tmpl | 39 ++++ manifests/csi-s3.yaml | 10 + manifests/the-eye.yaml | 9 + manifests/woodpecker-peek.yaml | 31 +++ manifests/xonsh-image.yaml | 6 + templates/clone-step.yaml | 25 +++ 12 files changed, 457 insertions(+) create mode 100644 docs/constants.md create mode 100644 generator/render.py create mode 100644 generator/templates/matrix/build.yaml.tmpl create mode 100644 generator/templates/matrix/manifest.yaml.tmpl create mode 100644 generator/templates/single/COMPONENT_STEP.tmpl create mode 100644 generator/templates/single/container.yaml.tmpl create mode 100644 generator/templates/split/PERARCH.yaml.tmpl create mode 100644 manifests/csi-s3.yaml create mode 100644 manifests/the-eye.yaml create mode 100644 manifests/woodpecker-peek.yaml create mode 100644 manifests/xonsh-image.yaml create mode 100644 templates/clone-step.yaml diff --git a/docs/constants.md b/docs/constants.md new file mode 100644 index 0000000..dd97907 --- /dev/null +++ b/docs/constants.md @@ -0,0 +1,46 @@ +# Centralized registry/host constants (oleks/ci-scripts#4) + +Woodpecker has no cross-repo variable store and no `include:` mechanism, so +these constants can't be defined once and referenced by name inside raw YAML +pipelines. Two real mechanisms exist instead — use whichever fits the value: + +1. **Woodpecker secret or org secret** — for anything that's actually a + credential, or that we want to be able to rotate without touching every + repo's pipeline file. Prefer this whenever a value isn't public. +2. **Manifest field, interpolated by the oleks/ci-scripts generator** + (`generator/render.py`, oleks/ci-scripts#2) — for values that are public + infrastructure addresses baked into rendered YAML. Put the value in the + per-repo manifest once; the generator stamps it into every rendered file. + This is the practical stand-in for a "constants file" Woodpecker can't + natively `include:`. + +There is no third option (a shared "constants.yaml" `include:`d by every +pipeline) — Woodpecker pipelines are standalone YAML with no include +directive, so the generator is the only place central substitution happens. + +## Inventory + +| Constant | Value(s) | Where it's inlined today | Mechanism | +|---|---|---|---| +| Git server / netrc machine | `git.oleks.space` | every `clone:` block | manifest field `git_host` (rarely changes; secret would be overkill) | +| Gitea clone token | — | every `clone:` block, `GITEA_CLONE_TOKEN` env | **secret** `gitea_clone_token` (org secret, already is one) | +| Registry push token | — | matrix/manifest build steps | **secret** `registry_token` (org secret) | +| Attic cache token | — | split (attic-closure) build steps | **secret** `attic_token` (org secret) | +| Build image | `git.oleks.space/oleks/nix-ci:latest` / `:latest-arm64` | every build step's `image:` | manifest field `image` + arch suffix rule (generator appends `-arm64` for arm64 legs) | +| BuildKit address | `tcp://buildkit-${ARCH}.infra.svc.cluster.local:1234`, `tcp://buildkit-rootless-${ARCH}.infra.svc.cluster.local:1234` | matrix-shape build/manifest steps | manifest field `buildkit_prefix` (`buildkit` or `buildkit-rootless`), arch interpolated by the generator | +| Resolver nameserver | `169.254.20.10` | s390x cross-build pipelines (building/s390x, out of this worker's scope) | manifest field `nameserver` where the shape needs it; coordinate with fleet-worker (owns building/s390x + building/arm64) | +| Kubernetes node selector overrides | e.g. `kubernetes.io/hostname: howard2404` (csi-s3, pinned to the only s390x/arm64-capable node) | matrix-shape `backend_options.kubernetes.nodeSelector` | manifest field `node_selector` (map, optional) | + +## Convention going forward + +- If a value is secret-shaped (token, password, key) → it MUST be a + Woodpecker secret, never a manifest field or literal in generated YAML. +- If a value is public infrastructure (hostname, image ref, in-cluster + service DNS name) that's the same across most repos → put it in the + per-repo manifest consumed by `generator/render.py` (oleks/ci-scripts#2), + not hand-typed into `.woodpecker/*.yaml`. +- If a value is genuinely repo-specific (a one-off node pin, a nonstandard + arch list) → it's still a manifest field, just not shared across repos. + The manifest is the single place per-repo variance lives; the rendered + YAML is never hand-edited (see the "GENERATED" header the generator + stamps on every output file). diff --git a/generator/render.py b/generator/render.py new file mode 100644 index 0000000..754c42b --- /dev/null +++ b/generator/render.py @@ -0,0 +1,186 @@ +#!/usr/bin/env python3 +"""Render Woodpecker buildx-quartet pipelines from a per-repo manifest. + +oleks/ci-scripts#2 — one generator, three supported shapes (`kind:` in the +manifest), because the repos in scope aren't actually one shape: + + matrix - single build.yaml with a `matrix: {ARCH: [...]}` + manifest.yaml + combining the per-arch tags (xonsh-image, csi-s3, ...). + split - separate per-arch files (amd64.yaml, arm64.yaml, ...), no + manifest step — used for attic-closure "nix run .#publish-*" + publishes rather than docker buildx (woodpecker-peek, ...). + single - one container.yaml, one native-arch build (+ optional extra + components like a grpc image), no matrix, no manifest + (the-eye, emdash.kotkanagrilli.fi). + +Usage: + python3 generator/render.py manifests/.yaml --out /path/to/repo/.woodpecker + +Every rendered file gets a "GENERATED by oleks/ci-scripts — do not hand-edit" +header. Re-running is idempotent: same manifest in, byte-identical files out. +""" +import argparse +import pathlib +import sys + +import yaml + +HERE = pathlib.Path(__file__).parent +TEMPLATES = HERE / "templates" + +GENERATED_HEADER = ( + "# GENERATED by oleks/ci-scripts — do not hand-edit.\n" + "# Source manifest: {manifest_path}\n" + "# Regenerate: python3 generator/render.py {manifest_path} --out /.woodpecker\n" + "\n" +) + + +def load_template(*parts): + return (TEMPLATES / pathlib.Path(*parts)).read_text() + + +def render_matrix(manifest: dict) -> dict: + arches = manifest["arches"] + image = manifest["image"] + script = manifest.get("script", "ci/local.sh") + buildkit_prefix = manifest.get("buildkit_prefix", "buildkit") + node_selector = manifest.get("node_selector") + timeout = manifest.get("timeout") + manifest_arches_env = manifest.get("manifest_arches_env") + + arch_list = "\n".join(f" - {a}" for a in arches) + timeout_block = f"\ntimeout: {timeout}\n" if timeout else "" + if node_selector: + lines = "\n".join(f" {k}: {v}" for k, v in node_selector.items()) + node_selector_block = f" nodeSelector:\n{lines}\n" + node_selector_top_block = ( + " backend_options:\n kubernetes:\n nodeSelector:\n" + lines + "\n" + ) + else: + node_selector_block = "" + node_selector_top_block = "" + + manifest_extra_env = "" + if manifest_arches_env: + manifest_extra_env = f' MANIFEST_ARCHES: "{manifest_arches_env}"\n' + if node_selector: + manifest_extra_env = ( + f' BUILDKIT_ADDR: "tcp://{buildkit_prefix}-{arches[0]}.infra.svc.cluster.local:1234"\n' + + manifest_extra_env + ) + + build = load_template("matrix", "build.yaml.tmpl").format( + arch_list=arch_list, + timeout_block=timeout_block, + script=script, + image=image, + buildkit_prefix=buildkit_prefix, + node_selector_block=node_selector_block, + ) + mnf = load_template("matrix", "manifest.yaml.tmpl").format( + script=script, + image=image, + manifest_extra_env=manifest_extra_env, + node_selector_top_block=node_selector_top_block, + ) + return {"build.yaml": build, "manifest.yaml": mnf} + + +def render_split(manifest: dict) -> dict: + repo = manifest["repo"] + git_host = manifest.get("git_host", "git.oleks.space") + setup_script = manifest.get("setup_script", "ci/setup.sh") + out = {} + tmpl = load_template("split", "PERARCH.yaml.tmpl") + for arch, leg in manifest["legs"].items(): + header = leg["header"].rstrip("\n") + "\n" + entrypoint_comment = "".join( + f" # {l}\n" if l else " #\n" + for l in leg["entrypoint_comment"].rstrip("\n").splitlines() + ) + out[f"{arch}.yaml"] = tmpl.format( + repo=repo, + nix_system=leg["nix_system"], + arch=arch, + header=header, + entrypoint_comment=entrypoint_comment, + git_host=git_host, + image=leg["image"], + setup_script=setup_script, + app=leg["app"], + ) + return out + + +def render_single(manifest: dict) -> dict: + repo = manifest["repo"] + arch = manifest["arch"] + git_host = manifest.get("git_host", "git.oleks.space") + branches = "[" + ", ".join(manifest["branches"]) + "]" + buildkit_addr = f"tcp://buildkit-{arch}.infra.svc.cluster.local:1234" + + step_tmpl = load_template("single", "COMPONENT_STEP.tmpl") + steps = [] + for c in manifest["components"]: + command = f"./ci/local.sh --arch {arch}" + if c.get("component"): + command += f" --component {c['component']}" + steps.append( + step_tmpl.format( + step_name=c["name"], + image=manifest["image"], + buildkit_addr=buildkit_addr, + command=command, + arch=arch, + memory=c.get("memory", "4Gi"), + ) + ) + container = load_template("single", "container.yaml.tmpl").format( + repo=repo, + arch=arch, + branches=branches, + git_host=git_host, + component_steps="\n".join(steps), + ) + return {"container.yaml": container} + + +RENDERERS = {"matrix": render_matrix, "split": render_split, "single": render_single} + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("manifest", type=pathlib.Path) + ap.add_argument("--out", type=pathlib.Path, required=True, help="target .woodpecker/ dir") + ap.add_argument("--check", action="store_true", help="don't write, just diff against --out") + args = ap.parse_args() + + manifest = yaml.safe_load(args.manifest.read_text()) + kind = manifest["kind"] + if kind not in RENDERERS: + sys.exit(f"unknown kind: {kind}") + + files = RENDERERS[kind](manifest) + header = GENERATED_HEADER.format(manifest_path=args.manifest) + + args.out.mkdir(parents=True, exist_ok=True) + changed = False + for name, body in files.items(): + target = args.out / name + content = header + body + if args.check: + existing = target.read_text() if target.exists() else "" + if existing != content: + changed = True + print(f"DIFF: {target}") + else: + target.write_text(content) + print(f"wrote {target}") + + if args.check and changed: + sys.exit(1) + + +if __name__ == "__main__": + main() diff --git a/generator/templates/matrix/build.yaml.tmpl b/generator/templates/matrix/build.yaml.tmpl new file mode 100644 index 0000000..fe141fc --- /dev/null +++ b/generator/templates/matrix/build.yaml.tmpl @@ -0,0 +1,33 @@ +matrix: + ARCH: +{arch_list} + +when: + - event: tag + ref: "refs/tags/v*" +{timeout_block} +labels: + arch: ${{ARCH}} + +# Thin wrapper (cluster #196, emmett#44): CI runs the SAME {script} a +# developer runs, so CI and local can't drift. CI-specific bits are env +# overrides only: BUILDKIT_ADDR -> the in-cluster NATIVE per-arch buildkit +# (no QEMU), PUBLISH=1 -> actually push. Output is unchanged: a per-arch +# :- image, later merged into :+:latest by manifest.yaml. +steps: + - name: build-and-push + image: {image}:latest + environment: + REGISTRY_TOKEN: + from_secret: registry_token + BUILDKIT_ADDR: "tcp://{buildkit_prefix}-${{ARCH}}.infra.svc.cluster.local:1234" + PUBLISH: "1" + commands: + - echo "▸ arch=$(uname -m)" + - {script} --arch "${{ARCH}}" + backend_options: + kubernetes: +{node_selector_block} labels: + commit-tag: "${{CI_COMMIT_TAG}}" + pipeline-number: "${{CI_PIPELINE_NUMBER}}" + build-arch: "${{ARCH}}" diff --git a/generator/templates/matrix/manifest.yaml.tmpl b/generator/templates/matrix/manifest.yaml.tmpl new file mode 100644 index 0000000..218afe4 --- /dev/null +++ b/generator/templates/matrix/manifest.yaml.tmpl @@ -0,0 +1,21 @@ +when: + - event: tag + ref: "refs/tags/v*" + +depends_on: + - build + +# Thin wrapper (cluster #196, emmett#44): the SAME {script} composes the +# per-arch tags into the multi-arch : + :latest manifest list. +# PUBLISH=1 is the only CI-specific override (local default is dry-run). +steps: + - name: manifest + image: {image}:latest + environment: + REGISTRY_TOKEN: + from_secret: registry_token +{manifest_extra_env} PUBLISH: "1" + commands: + - echo "▸ arch=$(uname -m)" + - {script} --manifest +{node_selector_top_block} \ No newline at end of file diff --git a/generator/templates/single/COMPONENT_STEP.tmpl b/generator/templates/single/COMPONENT_STEP.tmpl new file mode 100644 index 0000000..d71cdc5 --- /dev/null +++ b/generator/templates/single/COMPONENT_STEP.tmpl @@ -0,0 +1,26 @@ + - name: {step_name} + image: {image} + environment: + REGISTRY_TOKEN: + from_secret: registry_token + BUILDKIT_ADDR: {buildkit_addr} + PUBLISH: "1" + BRANCH: "${{CI_COMMIT_BRANCH}}" + VERSION: "0.1.${{CI_PIPELINE_NUMBER}}" + commands: + - echo "▸ arch=$(uname -m)" + - {command} + + backend_options: + kubernetes: + nodeSelector: + kubernetes.io/arch: {arch} + resources: + requests: + memory: {memory} + limits: + memory: {memory} + labels: + commit-branch: "${{CI_COMMIT_BRANCH}}" + commit-sha: "${{CI_COMMIT_SHA}}" + pipeline-number: "${{CI_PIPELINE_NUMBER}}" diff --git a/generator/templates/single/container.yaml.tmpl b/generator/templates/single/container.yaml.tmpl new file mode 100644 index 0000000..c30fc3f --- /dev/null +++ b/generator/templates/single/container.yaml.tmpl @@ -0,0 +1,25 @@ +# Build + push the {repo} container (cluster #196, emmett#44). +# +# Keep-Dockerfile OCI image build; native single-arch buildkit, no matrix, +# no manifest step. See {repo}/ci/local.sh for the actual build logic. + +labels: + arch: {arch} + +when: + - event: push + branch: {branches} + +clone: + - name: clone + image: woodpeckerci/plugin-git + environment: + CI_NETRC_MACHINE: {git_host} + CI_NETRC_USERNAME: oleks + CI_NETRC_PASSWORD: + from_secret: gitea_clone_token + PLUGIN_TAGS: "false" + PLUGIN_DEPTH: "1" + +steps: +{component_steps} \ No newline at end of file diff --git a/generator/templates/split/PERARCH.yaml.tmpl b/generator/templates/split/PERARCH.yaml.tmpl new file mode 100644 index 0000000..7a66a57 --- /dev/null +++ b/generator/templates/split/PERARCH.yaml.tmpl @@ -0,0 +1,39 @@ +{header} +when: + - event: push + branch: main + - event: tag + +clone: + - name: clone + image: woodpeckerci/plugin-git + environment: + CI_NETRC_MACHINE: {git_host} + CI_NETRC_USERNAME: oleks + CI_NETRC_PASSWORD: + from_secret: gitea_clone_token + PLUGIN_TAGS: "false" + PLUGIN_DEPTH: "1" + +steps: + - name: build-{arch} + image: {image} + environment: + ATTIC_TOKEN: + from_secret: attic_token + GITEA_CLONE_TOKEN: + from_secret: gitea_clone_token + backend_options: + kubernetes: + nodeSelector: + kubernetes.io/arch: {arch} + labels: + commit-tag: "${{CI_COMMIT_TAG}}" + commit-branch: "${{CI_COMMIT_BRANCH}}" + pipeline-number: "${{CI_PIPELINE_NUMBER}}" + commands: + - echo "▸ arch=$(uname -m)" + # nix.conf substituters + {git_host} netrc (to fetch the parity-lib + # flake input). Same bootstrap as flake-hub/ci/setup.sh. + - sh {setup_script} +{entrypoint_comment} - PUBLISH=1 nix run .#{app} -- --publish diff --git a/manifests/csi-s3.yaml b/manifests/csi-s3.yaml new file mode 100644 index 0000000..b7fb96c --- /dev/null +++ b/manifests/csi-s3.yaml @@ -0,0 +1,10 @@ +repo: csi-s3 +kind: matrix +image: git.oleks.space/oleks/nix-ci +script: ci/local.sh +arches: [amd64, arm64] +buildkit_prefix: buildkit-rootless +timeout: 240 +node_selector: + kubernetes.io/hostname: howard2404 +manifest_arches_env: "amd64 arm64" diff --git a/manifests/the-eye.yaml b/manifests/the-eye.yaml new file mode 100644 index 0000000..16b3dd3 --- /dev/null +++ b/manifests/the-eye.yaml @@ -0,0 +1,9 @@ +repo: the-eye +kind: single +image: git.oleks.space/oleks/nix-ci:latest-arm64 +arch: arm64 +branches: [develop, staging, production] +components: + - name: build-and-push + - name: build-and-push-grpc + component: grpc diff --git a/manifests/woodpecker-peek.yaml b/manifests/woodpecker-peek.yaml new file mode 100644 index 0000000..9c79bc5 --- /dev/null +++ b/manifests/woodpecker-peek.yaml @@ -0,0 +1,31 @@ +repo: woodpecker-peek +kind: split +git_host: git.oleks.space +setup_script: ci/setup.sh +legs: + amd64: + nix_system: x86_64-linux + image: git.oleks.space/oleks/nix-ci:latest + app: publish-x86_64-linux + header: | + # Build woodpecker-peek for x86_64-linux and push the closure to the + # attic binary cache. Mirrors ~/projects/flake-hub/.woodpecker/amd64.yaml. + entrypoint_comment: | + Same entrypoint as a local `nix run .#publish-x86_64-linux -- --publish`. + All build/login/push logic lives in parity-lib's mkAtticClosurePublish; + PUBLISH=1 makes the shared app actually push (local runs dry-run). + arm64: + nix_system: aarch64-linux + image: git.oleks.space/oleks/nix-ci:latest-arm64 + app: publish-aarch64-linux + header: | + # Build woodpecker-peek for aarch64-linux and push the closure to attic. + # + # NODE-BOUND LEG (emmett#44, attic-closure): this binary is cgo/fyne + + # OpenGL + X11 with no cross path, so aarch64-linux CANNOT be built on + # emmett (linux/amd64) and is intentionally NOT a local-parity flake app. + # It must run on an aarch64 node — hence the arch nodeSelector below. + # Mirrors ~/projects/flake-hub/.woodpecker/arm64.yaml. + entrypoint_comment: | + Identical shared parity-lib entrypoint as amd64, only the arch differs. + All build/login/push logic lives in parity-lib's mkAtticClosurePublish. diff --git a/manifests/xonsh-image.yaml b/manifests/xonsh-image.yaml new file mode 100644 index 0000000..a86f4d9 --- /dev/null +++ b/manifests/xonsh-image.yaml @@ -0,0 +1,6 @@ +repo: xonsh-image +kind: matrix +image: git.oleks.space/oleks/nix-ci +script: ci/local.sh +arches: [amd64, arm64] +buildkit_prefix: buildkit diff --git a/templates/clone-step.yaml b/templates/clone-step.yaml new file mode 100644 index 0000000..a428eb5 --- /dev/null +++ b/templates/clone-step.yaml @@ -0,0 +1,25 @@ +# Canonical Woodpecker clone step (oleks/ci-scripts#3). +# +# Copy this `clone:` block verbatim into any pipeline that needs an explicit +# clone override (private repo over HTTPS via a Gitea PAT, shallow clone, or +# tags disabled). Most simple pipelines don't need this at all — Woodpecker's +# built-in implicit clone already authenticates the same way when the repo is +# private and the server has forge credentials wired up. Reach for this block +# when you need to tune PLUGIN_TAGS/PLUGIN_DEPTH or you're on a runner that +# doesn't have implicit forge auth (e.g. a Kubernetes backend crossing +# namespaces). +# +# Required secret: `gitea_clone_token` — a Gitea PAT with repo read scope, +# set as an org secret so every repo can reference it without redefining it +# (see docs/constants.md, oleks/ci-scripts#4). + +clone: + - name: clone + image: woodpeckerci/plugin-git + environment: + CI_NETRC_MACHINE: git.oleks.space + CI_NETRC_USERNAME: oleks + CI_NETRC_PASSWORD: + from_secret: gitea_clone_token + PLUGIN_TAGS: "false" # set "true" if the build needs tag history (e.g. nix-shell git describe) + PLUGIN_DEPTH: "1" # shallow clone; drop this key entirely for full history