From 7c28db076878bd6a6fe83d5f518975ef670efb38 Mon Sep 17 00:00:00 2001 From: oleks Date: Wed, 8 Jul 2026 21:10:36 +0300 Subject: [PATCH] Add s390x no-build policy section to Spec/Current-CI-Ecosystem --- Spec%2FCurrent-CI-Ecosystem.-.md | 66 +++++++++++++++++--------------- 1 file changed, 35 insertions(+), 31 deletions(-) diff --git a/Spec%2FCurrent-CI-Ecosystem.-.md b/Spec%2FCurrent-CI-Ecosystem.-.md index 262a182..54b4f3a 100644 --- a/Spec%2FCurrent-CI-Ecosystem.-.md +++ b/Spec%2FCurrent-CI-Ecosystem.-.md @@ -2,54 +2,58 @@ Woodpecker CI server running at `ci.oleks.space` (host-native systemd service on armer, agents distributed in k3s). Gitea forge at `git.oleks.space` serving as both source control and OCI/npm/PyPI registry. Fleet covers ~50 repositories with ~97 Woodpecker pipeline files as of 2026-07-04. -Woodpecker has no native `include:` mechanism for cross-repo YAML reuse. This repo (`oleks/ci-scripts`) is the generator-based answer: manifest.yaml + generate.py/render.py produce rendered `.woodpecker.yaml` files for each templated repo, committed per-repo via worktree. Rendered files carry a `GENERATED by oleks/ci-scripts — do not hand-edit` header. +Woodpecker has no native `include:` mechanism for cross-repo YAML reuse. This repo (`oleks/ci-scripts`) is the generator-based answer: manifest.yaml + generate.py/render.py produce rendered `.woodpecker.yaml` files for each templated repo, committed per-repo via worktree. Rendered files carry a `GENERATED by oleks/ci-scripts ← do not hand-edit` header. ## The Generator (This Repo) **Entry points:** -- `generate.py` — main orchestrator -- `render.py` — template rendering logic -- `manifest.yaml` — single source of truth for all templated repos +- `generate.py` ← main orchestrator +- `render.py` ← template rendering logic +- `manifest.yaml` ← single source of truth for all templated repos **Template kinds** (in `generator/templates/`): -1. **Identical-fleet** (10 repos) — `pypi-wheel-attic` family ([#1](https://git.oleks.space/oleks/ci-scripts/issues/1)): s390x cross-build repos (lxml, pillow, psycopg, psycopg2-binary, tiktoken, duckdb, faiss-cpu, numpy, pymupdf, markupsafe). Single build-and-publish step. Inline `attic watch-store` accelerator. Per-repo tuning: cores, memory, max_jobs, attic+redis toggle. +1. **Identical-fleet** (10 repos) ← `pypi-wheel-attic` family (#1): s390x cross-build repos (lxml, pillow, psycopy, psycopy2-binary, tiktoken, duckdb, faiass-cpu, numpy, pymupdf, markupsafe). Single build-and-publish step. Inline `attic watch-store` accelerator. Per-repo tuning: cores, memory, max_jobs, attic+redis toggle. -2. **Thin-parity-lib publish** (5 repos) ([#10](https://git.oleks.space/oleks/ci-scripts/issues/10)): asyncpg, cryptography, lightningcss, nextjs-swc, rollup (s390x). No inline attic. Publish is pure `nix run .#publish[-s390x]` or `PUBLISH=1 nix run .#publish`. Real per-repo variance: doctor-gate presence, token-direct mode, publish-style, tags-full toggle, dead-secrets list, gitea-clone-token opt-out. +2. **Thin-parity-lib publish** (5 repos) (#10): asyncpg, cryptography, lightningcss, nextjs-swc, rollup (s390x). No inline attic. Publish is pure `nix run .#publish[-s390x]` or `PUBLISH=1 nix run .#publish`. Real per-repo variance: doctor-gate presence, token-direct mode, publish-style, tags-full toggle, dead-secrets list, gitea-clone-token opt-out. -3. **Buildx quartet** (6 repos) ([#2](https://git.oleks.space/oleks/ci-scripts/issues/2)): xonsh-image, woodpecker-peek, csi-s3, the-eye, emdash, flake-hub. Matrix/split/single OCI image build shapes, each feeding into a manifest consolidation step. Tuned per repo: arch matrix (amd64/arm64 combinations), buildkit mode (normal vs rootless), Kubernetes node selectors. +3. **Buildx quartet** (6 repos) (#2): xonsh-image, woodpecker-peek, csi-s3, the-eye, emdash, flake-hub. Matrix/split/single OCI image build shapes, each feeding into a manifest consolidation step. Tuned per repo: arch matrix (amd64/arm64 combinations), buildkit mode (normal vs rootless), Kubernetes node selectors. **Verification:** ```bash -python3 generate.py --check # fails if any rendered file is stale +python3 generate.py --check # fails if any rendered file is stale woodpecker-cli lint /.woodpecker.yaml ``` ## Conventions (Enforced Across Fleet) -- **Arch banner** ([#7](https://git.oleks.space/oleks/ci-scripts/issues/7)): Every step's first command is `echo "▸ arch=$(uname -m)"` — enables per-arch debugging and artifact naming. +- **Arch banner** (#7): Every step's first command is `echo "▸ arch=$(uname -m)"` ← enables per-arch debugging and artifact naming. -- **Node pinning** ([#5](https://git.oleks.space/oleks/ci-scripts/issues/5)): Woodpecker agents advertise custom labels `{arch: amd64|arm64|s390x-native, worker: }`. Pipelines pin work via top-level `labels: {arch: amd64|arm64|s390x-native}` only. Deprecated: old `platform: mermaid` aliases and literal `backend_options.kubernetes.nodeSelector` (static node labels are fragile). +- **Node pinning** (#5): Woodpecker agents advertise custom labels `{arch: amd64|arm64|s390x-native, worker: }`. Pipelines pin work via top-level `labels: {arch: amd64|arm64|s390x-native}` only. Deprecated: old `platform: mermaid` aliases and literal `backend_options.kubernetes.nodeSelector` (static node labels are fragile). -- **No `|| true`** ([#6](https://git.oleks.space/oleks/ci-scripts/issues/6)): Exit-code masking banned. Explicit `if` logic for benign failures (e.g., attic watch-store teardown). Example: +- **No `|| true`** (#6): Exit-code masking banned. Explicit `if` logic for benign failures (e.g., attic watch-store teardown). Example: ```bash if ! kill $ATTIC_PID 2>/dev/null; then echo "attic watch-store already exited" fi ``` -- **Canonical clone step** ([#3](https://git.oleks.space/oleks/ci-scripts/issues/3)): Copy `templates/clone-step.yaml` verbatim into any pipeline needing clone override (private repo over HTTPS, shallow, tags disabled, full history). Uses `woodpeckierci/plugin-git` + `CI_NETRC_MACHINE: git.oleks.space` + `gitea_clone_token` secret. +- **Canonical clone step** (#3): Copy `templates/clone-step.yaml` verbatim into any pipeline needing clone override (private repo over HTTPS, shallow, tags disabled, full history). Uses `woodpeckierci/plugin-git` + `CI_NETRC_MACHINE: git.oleks.space` + `gitea_clone_token` secret. -- **Manifest step on partial success** ([#9](https://git.oleks.space/oleks/ci-scripts/issues/9)): Workflow-level `runs_on: [success, failure]` ensures multi-arch manifest consolidation runs even if one leg fails. Registry probe (attempt push, catch 404 on missing per-arch tag) gates whether manifest is built. +- **Manifest step on partial success** (#9): Workflow-level `runs_on: [success, failure]` ensures multi-arch manifest consolidation runs even if one leg fails. Registry probe (attempt push, catch 404 on missing per-arch tag) gates whether manifest is built. -- **Parse-time `${...}` substitution in Woodpecker** (architectural): Braced `${VAR}` is interpolated at parse time (undefined → empty, no error). Shell runtime uses bare `$VAR` or `$$` (escaped brace). This is a parse-phase footgun — manifests/matrix variables expanded before agent sees them. +- **Parse-time `${...}` substitution in Woodpecker** (architectural): Braced `${VAR}` is interpolated at parse time (undefined → empty, no error). Shell runtime uses bare `$VAR` or `$$` (escaped brace). This is a parse-phase footgun ← manifests/matrix variables expanded before agent sees them. + +## s390x Architecture (No-Build Policy) + +**s390x = no-build.** The s390x cross-build fleet (`building/s390x/*` — fastuuid, lxml, numpy, cryptography, pillow, etc.) produces Python wheels for the s390x architecture (IBM Z / LinuxONE), but that hardware is **currently unavailable**, so those wheels are **not built or published**. The s390x repo code is kept only as a **cross-compile reference/example** (how to Nix `pkgsCross` cross-compile a Rust/C Python extension). The generator (`generate.py`) treats every `s390x/` manifest entry as `no_build: true` — rendering a neutralized pipeline (`when: event: manual`, an alpine step that just prints why) so tag pushes never trigger a build. Re-enable a repo by setting `no_build: false` in `manifest.yaml`. **arm64 builds are unaffected** — arm64 hardware is available. Enforced in generator commit bf36991. ## Constants (Central Registry, Host, Image Addresses) See `docs/constants.md`. Two mechanisms: -1. **Woodpecker secrets** — `registry_token`, `attic_token`, `gitea_clone_token` (org-level, shared across repos). Never inlined into YAML. +1. **Woodpecker secrets** ← `registry_token`, `attic_token`, `gitea_clone_token` (org-level, shared across repos). Never inlined into YAML. -2. **Manifest fields** (in per-repo `manifest.yaml` entry) — interpolated by generator into rendered YAML: +2. **Manifest fields** (in per-repo `manifest.yaml` entry) ← interpolated by generator into rendered YAML: - `git_host`: `git.oleks.space` - `image`: `git.oleks.space/oleks/nix-ci:latest` (generator appends `-arm64` for arm64 legs) - `buildkit_prefix`: `buildkit` or `buildkit-rootless` (Kubernetes in-cluster service DNS) @@ -60,35 +64,35 @@ See `docs/constants.md`. Two mechanisms: Repos opt out of templating for architectural reasons (no downside to hand-maintaining if the shape is truly one-off): -- **Quartet cosign/digest-pinned** — `builder-arbitrage`: cosign signing + digest-pinned OCI images + multi-step orchestration for dual-arch operator. +- **Buildx quartet cosign/digest-pinned** ← `builder-arbitrage`: cosign signing + digest-pinned OCI images + multi-step orchestration for dual-arch operator. -- **Chained steps** — `nixos-ci`: sequential Nix build steps with conditional dispatch (push to nixbuild.net vs local cache). Too many branches for a single template. +- **Chained steps** ← `nixos-ci`: sequential Nix build steps with conditional dispatch (push to nixbuild.net vs local cache). Too many branches for a single template. -- **nix2container + regctl** — `ii-agent`: proprietary image build + registry introspection. Bespoke until nix2container / Cargo.nix shape stabilizes. +- **nix2container + regctl** ← `iia-agent`: proprietary image build + registry introspection. Bespoke until nix2container / Cargo.nix shape stabilizes. -- **Dockerfile amd64-only** — `ii-researcher`, `commonground-legacy`: pre-existing Dockerfile layers. Declare explicitly as amd64-only rather than force-fit into buildx. +- **Dockerfile amd64-only** ← `iia-researcher`, `commonground-legacy`: pre-existing Dockerfile layers. Declare explicitly as amd64-only in manifest + skip arm64 CI entirely. -- **Python wheel + moby-buildkit** — `terminal-agent`: Docker SDK build, not Buildx/Nix. One-off infra. +- **Python wheel + moby-buildkit** ← `terminal-agent`: Docker SDK build, not Buildx/Nix. One-off infra. -- **nixbuild.net single-step** — `mermaid-gpu-dashboard`: remote builder farm. Simpler as a single invocation. +- **nixbuild.net single-step** ← `mermaid-gpu-dashboard`: remote builder farm. Simpler as a single invocation. - **Fleet helpers** (not pipelines, infra): - - `geesefs-s390x`, `sentry-cli-s390x`, `attic-client-s390x` — custom binary builds, not pipelines. - - `onnxruntime-s390x`, `scikit-learn-s390x`, `scipy-s390x` — **Latent template bugs flagged, not silently "fixed"** (see manifest.yaml comments): - - `fastuuid-s390x`, `onnxruntime-s390x` declare `CI_REGISTRY_TOKEN` but lack `export REGISTRY_TOKEN=...` remap; publish would run with token unset. Left hand-maintained pending policy decision. + - `geesefs-s390x`, `sentry-cli-s390x`, `attic-client-s390x` ← custom binary builds, no pipelines. + - `onnyxruntime-s390x`, `scikit-learn-s390x`, `scipy-s390x` ← **(LATENT template bugs flagged, not silently "fixed")** (see manifest.yaml comments): + - `fastuuid-s390x`, `onnyxruntime-s390x` declare `CI_REGISTRY_TOKEN` in `environment:` but lack `export REGISTRY_TOKEN=...` remap; publish would run with token unset. Left hand-maintained pending policy decision. - `scikit-learn-s390x`, `scipy-s390x` use heredoc Redis setup, not direct export. Behavior unconfirmed; too risky to template without re-audit. ## Known Issues -- **Latent token remap bug** (`fastuuid-s390x`, `onnxruntime-s390x`): Manifest lists `CI_REGISTRY_TOKEN` in `environment:` but never exports `REGISTRY_TOKEN` before `PUBLISH=1 nix run .#publish`. Token arrives unset. Not a generator bug (template never used); flagged in manifest comments for next audit. +- **Latent token remap bug** (`fastuuid-s390x`, `onnyxruntime-s390x`): Manifest lists `CI_REGISTRY_TOKEN` in `environment:` but never exports `REGISTRY_TOKEN` before `PUBLISH=1 nix run .#publish`. Token arrives unset. Not a generator bug (template never used); flagged in manifest comments for next audit. -- **Drift possible** — nothing enforces "rendered file == template output." A hand-edit to a generated `.woodpecker.yaml` won't be caught until `generate.py --check` is run (not gated by CI). +- **Drift possible** ← nothing enforces "rendered file == template output." A hand-edit to a generated `.woodpecker.yaml` won't be caught until `generate.py --check` is run (not gated by CI). -- **Secrets consistency** — per-repo secrets (`registry_token`, `attic_token`, `gitea_clone_token`) named inconsistently across Woodpecker UI (some repos declare them, some inherit org secrets). No audit yet. +- **Secrets consistency** ← per-repo secrets (registry_token, attic_token, gitea_clone_token) named inconsistently across Woodpecker UI (some repos declare them, some inherit org secrets). No audit yet. -- **Woodpecker-peek API 404** — `oleks/woodpecker-peek` (#13, unresolved) returns 404 in Woodpecker UI API even though the repo has registered pipelines. Possibly a plugin registration issue or UI cache stale. +- **Woodpecker-peek API 404** ← `oleks/woodpecker-peek` (#13, unresolved) returns 404 in Woodpecker UI API even though the repo has registered pipelines. Possibly a plugin registration issue or UI cache stale. ## See also -- [Spec/Modern CI Target](wiki/Spec/Modern-CI-Target) — Target architecture for modern CI (Phases 1-4) -- [Spec/Fleet Publish Plugin System](wiki/Spec/Fleet-Publish-Plugin-System) — Plugin architecture & thin pipeline contract for Phase 2+3 +- [Spec/Modern CI Target](wiki/Spec/Modern-CI-Target) ← Target architecture for modern CI (Phases 1-4) +- [Spec/Fleet Publish Plugin System](wiki/Spec/Fleet-Publish-Plugin-System) ← Plugin architecture & thin pipeline contract for Phase 2+3