oleks/ci-scripts
1
0
Fork 0
Code Issues 8 Pull Requests Actions Packages Projects Releases Wiki Activity

Add s390x no-build policy section to Spec/Current-CI-Ecosystem

oleks
2026-07-08 21:10:36 +03:00
parent de9800f611
commit 7c28db0768
1 changed files with 35 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Spec%2FCurrent-CI-Ecosystem.-.md
+35 -31
@@ -2,54 +2,58 @@
Woodpecker CI server running at `ci.oleks.space` (host-native systemd service on armer, agents distributed in k3s). Gitea forge at `git.oleks.space` serving as both source control and OCI/npm/PyPI registry. Fleet covers ~50 repositories with ~97 Woodpecker pipeline files as of 2026-07-04.
Woodpecker has no native `include:` mechanism for cross-repo YAML reuse. This repo (`oleks/ci-scripts`) is the generator-based answer: manifest.yaml + generate.py/render.py produce rendered `.woodpecker.yaml` files for each templated repo, committed per-repo via worktree. Rendered files carry a `GENERATED by oleks/ci-scripts — do not hand-edit` header.
Woodpecker has no native `include:` mechanism for cross-repo YAML reuse. This repo (`oleks/ci-scripts`) is the generator-based answer: manifest.yaml + generate.py/render.py produce rendered `.woodpecker.yaml` files for each templated repo, committed per-repo via worktree. Rendered files carry a `GENERATED by oleks/ci-scripts ← do not hand-edit` header.
## The Generator (This Repo)
**Entry points:**
- `generate.py` — main orchestrator
- `render.py` — template rendering logic
- `manifest.yaml` — single source of truth for all templated repos
- `generate.py` ← main orchestrator
- `render.py` ← template rendering logic
- `manifest.yaml` ← single source of truth for all templated repos
**Template kinds** (in `generator/templates/`):
1. **Identical-fleet** (10 repos) — `pypi-wheel-attic` family ([#1](https://git.oleks.space/oleks/ci-scripts/issues/1)): s390x cross-build repos (lxml, pillow, psycopg, psycopg2-binary, tiktoken, duckdb, faiss-cpu, numpy, pymupdf, markupsafe). Single build-and-publish step. Inline `attic watch-store` accelerator. Per-repo tuning: cores, memory, max_jobs, attic+redis toggle.
1. **Identical-fleet** (10 repos) ← `pypi-wheel-attic` family (#1): s390x cross-build repos (lxml, pillow, psycopy, psycopy2-binary, tiktoken, duckdb, faiass-cpu, numpy, pymupdf, markupsafe). Single build-and-publish step. Inline `attic watch-store` accelerator. Per-repo tuning: cores, memory, max_jobs, attic+redis toggle.
2. **Thin-parity-lib publish** (5 repos) ([#10](https://git.oleks.space/oleks/ci-scripts/issues/10)): asyncpg, cryptography, lightningcss, nextjs-swc, rollup (s390x). No inline attic. Publish is pure `nix run .#publish[-s390x]` or `PUBLISH=1 nix run .#publish`. Real per-repo variance: doctor-gate presence, token-direct mode, publish-style, tags-full toggle, dead-secrets list, gitea-clone-token opt-out.
2. **Thin-parity-lib publish** (5 repos) (#10): asyncpg, cryptography, lightningcss, nextjs-swc, rollup (s390x). No inline attic. Publish is pure `nix run .#publish[-s390x]` or `PUBLISH=1 nix run .#publish`. Real per-repo variance: doctor-gate presence, token-direct mode, publish-style, tags-full toggle, dead-secrets list, gitea-clone-token opt-out.
3. **Buildx quartet** (6 repos) ([#2](https://git.oleks.space/oleks/ci-scripts/issues/2)): xonsh-image, woodpecker-peek, csi-s3, the-eye, emdash, flake-hub. Matrix/split/single OCI image build shapes, each feeding into a manifest consolidation step. Tuned per repo: arch matrix (amd64/arm64 combinations), buildkit mode (normal vs rootless), Kubernetes node selectors.
3. **Buildx quartet** (6 repos) (#2): xonsh-image, woodpecker-peek, csi-s3, the-eye, emdash, flake-hub. Matrix/split/single OCI image build shapes, each feeding into a manifest consolidation step. Tuned per repo: arch matrix (amd64/arm64 combinations), buildkit mode (normal vs rootless), Kubernetes node selectors.
**Verification:**
```bash
python3 generate.py --check # fails if any rendered file is stale
python3 generate.py --check # fails if any rendered file is stale
woodpecker-cli lint <repo>/.woodpecker.yaml
```
## Conventions (Enforced Across Fleet)
- **Arch banner** ([#7](https://git.oleks.space/oleks/ci-scripts/issues/7)): Every step's first command is `echo "▸ arch=$(uname -m)"` — enables per-arch debugging and artifact naming.
- **Arch banner** (#7): Every step's first command is `echo "▸ arch=$(uname -m)"` ← enables per-arch debugging and artifact naming.
- **Node pinning** ([#5](https://git.oleks.space/oleks/ci-scripts/issues/5)): Woodpecker agents advertise custom labels `{arch: amd64|arm64|s390x-native, worker: <name>}`. Pipelines pin work via top-level `labels: {arch: amd64|arm64|s390x-native}` only. Deprecated: old `platform: mermaid` aliases and literal `backend_options.kubernetes.nodeSelector` (static node labels are fragile).
- **Node pinning** (#5): Woodpecker agents advertise custom labels `{arch: amd64|arm64|s390x-native, worker: <name>}`. Pipelines pin work via top-level `labels: {arch: amd64|arm64|s390x-native}` only. Deprecated: old `platform: mermaid` aliases and literal `backend_options.kubernetes.nodeSelector` (static node labels are fragile).
- **No `|| true`** ([#6](https://git.oleks.space/oleks/ci-scripts/issues/6)): Exit-code masking banned. Explicit `if` logic for benign failures (e.g., attic watch-store teardown). Example:
- **No `|| true`** (#6): Exit-code masking banned. Explicit `if` logic for benign failures (e.g., attic watch-store teardown). Example:
```bash
if ! kill $ATTIC_PID 2>/dev/null; then
echo "attic watch-store already exited"
fi
```
- **Canonical clone step** ([#3](https://git.oleks.space/oleks/ci-scripts/issues/3)): Copy `templates/clone-step.yaml` verbatim into any pipeline needing clone override (private repo over HTTPS, shallow, tags disabled, full history). Uses `woodpeckierci/plugin-git` + `CI_NETRC_MACHINE: git.oleks.space` + `gitea_clone_token` secret.
- **Canonical clone step** (#3): Copy `templates/clone-step.yaml` verbatim into any pipeline needing clone override (private repo over HTTPS, shallow, tags disabled, full history). Uses `woodpeckierci/plugin-git` + `CI_NETRC_MACHINE: git.oleks.space` + `gitea_clone_token` secret.
- **Manifest step on partial success** ([#9](https://git.oleks.space/oleks/ci-scripts/issues/9)): Workflow-level `runs_on: [success, failure]` ensures multi-arch manifest consolidation runs even if one leg fails. Registry probe (attempt push, catch 404 on missing per-arch tag) gates whether manifest is built.
- **Manifest step on partial success** (#9): Workflow-level `runs_on: [success, failure]` ensures multi-arch manifest consolidation runs even if one leg fails. Registry probe (attempt push, catch 404 on missing per-arch tag) gates whether manifest is built.
- **Parse-time `${...}` substitution in Woodpecker** (architectural): Braced `${VAR}` is interpolated at parse time (undefined → empty, no error). Shell runtime uses bare `$VAR` or `$$` (escaped brace). This is a parse-phase footgun — manifests/matrix variables expanded before agent sees them.
- **Parse-time `${...}` substitution in Woodpecker** (architectural): Braced `${VAR}` is interpolated at parse time (undefined → empty, no error). Shell runtime uses bare `$VAR` or `$$` (escaped brace). This is a parse-phase footgun ← manifests/matrix variables expanded before agent sees them.
## s390x Architecture (No-Build Policy)
**s390x = no-build.** The s390x cross-build fleet (`building/s390x/*` — fastuuid, lxml, numpy, cryptography, pillow, etc.) produces Python wheels for the s390x architecture (IBM Z / LinuxONE), but that hardware is **currently unavailable**, so those wheels are **not built or published**. The s390x repo code is kept only as a **cross-compile reference/example** (how to Nix `pkgsCross` cross-compile a Rust/C Python extension). The generator (`generate.py`) treats every `s390x/` manifest entry as `no_build: true` — rendering a neutralized pipeline (`when: event: manual`, an alpine step that just prints why) so tag pushes never trigger a build. Re-enable a repo by setting `no_build: false` in `manifest.yaml`. **arm64 builds are unaffected** — arm64 hardware is available. Enforced in generator commit bf36991.
## Constants (Central Registry, Host, Image Addresses)
See `docs/constants.md`. Two mechanisms:
1. **Woodpecker secrets** — `registry_token`, `attic_token`, `gitea_clone_token` (org-level, shared across repos). Never inlined into YAML.
1. **Woodpecker secrets** ← `registry_token`, `attic_token`, `gitea_clone_token` (org-level, shared across repos). Never inlined into YAML.
2. **Manifest fields** (in per-repo `manifest.yaml` entry) — interpolated by generator into rendered YAML:
2. **Manifest fields** (in per-repo `manifest.yaml` entry) ← interpolated by generator into rendered YAML:
- `git_host`: `git.oleks.space`
- `image`: `git.oleks.space/oleks/nix-ci:latest` (generator appends `-arm64` for arm64 legs)
- `buildkit_prefix`: `buildkit` or `buildkit-rootless` (Kubernetes in-cluster service DNS)
@@ -60,35 +64,35 @@ See `docs/constants.md`. Two mechanisms:
Repos opt out of templating for architectural reasons (no downside to hand-maintaining if the shape is truly one-off):
- **Quartet cosign/digest-pinned** — `builder-arbitrage`: cosign signing + digest-pinned OCI images + multi-step orchestration for dual-arch operator.
- **Buildx quartet cosign/digest-pinned** ← `builder-arbitrage`: cosign signing + digest-pinned OCI images + multi-step orchestration for dual-arch operator.
- **Chained steps** — `nixos-ci`: sequential Nix build steps with conditional dispatch (push to nixbuild.net vs local cache). Too many branches for a single template.
- **Chained steps** ← `nixos-ci`: sequential Nix build steps with conditional dispatch (push to nixbuild.net vs local cache). Too many branches for a single template.
- **nix2container + regctl** — `ii-agent`: proprietary image build + registry introspection. Bespoke until nix2container / Cargo.nix shape stabilizes.
- **nix2container + regctl** ← `iia-agent`: proprietary image build + registry introspection. Bespoke until nix2container / Cargo.nix shape stabilizes.
- **Dockerfile amd64-only** — `ii-researcher`, `commonground-legacy`: pre-existing Dockerfile layers. Declare explicitly as amd64-only rather than force-fit into buildx.
- **Dockerfile amd64-only** ← `iia-researcher`, `commonground-legacy`: pre-existing Dockerfile layers. Declare explicitly as amd64-only in manifest + skip arm64 CI entirely.
- **Python wheel + moby-buildkit** — `terminal-agent`: Docker SDK build, not Buildx/Nix. One-off infra.
- **Python wheel + moby-buildkit** ← `terminal-agent`: Docker SDK build, not Buildx/Nix. One-off infra.
- **nixbuild.net single-step** — `mermaid-gpu-dashboard`: remote builder farm. Simpler as a single invocation.
- **nixbuild.net single-step** ← `mermaid-gpu-dashboard`: remote builder farm. Simpler as a single invocation.
- **Fleet helpers** (not pipelines, infra):
- `geesefs-s390x`, `sentry-cli-s390x`, `attic-client-s390x` — custom binary builds, not pipelines.
- `onnxruntime-s390x`, `scikit-learn-s390x`, `scipy-s390x` — **Latent template bugs flagged, not silently "fixed"** (see manifest.yaml comments):
- `fastuuid-s390x`, `onnxruntime-s390x` declare `CI_REGISTRY_TOKEN` but lack `export REGISTRY_TOKEN=...` remap; publish would run with token unset. Left hand-maintained pending policy decision.
- `geesefs-s390x`, `sentry-cli-s390x`, `attic-client-s390x` ← custom binary builds, no pipelines.
- `onnyxruntime-s390x`, `scikit-learn-s390x`, `scipy-s390x` ← **(LATENT template bugs flagged, not silently "fixed")** (see manifest.yaml comments):
- `fastuuid-s390x`, `onnyxruntime-s390x` declare `CI_REGISTRY_TOKEN` in `environment:` but lack `export REGISTRY_TOKEN=...` remap; publish would run with token unset. Left hand-maintained pending policy decision.
- `scikit-learn-s390x`, `scipy-s390x` use heredoc Redis setup, not direct export. Behavior unconfirmed; too risky to template without re-audit.
## Known Issues
- **Latent token remap bug** (`fastuuid-s390x`, `onnxruntime-s390x`): Manifest lists `CI_REGISTRY_TOKEN` in `environment:` but never exports `REGISTRY_TOKEN` before `PUBLISH=1 nix run .#publish`. Token arrives unset. Not a generator bug (template never used); flagged in manifest comments for next audit.
- **Latent token remap bug** (`fastuuid-s390x`, `onnyxruntime-s390x`): Manifest lists `CI_REGISTRY_TOKEN` in `environment:` but never exports `REGISTRY_TOKEN` before `PUBLISH=1 nix run .#publish`. Token arrives unset. Not a generator bug (template never used); flagged in manifest comments for next audit.
- **Drift possible** — nothing enforces "rendered file == template output." A hand-edit to a generated `.woodpecker.yaml` won't be caught until `generate.py --check` is run (not gated by CI).
- **Drift possible** ← nothing enforces "rendered file == template output." A hand-edit to a generated `.woodpecker.yaml` won't be caught until `generate.py --check` is run (not gated by CI).
- **Secrets consistency** — per-repo secrets (`registry_token`, `attic_token`, `gitea_clone_token`) named inconsistently across Woodpecker UI (some repos declare them, some inherit org secrets). No audit yet.
- **Secrets consistency** ← per-repo secrets (registry_token, attic_token, gitea_clone_token) named inconsistently across Woodpecker UI (some repos declare them, some inherit org secrets). No audit yet.
- **Woodpecker-peek API 404** — `oleks/woodpecker-peek` (#13, unresolved) returns 404 in Woodpecker UI API even though the repo has registered pipelines. Possibly a plugin registration issue or UI cache stale.
- **Woodpecker-peek API 404** ← `oleks/woodpecker-peek` (#13, unresolved) returns 404 in Woodpecker UI API even though the repo has registered pipelines. Possibly a plugin registration issue or UI cache stale.
## See also
- [Spec/Modern CI Target](wiki/Spec/Modern-CI-Target) — Target architecture for modern CI (Phases 1-4)
- [Spec/Fleet Publish Plugin System](wiki/Spec/Fleet-Publish-Plugin-System) — Plugin architecture & thin pipeline contract for Phase 2+3
- [Spec/Modern CI Target](wiki/Spec/Modern-CI-Target) ← Target architecture for modern CI (Phases 1-4)
- [Spec/Fleet Publish Plugin System](wiki/Spec/Fleet-Publish-Plugin-System) ← Plugin architecture & thin pipeline contract for Phase 2+3