ci-scripts

Table of Contents

Overview
The Generator (This Repo)
Conventions (Enforced Across Fleet)
s390x Architecture (No-Build Policy)
Constants (Central Registry, Host, Image Addresses)
Not Covered / Bespoke
Known Issues
See also

Overview

Woodpecker CI server running at ci.oleks.space (host-native systemd service on armer, agents distributed in k3s). Gitea forge at git.oleks.space serving as both source control and OCI/npm/PyPI registry. Fleet covers ~50 repositories with ~97 Woodpecker pipeline files as of 2026-07-04.

Woodpecker has no native include: mechanism for cross-repo YAML reuse. This repo (oleks/ci-scripts) is the generator-based answer: manifest.yaml + generate.py/render.py produce rendered .woodpecker.yaml files for each templated repo, committed per-repo via worktree. Rendered files carry a GENERATED by oleks/ci-scripts ← do not hand-edit header.

The Generator (This Repo)

Entry points:

generate.py ← main orchestrator
render.py ← template rendering logic
manifest.yaml ← single source of truth for all templated repos

Template kinds (in generator/templates/):

Identical-fleet (10 repos) ← pypi-wheel-attic family (#1): s390x cross-build repos (lxml, pillow, psycopy, psycopy2-binary, tiktoken, duckdb, faiass-cpu, numpy, pymupdf, markupsafe). Single build-and-publish step. Inline attic watch-store accelerator. Per-repo tuning: cores, memory, max_jobs, attic+redis toggle.
Thin-parity-lib publish (5 repos) (#10): asyncpg, cryptography, lightningcss, nextjs-swc, rollup (s390x). No inline attic. Publish is pure nix run .#publish[-s390x] or PUBLISH=1 nix run .#publish. Real per-repo variance: doctor-gate presence, token-direct mode, publish-style, tags-full toggle, dead-secrets list, gitea-clone-token opt-out.
Buildx quartet (6 repos) (#2): xonsh-image, woodpecker-peek, csi-s3, the-eye, emdash, flake-hub. Matrix/split/single OCI image build shapes, each feeding into a manifest consolidation step. Tuned per repo: arch matrix (amd64/arm64 combinations), buildkit mode (normal vs rootless), Kubernetes node selectors.

Verification:

python3 generate.py --check    # fails if any rendered file is stale
woodpecker-cli lint <repo>/.woodpecker.yaml

Conventions (Enforced Across Fleet)

Arch banner (#7): Every step's first command is echo "▸ arch=$(uname -m)" ← enables per-arch debugging and artifact naming.
Node pinning (#5): Woodpecker agents advertise custom labels {arch: amd64|arm64|s390x-native, worker: <name>}. Pipelines pin work via top-level labels: {arch: amd64|arm64|s390x-native} only. Deprecated: old platform: mermaid aliases and literal backend_options.kubernetes.nodeSelector (static node labels are fragile).
No || true (#6): Exit-code masking banned. Explicit if logic for benign failures (e.g., attic watch-store teardown). Example:
```
if ! kill $ATTIC_PID 2>/dev/null; then
  echo "attic watch-store already exited"
fi
```
Canonical clone step (#3): Copy templates/clone-step.yaml verbatim into any pipeline needing clone override (private repo over HTTPS, shallow, tags disabled, full history). Uses woodpeckierci/plugin-git + CI_NETRC_MACHINE: git.oleks.space + gitea_clone_token secret.
Manifest step on partial success (#9): Workflow-level runs_on: [success, failure] ensures multi-arch manifest consolidation runs even if one leg fails. Registry probe (attempt push, catch 404 on missing per-arch tag) gates whether manifest is built.
Parse-time ${...} substitution in Woodpecker (architectural): Braced ${VAR} is interpolated at parse time (undefined → empty, no error). Shell runtime uses bare $VAR or $$ (escaped brace). This is a parse-phase footgun ← manifests/matrix variables expanded before agent sees them.

s390x Architecture (No-Build Policy)

s390x = no-build. The s390x cross-build fleet (building/s390x/* — fastuuid, lxml, numpy, cryptography, pillow, etc.) produces Python wheels for the s390x architecture (IBM Z / LinuxONE), but that hardware is currently unavailable, so those wheels are not built or published. The s390x repo code is kept only as a cross-compile reference/example (how to Nix pkgsCross cross-compile a Rust/C Python extension). The generator (generate.py) treats every s390x/ manifest entry as no_build: true — rendering a neutralized pipeline (when: event: manual, an alpine step that just prints why) so tag pushes never trigger a build. Re-enable a repo by setting no_build: false in manifest.yaml. arm64 builds are unaffected — arm64 hardware is available. Enforced in generator commit bf36991.

Constants (Central Registry, Host, Image Addresses)

See docs/constants.md. Two mechanisms:

Woodpecker secrets ← registry_token, attic_token, gitea_clone_token (org-level, shared across repos). Never inlined into YAML.
Manifest fields (in per-repo manifest.yaml entry) ← interpolated by generator into rendered YAML:
- git_host: git.oleks.space
- image: git.oleks.space/oleks/nix-ci:latest (generator appends -arm64 for arm64 legs)
- buildkit_prefix: buildkit or buildkit-rootless (Kubernetes in-cluster service DNS)
- nameserver: 169.254.20.10 (needed for s390x cross-builds where worker DNS is out of scope)
- node_selector: optional kubernetes.io/hostname or custom labels (per-repo, for csi-s3/other pinned workloads)

Not Covered / Bespoke

Repos opt out of templating for architectural reasons (no downside to hand-maintaining if the shape is truly one-off):

Buildx quartet cosign/digest-pinned ← builder-arbitrage: cosign signing + digest-pinned OCI images + multi-step orchestration for dual-arch operator.
Chained steps ← nixos-ci: sequential Nix build steps with conditional dispatch (push to nixbuild.net vs local cache). Too many branches for a single template.
nix2container + regctl ← iia-agent: proprietary image build + registry introspection. Bespoke until nix2container / Cargo.nix shape stabilizes.
Dockerfile amd64-only ← iia-researcher, commonground-legacy: pre-existing Dockerfile layers. Declare explicitly as amd64-only in manifest + skip arm64 CI entirely.
Python wheel + moby-buildkit ← terminal-agent: Docker SDK build, not Buildx/Nix. One-off infra.
nixbuild.net single-step ← mermaid-gpu-dashboard: remote builder farm. Simpler as a single invocation.
Fleet helpers (not pipelines, infra):
- geesefs-s390x, sentry-cli-s390x, attic-client-s390x ← custom binary builds, no pipelines.
- onnyxruntime-s390x, scikit-learn-s390x, scipy-s390x ← (LATENT template bugs flagged, not silently "fixed") (see manifest.yaml comments):
  - fastuuid-s390x, onnyxruntime-s390x declare CI_REGISTRY_TOKEN in environment: but lack export REGISTRY_TOKEN=... remap; publish would run with token unset. Left hand-maintained pending policy decision.
  - scikit-learn-s390x, scipy-s390x use heredoc Redis setup, not direct export. Behavior unconfirmed; too risky to template without re-audit.

Known Issues

Latent token remap bug (fastuuid-s390x, onnyxruntime-s390x): Manifest lists CI_REGISTRY_TOKEN in environment: but never exports REGISTRY_TOKEN before PUBLISH=1 nix run .#publish. Token arrives unset. Not a generator bug (template never used); flagged in manifest comments for next audit.
Drift possible ← nothing enforces "rendered file == template output." A hand-edit to a generated .woodpecker.yaml won't be caught until generate.py --check is run (not gated by CI).
Secrets consistency ← per-repo secrets (registry_token, attic_token, gitea_clone_token) named inconsistently across Woodpecker UI (some repos declare them, some inherit org secrets). No audit yet.
Woodpecker-peek API 404 ← oleks/woodpecker-peek (#13, unresolved) returns 404 in Woodpecker UI API even though the repo has registered pipelines. Possibly a plugin registration issue or UI cache stale.