commit a6e37ccc93294e2ee4cd424883e487257b1a834c Author: oleks Date: Sat Jul 4 20:19:42 2026 +0300 Add wiki page: Spec/Current CI Ecosystem (2026-07-04) diff --git a/Spec%2FCurrent-CI-Ecosystem.md b/Spec%2FCurrent-CI-Ecosystem.md new file mode 100644 index 0000000..95c27f2 --- /dev/null +++ b/Spec%2FCurrent-CI-Ecosystem.md @@ -0,0 +1,89 @@ +## Overview + +Woodpecker CI server running at `ci.oleks.space` (host-native systemd service on armer, agents distributed in k3s). Gitea forge at `git.oleks.space` serving as both source control and OCI/npm/PyPI registry. Fleet covers ~50 repositories with ~97 Woodpecker pipeline files as of 2026-07-04. + +Woodpecker has no native `include:` mechanism for cross-repo YAML reuse. This repo (`oleks/ci-scripts`) is the generator-based answer: manifest.yaml + generate.py/render.py produce rendered `.woodpecker.yaml` files for each templated repo, committed per-repo via worktree. Rendered files carry a `GENERATED by oleks/ci-scripts — do not hand-edit` header. + +## The Generator (This Repo) + +**Entry points:** +- `generate.py` — main orchestrator +- `render.py` — template rendering logic +- `manifest.yaml` — single source of truth for all templated repos + +**Template kinds** (in `generator/templates/`): +1. **Identical-fleet** (10 repos) — `pypi-wheel-attic` family ([#1](https://git.oleks.space/oleks/ci-scripts/issues/1)): s390x cross-build repos (lxml, pillow, psycopg, psycopg2-binary, tiktoken, duckdb, faiss-cpu, numpy, pymupdf, markupsafe). Single build-and-publish step. Inline `attic watch-store` accelerator. Per-repo tuning: cores, memory, max_jobs, attic+redis toggle. + +2. **Thin-parity-lib publish** (5 repos) ([#10](https://git.oleks.space/oleks/ci-scripts/issues/10)): asyncpg, cryptography, lightningcss, nextjs-swc, rollup (s390x). No inline attic. Publish is pure `nix run .#publish[-s390x]` or `PUBLISH=1 nix run .#publish`. Real per-repo variance: doctor-gate presence, token-direct mode, publish-style, tags-full toggle, dead-secrets list, gitea-clone-token opt-out. + +3. **Buildx quartet** (6 repos) ([#2](https://git.oleks.space/oleks/ci-scripts/issues/2)): xonsh-image, woodpecker-peek, csi-s3, the-eye, emdash, flake-hub. Matrix/split/single OCI image build shapes, each feeding into a manifest consolidation step. Tuned per repo: arch matrix (amd64/arm64 combinations), buildkit mode (normal vs rootless), Kubernetes node selectors. + +**Verification:** +```bash +python3 generate.py --check # fails if any rendered file is stale +woodpecker-cli lint /.woodpecker.yaml +``` + +## Conventions (Enforced Across Fleet) + +- **Arch banner** ([#7](https://git.oleks.space/oleks/ci-scripts/issues/7)): Every step's first command is `echo "▸ arch=$(uname -m)"` — enables per-arch debugging and artifact naming. + +- **Node pinning** ([#5](https://git.oleks.space/oleks/ci-scripts/issues/5)): Woodpecker agents advertise custom labels `{arch: amd64|arm64|s390x-native, worker: }`. Pipelines pin work via top-level `labels: {arch: amd64|arm64|s390x-native}` only. Deprecated: old `platform: mermaid` aliases and literal `backend_options.kubernetes.nodeSelector` (static node labels are fragile). + +- **No `|| true`** ([#6](https://git.oleks.space/oleks/ci-scripts/issues/6)): Exit-code masking banned. Explicit `if` logic for benign failures (e.g., attic watch-store teardown). Example: + ```bash + if ! kill $ATTIC_PID 2>/dev/null; then + echo "attic watch-store already exited" + fi + ``` + +- **Canonical clone step** ([#3](https://git.oleks.space/oleks/ci-scripts/issues/3)): Copy `templates/clone-step.yaml` verbatim into any pipeline needing clone override (private repo over HTTPS, shallow, tags disabled, full history). Uses `woodpeckierci/plugin-git` + `CI_NETRC_MACHINE: git.oleks.space` + `gitea_clone_token` secret. + +- **Manifest step on partial success** ([#9](https://git.oleks.space/oleks/ci-scripts/issues/9)): Workflow-level `runs_on: [success, failure]` ensures multi-arch manifest consolidation runs even if one leg fails. Registry probe (attempt push, catch 404 on missing per-arch tag) gates whether manifest is built. + +- **Parse-time `${...}` substitution in Woodpecker** (architectural): Braced `${VAR}` is interpolated at parse time (undefined → empty, no error). Shell runtime uses bare `$VAR` or `$$` (escaped brace). This is a parse-phase footgun — manifests/matrix variables expanded before agent sees them. + +## Constants (Central Registry, Host, Image Addresses) + +See `docs/constants.md`. Two mechanisms: + +1. **Woodpecker secrets** — `registry_token`, `attic_token`, `gitea_clone_token` (org-level, shared across repos). Never inlined into YAML. + +2. **Manifest fields** (in per-repo `manifest.yaml` entry) — interpolated by generator into rendered YAML: + - `git_host`: `git.oleks.space` + - `image`: `git.oleks.space/oleks/nix-ci:latest` (generator appends `-arm64` for arm64 legs) + - `buildkit_prefix`: `buildkit` or `buildkit-rootless` (Kubernetes in-cluster service DNS) + - `nameserver`: `169.254.20.10` (needed for s390x cross-builds where worker DNS is out of scope) + - `node_selector`: optional `kubernetes.io/hostname` or custom labels (per-repo, for csi-s3/other pinned workloads) + +## Not Covered / Bespoke + +Repos opt out of templating for architectural reasons (no downside to hand-maintaining if the shape is truly one-off): + +- **Quartet cosign/digest-pinned** — `builder-arbitrage`: cosign signing + digest-pinned OCI images + multi-step orchestration for dual-arch operator. + +- **Chained steps** — `nixos-ci`: sequential Nix build steps with conditional dispatch (push to nixbuild.net vs local cache). Too many branches for a single template. + +- **nix2container + regctl** — `ii-agent`: proprietary image build + registry introspection. Bespoke until nix2container / Cargo.nix shape stabilizes. + +- **Dockerfile amd64-only** — `ii-researcher`, `commonground-legacy`: pre-existing Dockerfile layers. Declare explicitly as amd64-only rather than force-fit into buildx. + +- **Python wheel + moby-buildkit** — `terminal-agent`: Docker SDK build, not Buildx/Nix. One-off infra. + +- **nixbuild.net single-step** — `mermaid-gpu-dashboard`: remote builder farm. Simpler as a single invocation. + +- **Fleet helpers** (not pipelines, infra): + - `geesefs-s390x`, `sentry-cli-s390x`, `attic-client-s390x` — custom binary builds, not pipelines. + - `onnxruntime-s390x`, `scikit-learn-s390x`, `scipy-s390x` — **Latent template bugs flagged, not silently "fixed"** (see manifest.yaml comments): + - `fastuuid-s390x`, `onnxruntime-s390x` declare `CI_REGISTRY_TOKEN` but lack `export REGISTRY_TOKEN=...` remap; publish would run with token unset. Left hand-maintained pending policy decision. + - `scikit-learn-s390x`, `scipy-s390x` use heredoc Redis setup, not direct export. Behavior unconfirmed; too risky to template without re-audit. + +## Known Issues + +- **Latent token remap bug** (`fastuuid-s390x`, `onnxruntime-s390x`): Manifest lists `CI_REGISTRY_TOKEN` in `environment:` but never exports `REGISTRY_TOKEN` before `PUBLISH=1 nix run .#publish`. Token arrives unset. Not a generator bug (template never used); flagged in manifest comments for next audit. + +- **Drift possible** — nothing enforces "rendered file == template output." A hand-edit to a generated `.woodpecker.yaml` won't be caught until `generate.py --check` is run (not gated by CI). + +- **Secrets consistency** — per-repo secrets (`registry_token`, `attic_token`, `gitea_clone_token`) named inconsistently across Woodpecker UI (some repos declare them, some inherit org secrets). No audit yet. + +- **Woodpecker-peek API 404** — `oleks/woodpecker-peek` (#13, unresolved) returns 404 in Woodpecker UI API even though the repo has registered pipelines. Possibly a plugin registration issue or UI cache stale.