Add wiki page: Spec/Current CI Ecosystem (2026-07-04)
@@ -0,0 +1,89 @@
|
||||
## Overview
|
||||
|
||||
Woodpecker CI server running at `ci.oleks.space` (host-native systemd service on armer, agents distributed in k3s). Gitea forge at `git.oleks.space` serving as both source control and OCI/npm/PyPI registry. Fleet covers ~50 repositories with ~97 Woodpecker pipeline files as of 2026-07-04.
|
||||
|
||||
Woodpecker has no native `include:` mechanism for cross-repo YAML reuse. This repo (`oleks/ci-scripts`) is the generator-based answer: manifest.yaml + generate.py/render.py produce rendered `.woodpecker.yaml` files for each templated repo, committed per-repo via worktree. Rendered files carry a `GENERATED by oleks/ci-scripts — do not hand-edit` header.
|
||||
|
||||
## The Generator (This Repo)
|
||||
|
||||
**Entry points:**
|
||||
- `generate.py` — main orchestrator
|
||||
- `render.py` — template rendering logic
|
||||
- `manifest.yaml` — single source of truth for all templated repos
|
||||
|
||||
**Template kinds** (in `generator/templates/`):
|
||||
1. **Identical-fleet** (10 repos) — `pypi-wheel-attic` family ([#1](https://git.oleks.space/oleks/ci-scripts/issues/1)): s390x cross-build repos (lxml, pillow, psycopg, psycopg2-binary, tiktoken, duckdb, faiss-cpu, numpy, pymupdf, markupsafe). Single build-and-publish step. Inline `attic watch-store` accelerator. Per-repo tuning: cores, memory, max_jobs, attic+redis toggle.
|
||||
|
||||
2. **Thin-parity-lib publish** (5 repos) ([#10](https://git.oleks.space/oleks/ci-scripts/issues/10)): asyncpg, cryptography, lightningcss, nextjs-swc, rollup (s390x). No inline attic. Publish is pure `nix run .#publish[-s390x]` or `PUBLISH=1 nix run .#publish`. Real per-repo variance: doctor-gate presence, token-direct mode, publish-style, tags-full toggle, dead-secrets list, gitea-clone-token opt-out.
|
||||
|
||||
3. **Buildx quartet** (6 repos) ([#2](https://git.oleks.space/oleks/ci-scripts/issues/2)): xonsh-image, woodpecker-peek, csi-s3, the-eye, emdash, flake-hub. Matrix/split/single OCI image build shapes, each feeding into a manifest consolidation step. Tuned per repo: arch matrix (amd64/arm64 combinations), buildkit mode (normal vs rootless), Kubernetes node selectors.
|
||||
|
||||
**Verification:**
|
||||
```bash
|
||||
python3 generate.py --check # fails if any rendered file is stale
|
||||
woodpecker-cli lint <repo>/.woodpecker.yaml
|
||||
```
|
||||
|
||||
## Conventions (Enforced Across Fleet)
|
||||
|
||||
- **Arch banner** ([#7](https://git.oleks.space/oleks/ci-scripts/issues/7)): Every step's first command is `echo "▸ arch=$(uname -m)"` — enables per-arch debugging and artifact naming.
|
||||
|
||||
- **Node pinning** ([#5](https://git.oleks.space/oleks/ci-scripts/issues/5)): Woodpecker agents advertise custom labels `{arch: amd64|arm64|s390x-native, worker: <name>}`. Pipelines pin work via top-level `labels: {arch: amd64|arm64|s390x-native}` only. Deprecated: old `platform: mermaid` aliases and literal `backend_options.kubernetes.nodeSelector` (static node labels are fragile).
|
||||
|
||||
- **No `|| true`** ([#6](https://git.oleks.space/oleks/ci-scripts/issues/6)): Exit-code masking banned. Explicit `if` logic for benign failures (e.g., attic watch-store teardown). Example:
|
||||
```bash
|
||||
if ! kill $ATTIC_PID 2>/dev/null; then
|
||||
echo "attic watch-store already exited"
|
||||
fi
|
||||
```
|
||||
|
||||
- **Canonical clone step** ([#3](https://git.oleks.space/oleks/ci-scripts/issues/3)): Copy `templates/clone-step.yaml` verbatim into any pipeline needing clone override (private repo over HTTPS, shallow, tags disabled, full history). Uses `woodpeckierci/plugin-git` + `CI_NETRC_MACHINE: git.oleks.space` + `gitea_clone_token` secret.
|
||||
|
||||
- **Manifest step on partial success** ([#9](https://git.oleks.space/oleks/ci-scripts/issues/9)): Workflow-level `runs_on: [success, failure]` ensures multi-arch manifest consolidation runs even if one leg fails. Registry probe (attempt push, catch 404 on missing per-arch tag) gates whether manifest is built.
|
||||
|
||||
- **Parse-time `${...}` substitution in Woodpecker** (architectural): Braced `${VAR}` is interpolated at parse time (undefined → empty, no error). Shell runtime uses bare `$VAR` or `$$` (escaped brace). This is a parse-phase footgun — manifests/matrix variables expanded before agent sees them.
|
||||
|
||||
## Constants (Central Registry, Host, Image Addresses)
|
||||
|
||||
See `docs/constants.md`. Two mechanisms:
|
||||
|
||||
1. **Woodpecker secrets** — `registry_token`, `attic_token`, `gitea_clone_token` (org-level, shared across repos). Never inlined into YAML.
|
||||
|
||||
2. **Manifest fields** (in per-repo `manifest.yaml` entry) — interpolated by generator into rendered YAML:
|
||||
- `git_host`: `git.oleks.space`
|
||||
- `image`: `git.oleks.space/oleks/nix-ci:latest` (generator appends `-arm64` for arm64 legs)
|
||||
- `buildkit_prefix`: `buildkit` or `buildkit-rootless` (Kubernetes in-cluster service DNS)
|
||||
- `nameserver`: `169.254.20.10` (needed for s390x cross-builds where worker DNS is out of scope)
|
||||
- `node_selector`: optional `kubernetes.io/hostname` or custom labels (per-repo, for csi-s3/other pinned workloads)
|
||||
|
||||
## Not Covered / Bespoke
|
||||
|
||||
Repos opt out of templating for architectural reasons (no downside to hand-maintaining if the shape is truly one-off):
|
||||
|
||||
- **Quartet cosign/digest-pinned** — `builder-arbitrage`: cosign signing + digest-pinned OCI images + multi-step orchestration for dual-arch operator.
|
||||
|
||||
- **Chained steps** — `nixos-ci`: sequential Nix build steps with conditional dispatch (push to nixbuild.net vs local cache). Too many branches for a single template.
|
||||
|
||||
- **nix2container + regctl** — `ii-agent`: proprietary image build + registry introspection. Bespoke until nix2container / Cargo.nix shape stabilizes.
|
||||
|
||||
- **Dockerfile amd64-only** — `ii-researcher`, `commonground-legacy`: pre-existing Dockerfile layers. Declare explicitly as amd64-only rather than force-fit into buildx.
|
||||
|
||||
- **Python wheel + moby-buildkit** — `terminal-agent`: Docker SDK build, not Buildx/Nix. One-off infra.
|
||||
|
||||
- **nixbuild.net single-step** — `mermaid-gpu-dashboard`: remote builder farm. Simpler as a single invocation.
|
||||
|
||||
- **Fleet helpers** (not pipelines, infra):
|
||||
- `geesefs-s390x`, `sentry-cli-s390x`, `attic-client-s390x` — custom binary builds, not pipelines.
|
||||
- `onnxruntime-s390x`, `scikit-learn-s390x`, `scipy-s390x` — **Latent template bugs flagged, not silently "fixed"** (see manifest.yaml comments):
|
||||
- `fastuuid-s390x`, `onnxruntime-s390x` declare `CI_REGISTRY_TOKEN` but lack `export REGISTRY_TOKEN=...` remap; publish would run with token unset. Left hand-maintained pending policy decision.
|
||||
- `scikit-learn-s390x`, `scipy-s390x` use heredoc Redis setup, not direct export. Behavior unconfirmed; too risky to template without re-audit.
|
||||
|
||||
## Known Issues
|
||||
|
||||
- **Latent token remap bug** (`fastuuid-s390x`, `onnxruntime-s390x`): Manifest lists `CI_REGISTRY_TOKEN` in `environment:` but never exports `REGISTRY_TOKEN` before `PUBLISH=1 nix run .#publish`. Token arrives unset. Not a generator bug (template never used); flagged in manifest comments for next audit.
|
||||
|
||||
- **Drift possible** — nothing enforces "rendered file == template output." A hand-edit to a generated `.woodpecker.yaml` won't be caught until `generate.py --check` is run (not gated by CI).
|
||||
|
||||
- **Secrets consistency** — per-repo secrets (`registry_token`, `attic_token`, `gitea_clone_token`) named inconsistently across Woodpecker UI (some repos declare them, some inherit org secrets). No audit yet.
|
||||
|
||||
- **Woodpecker-peek API 404** — `oleks/woodpecker-peek` (#13, unresolved) returns 404 in Woodpecker UI API even though the repo has registered pipelines. Possibly a plugin registration issue or UI cache stale.
|
||||
Reference in New Issue
Block a user