Add wiki page: Spec/Modern CI Target (2026-07-04)
@@ -0,0 +1,102 @@
|
||||
## Goals
|
||||
|
||||
1. **One source of truth** enforced (not manual): every repo's CI registration and pipeline shape live in a canonical registry, backed by a verification job that catches drift.
|
||||
|
||||
2. **Fewer pipeline shapes** → converged DSL: reduce 6+ custom YAML patterns down to ~2 (one for Nix+Attic builds, one for container multi-arch).
|
||||
|
||||
3. **No copy-paste** — secrets standardized, one token name per kind, every repo audited for registration + secret presence.
|
||||
|
||||
4. **Reproducible transitions** — phases are sequential; each phase closes a GitHub issue and is verifiable before the next begins.
|
||||
|
||||
## Target Architecture (By Phase)
|
||||
|
||||
### Phase 1: EPIC #8 (DONE 2026-07-04)
|
||||
|
||||
Baseline generator + fleet conventions established. Closed issues:
|
||||
|
||||
- [#1](https://git.oleks.space/oleks/ci-scripts/issues/1) Consolidate s390x/arm64 cross-build fleet into one shared template
|
||||
- [#2](https://git.oleks.space/oleks/ci-scripts/issues/2) Extract shared buildx multi-arch quartet (amd64/arm64/manifest/chart)
|
||||
- [#3](https://git.oleks.space/oleks/ci-scripts/issues/3) Shared clone step definition
|
||||
- [#4](https://git.oleks.space/oleks/ci-scripts/issues/4) Centralize registry/host constants
|
||||
- [#5](https://git.oleks.space/oleks/ci-scripts/issues/5) Unify node-pinning label keys
|
||||
- [#6](https://git.oleks.space/oleks/ci-scripts/issues/6) Audit and remove copied `|| true` failure-masking
|
||||
- [#7](https://git.oleks.space/oleks/ci-scripts/issues/7) Split monolithic step scripts + banner gaps
|
||||
- [#9](https://git.oleks.space/oleks/ci-scripts/issues/9) Manifest step must not be skipped on partial arch success
|
||||
- [#10](https://git.oleks.space/oleks/ci-scripts/issues/10) Second fleet template for the parity-lib publish family
|
||||
|
||||
**Artifacts:** `manifest.yaml` + `generate.py` + `render.py`, 15 templated repos, arch-banner + no `|| true` + node-pinning conventions.
|
||||
|
||||
### Phase 2: Enforcement & Hygiene
|
||||
|
||||
**Deliverables:**
|
||||
|
||||
1. **Drift detection pipeline** — scheduled job or per-push step in `oleks/ci-scripts`:
|
||||
- Re-render all manifests locally.
|
||||
- Diff against each repo's `HEAD:.woodpecker.yaml`.
|
||||
- Open/update issues on drift (title: "ci-scripts drift detected", body: diff). Auto-close when diff clears.
|
||||
- Prevents silent hand-edits and regeneration skips.
|
||||
|
||||
2. **Org-level Woodpecker secrets** — migrate per-repo copies of `registry_token`, `attic_token`, `gitea_clone_token` to org scope. Verify every repo that references a secret actually has it (no 404s from undefined secrets at runtime).
|
||||
|
||||
3. **Token naming fix** — standardize on one token name. Currently:
|
||||
- `registry_token` (correct, org secret).
|
||||
- `CI_REGISTRY_TOKEN` (per-repo env) → `REGISTRY_TOKEN` (shell export, latent bug on fastuuid/onnxruntime).
|
||||
- **Decision required:** rename all to `registry_token` org secret, or rename the secret to `ci_registry_token` and update all templates?
|
||||
|
||||
4. **CI registration audit** — verify every repo that **should** have a Woodpecker pipeline is actually registered (no 404s like `oleks/woodpecker-peek` #13). Link repos in manifest to Woodpecker API `/api/v1/repos/<owner>/<name>` and confirm at least one pipeline is present.
|
||||
|
||||
**Success criteria:** All registered repos have matching drift-free `.woodpecker.yaml` files, all secrets resolve, zero undefined-secret runtime errors.
|
||||
|
||||
### Phase 3: Shape Convergence
|
||||
|
||||
**Goal:** Replace generated YAML bulk with thin parameterized pipelines calling versioned shared code.
|
||||
|
||||
**Strategy:**
|
||||
|
||||
1. **OCI image bundle** — package `ci-scripts` releases as a versioned Woodpecker plugin image or standalone script bundle (`nix flake` app):
|
||||
- Encapsulates build → push → manifest-consolidate logic (buildx matrix/split/single, registry probe, digest pinning).
|
||||
- Per-repo pipeline becomes ~10 lines: clone, banner, `nix run git.oleks.space/oleks/ci-scripts#build-and-publish -- --config <per-repo-json>`.
|
||||
- Centralizes registry-probe + manifest logic (no per-repo duplication).
|
||||
|
||||
2. **Parity-lib convergence** — fold flat `nix` builds onto shared `oleks/parity-lib` helpers (precedent: `foldImageLayers`, shared flake-module for per-arch image builds). Reduce buildx templates to a single pluggable shape.
|
||||
|
||||
3. **Target result:** 3 pipeline shapes → 1 parameterized shape for Nix+container builds. Bespoke repos (quoted in "Not Covered") remain exempt but documented.
|
||||
|
||||
**Success criteria:** Every templated repo's `.woodpecker.yaml` is <100 lines, most logic is version-pinned calls to published scripts/images, generator still verifies correctness but doesn't emit bulk YAML.
|
||||
|
||||
### Phase 4: Fold Bespoke Tail
|
||||
|
||||
**Goal:** Migrate or explicitly declare permanent-bespoke repos.
|
||||
|
||||
**Case-by-case audit:**
|
||||
- `ii-researcher`, `commonground-legacy` (amd64-only Dockerfile) — add arm64 legs via `buildx` plugin or declare amd64-only in manifest + skip arm64 CI entirely?
|
||||
- `fastuuid-s390x`, `onnxruntime-s390x` (token remap bug) — template them into thin-publish once token bug is understood?
|
||||
- `scikit-learn-s390x`, `scipy-s390x` (heredoc Redis) — normalize Redis setup to direct export and fold into thin-publish?
|
||||
- `builder-arbitrage` (cosign + digest-pinned) — keep as bespoke (true multi-step orchestration), document rationale.
|
||||
- `nixos-ci`, `ii-agent`, `terminal-agent`, `mermaid-gpu-dashboard` — stay bespoke (architectural uniqueness), link to justification docs.
|
||||
|
||||
**Result:** A final "permanent bespoke" list in the manifest with reasons for each (and links to the issues/docs justifying them).
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- **No move off Woodpecker** — this is a Woodpecker optimization, not a migration to GitHub Actions / other CI.
|
||||
- **No forcing bespoke toolchains prematurely** — `nix2container`, `cosign`, moby-buildkit stay in repos that own them; don't fold them into shared templates until stable.
|
||||
|
||||
## Invariants (Must Survive Transition)
|
||||
|
||||
- **Arch banner** — every step's first line `echo "▸ arch=$(uname -m)"`.
|
||||
- **No `|| true`** — explicit exit-code handling everywhere.
|
||||
- **Arch-only labels** — `labels: {arch: amd64|arm64|s390x-native}` is the sole node-pinning knob; no `nodeSelector` or `platform` aliases.
|
||||
- **Parse-time `${}`** — braced variables are parse-phase (Woodpecker substitution); shell uses `$VAR` or `$$`. Discipline enforced by linter or guide.
|
||||
- **Manifest-on-partial-success** — multi-arch workflows must not skip manifest consolidation if one arch fails (registry probe gates it).
|
||||
- **GENERATED headers** — wherever `.woodpecker.yaml` files are still rendered (even in Phase 3, if thin pipelines call generated sub-pipelines), mark them. Hand-edit prevention.
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **Plugin image vs script bundle** — should Phase 3 wrap shared logic as a Woodpecker plugin (docker image, `image:` entry) or a Nix script/flake app (lower footprint, less duplication with the parent flake)?
|
||||
|
||||
2. **Drift-detection venue** — scheduled job in Woodpecker? Per-push hook in `oleks/ci-scripts`? Separate repo (`oleks/ci-drift-detector`)? Trade-offs: visibility, CPU budget, GitHub issue spam.
|
||||
|
||||
3. **Org secrets rollout order** — Phase 2 migration: do all tokens at once, or `registry_token` first (highest impact, lowest risk)?
|
||||
|
||||
4. **Bespoke redeclaration** — Phase 4: in manifest.yaml as a `bespoke:` section, or a separate `.bespoke.md` file in the repo?
|
||||
Reference in New Issue
Block a user