diff --git a/Spec%2FModern-CI-Target.md b/Spec%2FModern-CI-Target.md new file mode 100644 index 0000000..44656c4 --- /dev/null +++ b/Spec%2FModern-CI-Target.md @@ -0,0 +1,102 @@ +## Goals + +1. **One source of truth** enforced (not manual): every repo's CI registration and pipeline shape live in a canonical registry, backed by a verification job that catches drift. + +2. **Fewer pipeline shapes** → converged DSL: reduce 6+ custom YAML patterns down to ~2 (one for Nix+Attic builds, one for container multi-arch). + +3. **No copy-paste** — secrets standardized, one token name per kind, every repo audited for registration + secret presence. + +4. **Reproducible transitions** — phases are sequential; each phase closes a GitHub issue and is verifiable before the next begins. + +## Target Architecture (By Phase) + +### Phase 1: EPIC #8 (DONE 2026-07-04) + +Baseline generator + fleet conventions established. Closed issues: + +- [#1](https://git.oleks.space/oleks/ci-scripts/issues/1) Consolidate s390x/arm64 cross-build fleet into one shared template +- [#2](https://git.oleks.space/oleks/ci-scripts/issues/2) Extract shared buildx multi-arch quartet (amd64/arm64/manifest/chart) +- [#3](https://git.oleks.space/oleks/ci-scripts/issues/3) Shared clone step definition +- [#4](https://git.oleks.space/oleks/ci-scripts/issues/4) Centralize registry/host constants +- [#5](https://git.oleks.space/oleks/ci-scripts/issues/5) Unify node-pinning label keys +- [#6](https://git.oleks.space/oleks/ci-scripts/issues/6) Audit and remove copied `|| true` failure-masking +- [#7](https://git.oleks.space/oleks/ci-scripts/issues/7) Split monolithic step scripts + banner gaps +- [#9](https://git.oleks.space/oleks/ci-scripts/issues/9) Manifest step must not be skipped on partial arch success +- [#10](https://git.oleks.space/oleks/ci-scripts/issues/10) Second fleet template for the parity-lib publish family + +**Artifacts:** `manifest.yaml` + `generate.py` + `render.py`, 15 templated repos, arch-banner + no `|| true` + node-pinning conventions. + +### Phase 2: Enforcement & Hygiene + +**Deliverables:** + +1. **Drift detection pipeline** — scheduled job or per-push step in `oleks/ci-scripts`: + - Re-render all manifests locally. + - Diff against each repo's `HEAD:.woodpecker.yaml`. + - Open/update issues on drift (title: "ci-scripts drift detected", body: diff). Auto-close when diff clears. + - Prevents silent hand-edits and regeneration skips. + +2. **Org-level Woodpecker secrets** — migrate per-repo copies of `registry_token`, `attic_token`, `gitea_clone_token` to org scope. Verify every repo that references a secret actually has it (no 404s from undefined secrets at runtime). + +3. **Token naming fix** — standardize on one token name. Currently: + - `registry_token` (correct, org secret). + - `CI_REGISTRY_TOKEN` (per-repo env) → `REGISTRY_TOKEN` (shell export, latent bug on fastuuid/onnxruntime). + - **Decision required:** rename all to `registry_token` org secret, or rename the secret to `ci_registry_token` and update all templates? + +4. **CI registration audit** — verify every repo that **should** have a Woodpecker pipeline is actually registered (no 404s like `oleks/woodpecker-peek` #13). Link repos in manifest to Woodpecker API `/api/v1/repos//` and confirm at least one pipeline is present. + +**Success criteria:** All registered repos have matching drift-free `.woodpecker.yaml` files, all secrets resolve, zero undefined-secret runtime errors. + +### Phase 3: Shape Convergence + +**Goal:** Replace generated YAML bulk with thin parameterized pipelines calling versioned shared code. + +**Strategy:** + +1. **OCI image bundle** — package `ci-scripts` releases as a versioned Woodpecker plugin image or standalone script bundle (`nix flake` app): + - Encapsulates build → push → manifest-consolidate logic (buildx matrix/split/single, registry probe, digest pinning). + - Per-repo pipeline becomes ~10 lines: clone, banner, `nix run git.oleks.space/oleks/ci-scripts#build-and-publish -- --config `. + - Centralizes registry-probe + manifest logic (no per-repo duplication). + +2. **Parity-lib convergence** — fold flat `nix` builds onto shared `oleks/parity-lib` helpers (precedent: `foldImageLayers`, shared flake-module for per-arch image builds). Reduce buildx templates to a single pluggable shape. + +3. **Target result:** 3 pipeline shapes → 1 parameterized shape for Nix+container builds. Bespoke repos (quoted in "Not Covered") remain exempt but documented. + +**Success criteria:** Every templated repo's `.woodpecker.yaml` is <100 lines, most logic is version-pinned calls to published scripts/images, generator still verifies correctness but doesn't emit bulk YAML. + +### Phase 4: Fold Bespoke Tail + +**Goal:** Migrate or explicitly declare permanent-bespoke repos. + +**Case-by-case audit:** +- `ii-researcher`, `commonground-legacy` (amd64-only Dockerfile) — add arm64 legs via `buildx` plugin or declare amd64-only in manifest + skip arm64 CI entirely? +- `fastuuid-s390x`, `onnxruntime-s390x` (token remap bug) — template them into thin-publish once token bug is understood? +- `scikit-learn-s390x`, `scipy-s390x` (heredoc Redis) — normalize Redis setup to direct export and fold into thin-publish? +- `builder-arbitrage` (cosign + digest-pinned) — keep as bespoke (true multi-step orchestration), document rationale. +- `nixos-ci`, `ii-agent`, `terminal-agent`, `mermaid-gpu-dashboard` — stay bespoke (architectural uniqueness), link to justification docs. + +**Result:** A final "permanent bespoke" list in the manifest with reasons for each (and links to the issues/docs justifying them). + +## Non-Goals + +- **No move off Woodpecker** — this is a Woodpecker optimization, not a migration to GitHub Actions / other CI. +- **No forcing bespoke toolchains prematurely** — `nix2container`, `cosign`, moby-buildkit stay in repos that own them; don't fold them into shared templates until stable. + +## Invariants (Must Survive Transition) + +- **Arch banner** — every step's first line `echo "▸ arch=$(uname -m)"`. +- **No `|| true`** — explicit exit-code handling everywhere. +- **Arch-only labels** — `labels: {arch: amd64|arm64|s390x-native}` is the sole node-pinning knob; no `nodeSelector` or `platform` aliases. +- **Parse-time `${}`** — braced variables are parse-phase (Woodpecker substitution); shell uses `$VAR` or `$$`. Discipline enforced by linter or guide. +- **Manifest-on-partial-success** — multi-arch workflows must not skip manifest consolidation if one arch fails (registry probe gates it). +- **GENERATED headers** — wherever `.woodpecker.yaml` files are still rendered (even in Phase 3, if thin pipelines call generated sub-pipelines), mark them. Hand-edit prevention. + +## Open Questions + +1. **Plugin image vs script bundle** — should Phase 3 wrap shared logic as a Woodpecker plugin (docker image, `image:` entry) or a Nix script/flake app (lower footprint, less duplication with the parent flake)? + +2. **Drift-detection venue** — scheduled job in Woodpecker? Per-push hook in `oleks/ci-scripts`? Separate repo (`oleks/ci-drift-detector`)? Trade-offs: visibility, CPU budget, GitHub issue spam. + +3. **Org secrets rollout order** — Phase 2 migration: do all tokens at once, or `registry_token` first (highest impact, lowest risk)? + +4. **Bespoke redeclaration** — Phase 4: in manifest.yaml as a `bespoke:` section, or a separate `.bespoke.md` file in the repo?