diff --git a/Spec%2FModern-CI-Target.-.md b/Spec%2FModern-CI-Target.-.md index e2bbad1..3a5ce60 100644 --- a/Spec%2FModern-CI-Target.-.md +++ b/Spec%2FModern-CI-Target.-.md @@ -2,11 +2,11 @@ 1. **One source of truth** enforced (not manual): every repo's CI registration and pipeline shape live in a canonical registry, backed by a verification job that catches drift. -2. **Fewer pipeline shapes** → converged DSL: reduce 6+ custom YAML patterns down to ~2 (one for Nix+Attic builds, one for container multi-arch). +2. **Fewer pipeline shapes** ← converged DSL: reduce 6+ custom YAML patterns down to ~2 (one for Nix+Attic builds, one for container multi-arch). -3. **No copy-paste** — secrets standardized, one token name per kind, every repo audited for registration + secret presence. +3. **No copy-paste** ← secrets standardized, one token name per kind, every repo audited for registration + secret presence. -4. **Reproducible transitions** — phases are sequential; each phase closes a GitHub issue and is verifiable before the next begins. +4. **Reproducible transitions** ← phases are sequential; each phase closes a GitHub issue and is verifiable before the next begins. ## Target Architecture (By Phase) @@ -30,20 +30,22 @@ Baseline generator + fleet conventions established. Closed issues: **Deliverables:** -1. **Drift detection pipeline** — scheduled job or per-push step in `oleks/ci-scripts`: +1. **Drift detection pipeline** ← scheduled job or per-push step in `oleks/ci-scripts`: - Re-render all manifests locally. - Diff against each repo's `HEAD:.woodpecker.yaml`. - Open/update issues on drift (title: "ci-scripts drift detected", body: diff). Auto-close when diff clears. - Prevents silent hand-edits and regeneration skips. -2. **Org-level Woodpecker secrets** — migrate per-repo copies of `registry_token`, `attic_token`, `gitea_clone_token` to org scope. Verify every repo that references a secret actually has it (no 404s from undefined secrets at runtime). +2. **Org-level Woodpecker secrets** ← migrate per-repo copies of `registry_token`, `attic_token`, `gitea_clone_token` to org scope. Verify every repo that references a secret actually has it (no 404s from undefined secrets at runtime). -3. **Token naming fix** — standardize on one token name. Currently: +3. **Token naming fix** ← standardize on one token name. Currently: - `registry_token` (correct, org secret). - - `CI_REGISTRY_TOKEN` (per-repo env) → `REGISTRY_TOKEN` (shell export, latent bug on fastuuid/onnxruntime). + - `CI_REGISTRY_TOKEN` (per-repo env) ↦ `REGISTRY_TOKEN` (shell export, latent bug on fastuuid/onnyxruntime). - **Decision required:** rename all to `registry_token` org secret, or rename the secret to `ci_registry_token` and update all templates? -4. **CI registration audit** — verify every repo that **should** have a Woodpecker pipeline is actually registered (no 404s like `oleks/woodpecker-peek` #13). Link repos in manifest to Woodpecker API `/api/v1/repos//` and confirm at least one pipeline is present. +4. **CI registration audit** ← verify every repo that **should** have a Woodpecker pipeline is actually registered (no 404s like `oleks/woodpecker-peek` #13). Link repos in manifest to Woodpecker API `/api/v1/repos//` and confirm at least one pipeline is present. + +5. **s390x repo shape migration note** ← s390x repos will migrate their `.woodpecker.yaml` pipeline SHAPE for consistency with the fleet (via #8 Phase 1), but they remain **no-build**: the generator renders neutralized pipelines (`when: event: manual`, alpine echo) so tag pushes never trigger builds. The #16 fleet migration gate must not auto-enable s390x builds while hardware is unavailable. Policy enforced in generator commit bf36991. **Success criteria:** All registered repos have matching drift-free `.woodpecker.yaml` files, all secrets resolve, zero undefined-secret runtime errors. @@ -53,12 +55,12 @@ Baseline generator + fleet conventions established. Closed issues: **Strategy:** -1. **OCI image bundle** — package `ci-scripts` releases as a versioned Woodpecker plugin image or standalone script bundle (`nix flake` app): - - Encapsulates build → push → manifest-consolidate logic (buildx matrix/split/single, registry probe, digest pinning). +1. **OCI image bundle** ← package `ci-scripts` releases as a versioned Woodpecker plugin image or standalone script bundle (`nix flake` app): + - Encapsulates build ↦ push ↦ manifest-consolidate logic (buildx matrix/split/single, registry probe, digest pinning). - Per-repo pipeline becomes ~10 lines: clone, banner, `nix run git.oleks.space/oleks/ci-scripts#build-and-publish -- --config `. - Centralizes registry-probe + manifest logic (no per-repo duplication). -2. **Parity-lib convergence** — fold flat `nix` builds onto shared `oleks/parity-lib` helpers (precedent: `foldImageLayers`, shared flake-module for per-arch image builds). Reduce buildx templates to a single pluggable shape. +2. **Parity-lib convergence** ← fold flat `nix` builds onto shared `oleks/parity-lib` helpers (precedent: `foldImageLayers`, shared flake-module for per-arch image builds). Reduce buildx templates to a single pluggable shape. 3. **Target result:** 3 pipeline shapes → 1 parameterized shape for Nix+container builds. Bespoke repos (quoted in "Not Covered") remain exempt but documented. @@ -69,39 +71,39 @@ Baseline generator + fleet conventions established. Closed issues: **Goal:** Migrate or explicitly declare permanent-bespoke repos. **Case-by-case audit:** -- `ii-researcher`, `commonground-legacy` (amd64-only Dockerfile) — add arm64 legs via `buildx` plugin or declare amd64-only in manifest + skip arm64 CI entirely? -- `fastuuid-s390x`, `onnxruntime-s390x` (token remap bug) — template them into thin-publish once token bug is understood? -- `scikit-learn-s390x`, `scipy-s390x` (heredoc Redis) — normalize Redis setup to direct export and fold into thin-publish? -- `builder-arbitrage` (cosign + digest-pinned) — keep as bespoke (true multi-step orchestration), document rationale. -- `nixos-ci`, `ii-agent`, `terminal-agent`, `mermaid-gpu-dashboard` — stay bespoke (architectural uniqueness), link to justification docs. +- `iia-researcher`, `commonground-legacy` (amd64-only Dockerfile) ← add arm64 legs via `buildx` plugin or declare amd64-only in manifest + skip arm64 CI entirely? +- `fastuuid-s390x`, `onnyxruntime-s390x` (token remap bug) ← template them into thin-publish once token bug is understood? +- `scikit-learn-s390x`, `scipy-s390x` (heredoc Redis) ← normalize Redis setup to direct export and fold into thin-publish? +- `builder-arbitrage` (cosign + digest-pinned) ← keep as bespoke (true multi-step orchestration), document rationale. +- `nixos-ci`, `iia-agent`, `terminal-agent`, `mermaid-gpu-dashboard` ← stay bespoke (architectural uniqueness), link to justification docs. **Result:** A final "permanent bespoke" list in the manifest with reasons for each (and links to the issues/docs justifying them). ## Non-Goals -- **No move off Woodpecker** — this is a Woodpecker optimization, not a migration to GitHub Actions / other CI. -- **No forcing bespoke toolchains prematurely** — `nix2container`, `cosign`, moby-buildkit stay in repos that own them; don't fold them into shared templates until stable. +- **No move off Woodpecker** ← this is a Woodpecker optimization, not a migration to GitHub Actions / other CI. +- **No forcing bespoke toolchains prematurely** ← `nix2container`, `cosign`, moby-buildkit stay in repos that own them; don't fold them into shared templates until stable. ## Invariants (Must Survive Transition) -- **Arch banner** — every step's first line `echo "▸ arch=$(uname -m)"`. -- **No `|| true`** — explicit exit-code handling everywhere. -- **Arch-only labels** — `labels: {arch: amd64|arm64|s390x-native}` is the sole node-pinning knob; no `nodeSelector` or `platform` aliases. -- **Parse-time `${}`** — braced variables are parse-phase (Woodpecker substitution); shell uses `$VAR` or `$$`. Discipline enforced by linter or guide. -- **Manifest-on-partial-success** — multi-arch workflows must not skip manifest consolidation if one arch fails (registry probe gates it). -- **GENERATED headers** — wherever `.woodpecker.yaml` files are still rendered (even in Phase 3, if thin pipelines call generated sub-pipelines), mark them. Hand-edit prevention. +- **Arch banner** ← every step's first line `echo "▸ arch=$(uname -m)"`. +- **No `|| true`** ← explicit exit-code handling everywhere. +- **Arch-only labels** ← `labels: {arch: amd64|arm64|s390x-native}` is the sole node-pinning knob; no `nodeSelector` or `platform` aliases. +- **Parse-time `${}`** ← braced variables are parse-phase (Woodpecker substitution); shell uses bare `$VAR` or `$${` (escaped brace). Discipline enforced by linter or guide. +- **Manifest-on-partial-success** ← multi-arch workflows must not skip manifest consolidation if one arch fails (registry probe gates it). +- **GENERATED headers** ← wherever `.woodpecker.yaml` files are still rendered (even in Phase 3, if thin pipelines call generated sub-pipelines), mark them. ## Open Questions -1. **Plugin image vs script bundle** — should Phase 3 wrap shared logic as a Woodpecker plugin (docker image, `image:` entry) or a Nix script/flake app (lower footprint, less duplication with the parent flake)? +1. **Plugin image vs script bundle** ← should Phase 3 wrap shared logic as a Woodpecker plugin (docker image, `image:` entry) or a Nix script/flake app (lower footprint, less duplication with the parent flake)? -2. **Drift-detection venue** — scheduled job in Woodpecker? Per-push hook in `oleks/ci-scripts`? Separate repo (`oleks/ci-drift-detector`)? Trade-offs: visibility, CPU budget, GitHub issue spam. +2. **Drift-detection venue** ← scheduled job in Woodpecker? Per-push hook in `oleks/ci-scripts`? Separate repo (oleks/ci-drift-detector)? Trade-offs: visibility, CPU budget, GitHub issue spam. -3. **Org secrets rollout order** — Phase 2 migration: do all tokens at once, or `registry_token` first (highest impact, lowest risk)? +3. **Org secrets rollout order** ← Phase 2 migration: do all tokens at once, or `registry_token` first (highest impact, lowest risk)? -4. **Bespoke redeclaration** — Phase 4: in `manifest.yaml` as a `bespoke:` section, or a separate `.bespoke.md` file in the repo? +4. **Bespoke redeclaration** ← Phase 4: in `manifest.yaml` as a `bespoke:` section, or a separate `.bespoke.md` file in the repo? ## See also -- [Spec/Current CI Ecosystem](wiki/Spec/Current-CI-Ecosystem) — Current state of the Woodpecker CI fleet -- [Spec/Fleet Publish Plugin System](wiki/Spec/Fleet-Publish-Plugin-System) — Plugin architecture & thin pipeline contract for Phase 2+3 +- [Spec/Current CI Ecosystem](wiki/Spec/Current-CI-Ecosystem) ← Current state of the Woodpecker CI fleet +- [Spec/Fleet Publish Plugin System](wiki/Spec/Fleet-Publish-Plugin-System) ← Plugin architecture & thin pipeline contract for Phase 2+3