Add s390x shape-migration note to Spec/Modern-CI-Target Phase 2
+32
-30
@@ -2,11 +2,11 @@
|
||||
|
||||
1. **One source of truth** enforced (not manual): every repo's CI registration and pipeline shape live in a canonical registry, backed by a verification job that catches drift.
|
||||
|
||||
2. **Fewer pipeline shapes** → converged DSL: reduce 6+ custom YAML patterns down to ~2 (one for Nix+Attic builds, one for container multi-arch).
|
||||
2. **Fewer pipeline shapes** ← converged DSL: reduce 6+ custom YAML patterns down to ~2 (one for Nix+Attic builds, one for container multi-arch).
|
||||
|
||||
3. **No copy-paste** — secrets standardized, one token name per kind, every repo audited for registration + secret presence.
|
||||
3. **No copy-paste** ← secrets standardized, one token name per kind, every repo audited for registration + secret presence.
|
||||
|
||||
4. **Reproducible transitions** — phases are sequential; each phase closes a GitHub issue and is verifiable before the next begins.
|
||||
4. **Reproducible transitions** ← phases are sequential; each phase closes a GitHub issue and is verifiable before the next begins.
|
||||
|
||||
## Target Architecture (By Phase)
|
||||
|
||||
@@ -30,20 +30,22 @@ Baseline generator + fleet conventions established. Closed issues:
|
||||
|
||||
**Deliverables:**
|
||||
|
||||
1. **Drift detection pipeline** — scheduled job or per-push step in `oleks/ci-scripts`:
|
||||
1. **Drift detection pipeline** ← scheduled job or per-push step in `oleks/ci-scripts`:
|
||||
- Re-render all manifests locally.
|
||||
- Diff against each repo's `HEAD:.woodpecker.yaml`.
|
||||
- Open/update issues on drift (title: "ci-scripts drift detected", body: diff). Auto-close when diff clears.
|
||||
- Prevents silent hand-edits and regeneration skips.
|
||||
|
||||
2. **Org-level Woodpecker secrets** — migrate per-repo copies of `registry_token`, `attic_token`, `gitea_clone_token` to org scope. Verify every repo that references a secret actually has it (no 404s from undefined secrets at runtime).
|
||||
2. **Org-level Woodpecker secrets** ← migrate per-repo copies of `registry_token`, `attic_token`, `gitea_clone_token` to org scope. Verify every repo that references a secret actually has it (no 404s from undefined secrets at runtime).
|
||||
|
||||
3. **Token naming fix** — standardize on one token name. Currently:
|
||||
3. **Token naming fix** ← standardize on one token name. Currently:
|
||||
- `registry_token` (correct, org secret).
|
||||
- `CI_REGISTRY_TOKEN` (per-repo env) → `REGISTRY_TOKEN` (shell export, latent bug on fastuuid/onnxruntime).
|
||||
- `CI_REGISTRY_TOKEN` (per-repo env) ↦ `REGISTRY_TOKEN` (shell export, latent bug on fastuuid/onnyxruntime).
|
||||
- **Decision required:** rename all to `registry_token` org secret, or rename the secret to `ci_registry_token` and update all templates?
|
||||
|
||||
4. **CI registration audit** — verify every repo that **should** have a Woodpecker pipeline is actually registered (no 404s like `oleks/woodpecker-peek` #13). Link repos in manifest to Woodpecker API `/api/v1/repos/<owner>/<name>` and confirm at least one pipeline is present.
|
||||
4. **CI registration audit** ← verify every repo that **should** have a Woodpecker pipeline is actually registered (no 404s like `oleks/woodpecker-peek` #13). Link repos in manifest to Woodpecker API `/api/v1/repos/<owner>/<name>` and confirm at least one pipeline is present.
|
||||
|
||||
5. **s390x repo shape migration note** ← s390x repos will migrate their `.woodpecker.yaml` pipeline SHAPE for consistency with the fleet (via #8 Phase 1), but they remain **no-build**: the generator renders neutralized pipelines (`when: event: manual`, alpine echo) so tag pushes never trigger builds. The #16 fleet migration gate must not auto-enable s390x builds while hardware is unavailable. Policy enforced in generator commit bf36991.
|
||||
|
||||
**Success criteria:** All registered repos have matching drift-free `.woodpecker.yaml` files, all secrets resolve, zero undefined-secret runtime errors.
|
||||
|
||||
@@ -53,12 +55,12 @@ Baseline generator + fleet conventions established. Closed issues:
|
||||
|
||||
**Strategy:**
|
||||
|
||||
1. **OCI image bundle** — package `ci-scripts` releases as a versioned Woodpecker plugin image or standalone script bundle (`nix flake` app):
|
||||
- Encapsulates build → push → manifest-consolidate logic (buildx matrix/split/single, registry probe, digest pinning).
|
||||
1. **OCI image bundle** ← package `ci-scripts` releases as a versioned Woodpecker plugin image or standalone script bundle (`nix flake` app):
|
||||
- Encapsulates build ↦ push ↦ manifest-consolidate logic (buildx matrix/split/single, registry probe, digest pinning).
|
||||
- Per-repo pipeline becomes ~10 lines: clone, banner, `nix run git.oleks.space/oleks/ci-scripts#build-and-publish -- --config <per-repo-json>`.
|
||||
- Centralizes registry-probe + manifest logic (no per-repo duplication).
|
||||
|
||||
2. **Parity-lib convergence** — fold flat `nix` builds onto shared `oleks/parity-lib` helpers (precedent: `foldImageLayers`, shared flake-module for per-arch image builds). Reduce buildx templates to a single pluggable shape.
|
||||
2. **Parity-lib convergence** ← fold flat `nix` builds onto shared `oleks/parity-lib` helpers (precedent: `foldImageLayers`, shared flake-module for per-arch image builds). Reduce buildx templates to a single pluggable shape.
|
||||
|
||||
3. **Target result:** 3 pipeline shapes → 1 parameterized shape for Nix+container builds. Bespoke repos (quoted in "Not Covered") remain exempt but documented.
|
||||
|
||||
@@ -69,39 +71,39 @@ Baseline generator + fleet conventions established. Closed issues:
|
||||
**Goal:** Migrate or explicitly declare permanent-bespoke repos.
|
||||
|
||||
**Case-by-case audit:**
|
||||
- `ii-researcher`, `commonground-legacy` (amd64-only Dockerfile) — add arm64 legs via `buildx` plugin or declare amd64-only in manifest + skip arm64 CI entirely?
|
||||
- `fastuuid-s390x`, `onnxruntime-s390x` (token remap bug) — template them into thin-publish once token bug is understood?
|
||||
- `scikit-learn-s390x`, `scipy-s390x` (heredoc Redis) — normalize Redis setup to direct export and fold into thin-publish?
|
||||
- `builder-arbitrage` (cosign + digest-pinned) — keep as bespoke (true multi-step orchestration), document rationale.
|
||||
- `nixos-ci`, `ii-agent`, `terminal-agent`, `mermaid-gpu-dashboard` — stay bespoke (architectural uniqueness), link to justification docs.
|
||||
- `iia-researcher`, `commonground-legacy` (amd64-only Dockerfile) ← add arm64 legs via `buildx` plugin or declare amd64-only in manifest + skip arm64 CI entirely?
|
||||
- `fastuuid-s390x`, `onnyxruntime-s390x` (token remap bug) ← template them into thin-publish once token bug is understood?
|
||||
- `scikit-learn-s390x`, `scipy-s390x` (heredoc Redis) ← normalize Redis setup to direct export and fold into thin-publish?
|
||||
- `builder-arbitrage` (cosign + digest-pinned) ← keep as bespoke (true multi-step orchestration), document rationale.
|
||||
- `nixos-ci`, `iia-agent`, `terminal-agent`, `mermaid-gpu-dashboard` ← stay bespoke (architectural uniqueness), link to justification docs.
|
||||
|
||||
**Result:** A final "permanent bespoke" list in the manifest with reasons for each (and links to the issues/docs justifying them).
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- **No move off Woodpecker** — this is a Woodpecker optimization, not a migration to GitHub Actions / other CI.
|
||||
- **No forcing bespoke toolchains prematurely** — `nix2container`, `cosign`, moby-buildkit stay in repos that own them; don't fold them into shared templates until stable.
|
||||
- **No move off Woodpecker** ← this is a Woodpecker optimization, not a migration to GitHub Actions / other CI.
|
||||
- **No forcing bespoke toolchains prematurely** ← `nix2container`, `cosign`, moby-buildkit stay in repos that own them; don't fold them into shared templates until stable.
|
||||
|
||||
## Invariants (Must Survive Transition)
|
||||
|
||||
- **Arch banner** — every step's first line `echo "▸ arch=$(uname -m)"`.
|
||||
- **No `|| true`** — explicit exit-code handling everywhere.
|
||||
- **Arch-only labels** — `labels: {arch: amd64|arm64|s390x-native}` is the sole node-pinning knob; no `nodeSelector` or `platform` aliases.
|
||||
- **Parse-time `${}`** — braced variables are parse-phase (Woodpecker substitution); shell uses `$VAR` or `$$`. Discipline enforced by linter or guide.
|
||||
- **Manifest-on-partial-success** — multi-arch workflows must not skip manifest consolidation if one arch fails (registry probe gates it).
|
||||
- **GENERATED headers** — wherever `.woodpecker.yaml` files are still rendered (even in Phase 3, if thin pipelines call generated sub-pipelines), mark them. Hand-edit prevention.
|
||||
- **Arch banner** ← every step's first line `echo "▸ arch=$(uname -m)"`.
|
||||
- **No `|| true`** ← explicit exit-code handling everywhere.
|
||||
- **Arch-only labels** ← `labels: {arch: amd64|arm64|s390x-native}` is the sole node-pinning knob; no `nodeSelector` or `platform` aliases.
|
||||
- **Parse-time `${}`** ← braced variables are parse-phase (Woodpecker substitution); shell uses bare `$VAR` or `$${` (escaped brace). Discipline enforced by linter or guide.
|
||||
- **Manifest-on-partial-success** ← multi-arch workflows must not skip manifest consolidation if one arch fails (registry probe gates it).
|
||||
- **GENERATED headers** ← wherever `.woodpecker.yaml` files are still rendered (even in Phase 3, if thin pipelines call generated sub-pipelines), mark them.
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **Plugin image vs script bundle** — should Phase 3 wrap shared logic as a Woodpecker plugin (docker image, `image:` entry) or a Nix script/flake app (lower footprint, less duplication with the parent flake)?
|
||||
1. **Plugin image vs script bundle** ← should Phase 3 wrap shared logic as a Woodpecker plugin (docker image, `image:` entry) or a Nix script/flake app (lower footprint, less duplication with the parent flake)?
|
||||
|
||||
2. **Drift-detection venue** — scheduled job in Woodpecker? Per-push hook in `oleks/ci-scripts`? Separate repo (`oleks/ci-drift-detector`)? Trade-offs: visibility, CPU budget, GitHub issue spam.
|
||||
2. **Drift-detection venue** ← scheduled job in Woodpecker? Per-push hook in `oleks/ci-scripts`? Separate repo (oleks/ci-drift-detector)? Trade-offs: visibility, CPU budget, GitHub issue spam.
|
||||
|
||||
3. **Org secrets rollout order** — Phase 2 migration: do all tokens at once, or `registry_token` first (highest impact, lowest risk)?
|
||||
3. **Org secrets rollout order** ← Phase 2 migration: do all tokens at once, or `registry_token` first (highest impact, lowest risk)?
|
||||
|
||||
4. **Bespoke redeclaration** — Phase 4: in `manifest.yaml` as a `bespoke:` section, or a separate `.bespoke.md` file in the repo?
|
||||
4. **Bespoke redeclaration** ← Phase 4: in `manifest.yaml` as a `bespoke:` section, or a separate `.bespoke.md` file in the repo?
|
||||
|
||||
## See also
|
||||
|
||||
- [Spec/Current CI Ecosystem](wiki/Spec/Current-CI-Ecosystem) — Current state of the Woodpecker CI fleet
|
||||
- [Spec/Fleet Publish Plugin System](wiki/Spec/Fleet-Publish-Plugin-System) — Plugin architecture & thin pipeline contract for Phase 2+3
|
||||
- [Spec/Current CI Ecosystem](wiki/Spec/Current-CI-Ecosystem) ← Current state of the Woodpecker CI fleet
|
||||
- [Spec/Fleet Publish Plugin System](wiki/Spec/Fleet-Publish-Plugin-System) ← Plugin architecture & thin pipeline contract for Phase 2+3
|
||||
|
||||
Reference in New Issue
Block a user