106 lines
4.9 KiB
Markdown
106 lines
4.9 KiB
Markdown
# ci-scripts
|
|
|
|
Shared Woodpecker CI templates, plugins, and centralized pipeline conventions
|
|
for the Oleks ecosystem. This repository hosts reusable CI components (clone
|
|
steps, build templates, arch-specific runners) and tracks the cross-repo CI
|
|
optimization effort to reduce duplication across ~50 projects in ~/projects.
|
|
|
|
## pypi-wheel-attic template (oleks/ci-scripts#1)
|
|
|
|
`manifest.yaml` + `generate.py` render `.woodpecker.yaml` for the fleet of
|
|
s390x cross-build repos that share the "single build-and-publish step, inline
|
|
`attic watch-store` accelerator, tag-triggered PyPI wheel publish" shape.
|
|
Generated files carry a `GENERATED by oleks/ci-scripts — do not hand-edit`
|
|
header; edit `manifest.yaml` and re-run `python3 generate.py` instead.
|
|
|
|
`geesefs-s390x`, `sentry-cli-s390x`, `attic-client-s390x`, `angie-arm64`,
|
|
`devpi-arm64`, and `mempalace` are bespoke pipelines (OCI image publish,
|
|
binary builds, multi-file workflows) and are out of scope for either
|
|
template.
|
|
|
|
Verify with:
|
|
|
|
```sh
|
|
python3 generate.py --check # fails if any rendered file is stale
|
|
woodpecker-cli lint <repo>/.woodpecker.yaml
|
|
```
|
|
|
|
## thin-parity-lib-publish template (oleks/ci-scripts#10)
|
|
|
|
The same `generate.py` also renders a second, genuinely different shape via
|
|
`render_thin()` + the `thin_repos:` list in `manifest.yaml`: no inline attic
|
|
accelerator, publish is a pure `nix run .#publish[-s390x]` front door,
|
|
optionally gated by a `pipeline-doctor --strict` step first. Covers
|
|
`asyncpg`, `cryptography`, `lightningcss`, `nextjs-swc`, `rollup`-s390x.
|
|
|
|
Real per-repo variance parameterized (not papered over): `doctor` gate
|
|
presence, `token_direct` (`REGISTRY_TOKEN` straight from secret vs.
|
|
`CI_REGISTRY_TOKEN` + explicit remap), `publish_style` (`PUBLISH=1 nix run
|
|
.#publish` vs. `nix run .#publish-s390x -- --publish`), `tags_full` (full
|
|
clone with tags vs. shallow), and `dead_secrets` (some repos declare
|
|
`ATTIC_TOKEN`/`REDIS_PASSWORD` but never reference them — preserved as-is,
|
|
not this issue's scope to clean up).
|
|
|
|
Excluded after re-audit, not forced:
|
|
|
|
- `fastuuid-s390x` — declares `CI_REGISTRY_TOKEN` but has **no** `export
|
|
REGISTRY_TOKEN=...` remap before `PUBLISH=1 nix run .#publish`
|
|
(`cryptography-s390x`, the only other `CI_REGISTRY_TOKEN` user, does have
|
|
the remap). This looks like a real latent bug — publish would run with
|
|
`REGISTRY_TOKEN` unset. Flagged, not silently fixed via templating.
|
|
- `onnxruntime-s390x` — same `CI_REGISTRY_TOKEN`-without-remap pattern as
|
|
`fastuuid-s390x`, and it's actually attic+redis shaped (single step) like
|
|
the `pypi-wheel-attic` family above, not this one. Left hand-maintained.
|
|
- `scikit-learn-s390x`, `scipy-s390x` — attic+redis shaped like the
|
|
`pypi-wheel-attic` family, but their Redis setup writes exports to a file
|
|
via a heredoc that's never sourced elsewhere in the step, unlike
|
|
`tiktoken-s390x`'s direct-export style. Whether that heredoc form actually
|
|
does anything meaningful isn't confirmed; normalizing it to direct exports
|
|
would be a real behavior change, not a safe refactor. Left hand-maintained.
|
|
|
|
## Node-pinning convention (oleks/ci-scripts#5)
|
|
|
|
Woodpecker agents advertise `custom_labels` of `{arch, worker}` (e.g.
|
|
`arch: arm64, worker: kotkan`). Pipelines pin work to an architecture with the
|
|
top-level pipeline `labels:` key, matched against the agent's `arch` label:
|
|
|
|
```yaml
|
|
labels:
|
|
arch: amd64 # or arm64
|
|
```
|
|
|
|
This is the one fleet-wide convention. Two other mechanisms that used to
|
|
coexist are retired:
|
|
|
|
- `labels: {platform: mermaid}` — an older alias for `arch: amd64` from before
|
|
there were multiple amd64 agents. Migrated to `arch: amd64`.
|
|
- `backend_options.kubernetes.nodeSelector: {provider: digitalocean}` — a
|
|
literal Kubernetes node-label selector (a different mechanism than
|
|
Woodpecker agent labels entirely), left over from a single-provider
|
|
cluster. Dropped: agent-level `labels:` already routes the pipeline to the
|
|
right architecture, so pinning to a specific cloud provider's nodes on top
|
|
of that is redundant and brittle if node labels change.
|
|
|
|
`backend_options.kubernetes.nodeSelector` with `kubernetes.io/arch` or
|
|
`kubernetes.io/hostname` is still valid where a pipeline needs an actual k8s
|
|
node-level constraint (e.g. `sentry-cli-s390x` pinning to a specific host) —
|
|
that's a different, legitimate use of the mechanism and untouched here.
|
|
|
|
## `|| true` ban (oleks/ci-scripts#6)
|
|
|
|
Per standing convention, `|| true` is banned repo-wide — it silently swallows
|
|
any failure, not just the one it was written for. The fleet's `attic
|
|
watch-store` teardown (`kill $ATTIC_PID && wait $ATTIC_PID || true`) is
|
|
converted to explicit exit-code handling that still tolerates the benign case
|
|
(the backgrounded `attic watch-store` process exiting non-zero after being
|
|
killed) without masking anything else:
|
|
|
|
```yaml
|
|
- if ! kill $ATTIC_PID 2>/dev/null; then
|
|
echo "attic watch-store already exited"
|
|
fi
|
|
- if ! wait $ATTIC_PID 2>/dev/null; then
|
|
echo "attic watch-store exited non-zero after kill (expected)"
|
|
fi
|
|
```
|