This commit is contained in:
@@ -1,3 +1,70 @@
|
||||
# ci-scripts
|
||||
|
||||
Shared Woodpecker CI templates, plugins, and centralized pipeline conventions for the Oleks ecosystem. This repository hosts reusable CI components (clone steps, build templates, arch-specific runners) and tracks the cross-repo CI optimization effort to reduce duplication across ~50 projects in ~/projects.
|
||||
Shared Woodpecker CI templates, plugins, and centralized pipeline conventions for the Oleks ecosystem. This repository hosts reusable CI components (clone steps, build templates, arch-specific runners) and tracks the cross-repo CI optimization effort to reduce duplication across ~50 projects in ~/projects.
|
||||
|
||||
## pypi-wheel-attic template (oleks/ci-scripts#1)
|
||||
|
||||
`manifest.yaml` + `generate.py` render `.woodpecker.yaml` for the fleet of
|
||||
s390x cross-build repos that share the "single build-and-publish step, inline
|
||||
`attic watch-store` accelerator, tag-triggered PyPI wheel publish" shape.
|
||||
Generated files carry a `GENERATED by oleks/ci-scripts — do not hand-edit`
|
||||
header; edit `manifest.yaml` and re-run `python3 generate.py` instead.
|
||||
|
||||
Not every s390x/arm64 repo fits this template — some (`asyncpg`,
|
||||
`cryptography`, `fastuuid`, `lightningcss`, `nextjs-swc`, `rollup`,
|
||||
`onnxruntime`, `scikit-learn`, `scipy`) run a genuinely different "thin
|
||||
parity-lib publish" pipeline with per-repo Redis sccache wiring and a
|
||||
pipeline-doctor gate — those differences are load-bearing, not copy-paste
|
||||
drift, and were deliberately left un-templated (see the scope note atop
|
||||
`manifest.yaml`). `geesefs-s390x`, `sentry-cli-s390x`, `attic-client-s390x`,
|
||||
`angie-arm64`, `devpi-arm64`, and `mempalace` are bespoke pipelines (OCI image
|
||||
publish, binary builds, multi-file workflows) and are also out of scope.
|
||||
|
||||
Verify with:
|
||||
|
||||
```
|
||||
python3 generate.py --check # fails if any rendered file is stale
|
||||
woodpecker-cli lint <repo>/.woodpecker.yaml
|
||||
```
|
||||
|
||||
## Node-pinning convention (oleks/ci-scripts#5)
|
||||
|
||||
Woodpecker agents advertise `custom_labels` of `{arch, worker}` (e.g.
|
||||
`arch: arm64, worker: kotkan`). Pipelines pin work to an architecture with the
|
||||
top-level pipeline `labels:` key, matched against the agent's `arch` label:
|
||||
|
||||
```yaml
|
||||
labels:
|
||||
arch: amd64 # or arm64
|
||||
```
|
||||
|
||||
This is the one fleet-wide convention. Two other mechanisms that used to
|
||||
coexist are retired:
|
||||
|
||||
- `labels: {platform: mermaid}` — an older alias for `arch: amd64` from before
|
||||
there were multiple amd64 agents. Migrated to `arch: amd64`.
|
||||
- `backend_options.kubernetes.nodeSelector: {provider: digitalocean}` — a
|
||||
literal Kubernetes node-label selector (a different mechanism than
|
||||
Woodpecker agent labels entirely), left over from a single-provider
|
||||
cluster. Dropped: agent-level `labels:` already routes the pipeline to the
|
||||
right architecture, so pinning to a specific cloud provider's nodes on top
|
||||
of that is redundant and brittle if node labels change.
|
||||
|
||||
`backend_options.kubernetes.nodeSelector` with `kubernetes.io/arch` or
|
||||
`kubernetes.io/hostname` is still valid where a pipeline needs an actual k8s
|
||||
node-level constraint (e.g. `sentry-cli-s390x` pinning to a specific host) —
|
||||
that's a different, legitimate use of the mechanism and untouched here.
|
||||
|
||||
## `|| true` ban (oleks/ci-scripts#6)
|
||||
|
||||
Per standing convention, `|| true` is banned repo-wide — it silently swallows
|
||||
any failure, not just the one it was written for. The fleet's `attic
|
||||
watch-store` teardown (`kill $ATTIC_PID && wait $ATTIC_PID || true`) is
|
||||
converted to explicit exit-code handling that still tolerates the benign case
|
||||
(the backgrounded `attic watch-store` process exiting non-zero after being
|
||||
killed) without masking anything else:
|
||||
|
||||
```yaml
|
||||
- if ! kill $ATTIC_PID 2>/dev/null; then echo "attic watch-store already exited"; fi
|
||||
- if ! wait $ATTIC_PID 2>/dev/null; then echo "attic watch-store exited non-zero after kill (expected)"; fi
|
||||
```
|
||||
|
||||
+146
@@ -0,0 +1,146 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Render .woodpecker.yaml for the pypi-wheel-attic fleet family.
|
||||
|
||||
GENERATED files carry a "do not hand-edit" header; edit manifest.yaml instead
|
||||
and re-run this script. See manifest.yaml for scope notes (oleks/ci-scripts#1).
|
||||
"""
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
import yaml
|
||||
|
||||
HERE = pathlib.Path(__file__).parent
|
||||
BUILDING = HERE.parent / "building"
|
||||
|
||||
HEADER = """# GENERATED by oleks/ci-scripts — do not hand-edit.
|
||||
# Source: oleks/ci-scripts manifest.yaml + generate.py (oleks/ci-scripts#1).
|
||||
# Re-render with: python3 generate.py
|
||||
"""
|
||||
|
||||
|
||||
def render(repo: dict) -> str:
|
||||
arch = repo["arch"]
|
||||
mem_req = repo["memory_request"]
|
||||
mem_lim = repo["memory_limit"]
|
||||
cores = repo.get("cores")
|
||||
max_jobs = repo.get("max_jobs", False)
|
||||
attic = repo.get("attic", True)
|
||||
redis_cache = repo.get("redis_cache", False)
|
||||
|
||||
nix_conf_lines = []
|
||||
if max_jobs:
|
||||
nix_conf_lines.append(' - echo "max-jobs = 1" >> /etc/nix/nix.conf')
|
||||
if cores:
|
||||
nix_conf_lines.append(f' - echo "cores = {cores}" >> /etc/nix/nix.conf')
|
||||
nix_conf_lines.append(' - echo "sandbox = false" >> /etc/nix/nix.conf')
|
||||
nix_conf_block = "\n".join(nix_conf_lines)
|
||||
|
||||
env_lines = [
|
||||
" GITEA_CLONE_TOKEN:",
|
||||
" from_secret: gitea_clone_token",
|
||||
" REGISTRY_TOKEN:",
|
||||
" from_secret: registry_token",
|
||||
]
|
||||
if attic:
|
||||
env_lines += [
|
||||
" ATTIC_TOKEN:",
|
||||
" from_secret: attic_token",
|
||||
]
|
||||
if redis_cache:
|
||||
env_lines += [
|
||||
" REDIS_PASSWORD:",
|
||||
" from_secret: redis_password",
|
||||
]
|
||||
env_block = "\n".join(env_lines)
|
||||
|
||||
redis_block = ""
|
||||
if redis_cache:
|
||||
redis_block = (
|
||||
" # Set up sccache/ccache with Redis for the cross build.\n"
|
||||
" - export SCCACHE_REDIS_ENDPOINT=tcp://redis.default.svc.cluster.local:6379\n"
|
||||
" - export SCCACHE_REDIS_PASSWORD=$REDIS_PASSWORD\n"
|
||||
" - export CCACHE_REMOTE_STORAGE=redis://redis.default.svc.cluster.local:6379\n"
|
||||
" - export CCACHE_REMOTE_ONLY=1\n"
|
||||
)
|
||||
|
||||
if attic:
|
||||
attic_block = (
|
||||
' # Push built paths to the binary cache while we build.\n'
|
||||
' - attic login ci https://nix-cache-upload.oleks.space "$ATTIC_TOKEN"\n'
|
||||
" - attic watch-store attic-infra-cache-k3s-1 &\n"
|
||||
" - ATTIC_PID=$!\n"
|
||||
" - sleep 2\n"
|
||||
)
|
||||
attic_teardown = (
|
||||
' - if ! kill $ATTIC_PID 2>/dev/null; then echo "attic watch-store already exited"; fi\n'
|
||||
' - if ! wait $ATTIC_PID 2>/dev/null; then echo "attic watch-store exited non-zero after kill (expected)"; fi\n'
|
||||
)
|
||||
else:
|
||||
attic_block = ""
|
||||
attic_teardown = ""
|
||||
|
||||
return f"""{HEADER}labels:
|
||||
arch: {arch}
|
||||
|
||||
clone:
|
||||
- name: clone
|
||||
image: woodpeckerci/plugin-git
|
||||
environment:
|
||||
CI_NETRC_MACHINE: git.oleks.space
|
||||
CI_NETRC_USERNAME: oleks
|
||||
CI_NETRC_PASSWORD:
|
||||
from_secret: gitea_clone_token
|
||||
PLUGIN_TAGS: "false"
|
||||
PLUGIN_DEPTH: "1"
|
||||
|
||||
when:
|
||||
- event: tag
|
||||
ref: "refs/tags/v*"
|
||||
|
||||
steps:
|
||||
- name: build-and-publish
|
||||
image: git.oleks.space/oleks/nix-ci-ccpp-rust:latest
|
||||
backend_options:
|
||||
kubernetes:
|
||||
resources:
|
||||
requests:
|
||||
memory: {mem_req}
|
||||
limits:
|
||||
memory: {mem_lim}
|
||||
environment:
|
||||
{env_block}
|
||||
commands:
|
||||
- echo "▸ arch=$(uname -m)"
|
||||
- echo "nameserver 169.254.20.10" > /etc/resolv.conf && echo "options ndots:1" >> /etc/resolv.conf
|
||||
{nix_conf_block}
|
||||
- nixos-ci-entrypoint true
|
||||
{redis_block}{attic_block} # Thin parity entrypoint: build + publish all s390x wheels (CI mutates).
|
||||
- PUBLISH=1 nix run .#publish
|
||||
{attic_teardown}"""
|
||||
|
||||
|
||||
def main():
|
||||
manifest = yaml.safe_load((HERE / "manifest.yaml").read_text())
|
||||
check_only = "--check" in sys.argv
|
||||
changed = []
|
||||
for repo in manifest["repos"]:
|
||||
target = BUILDING / repo["dir"] / ".woodpecker.yaml"
|
||||
rendered = render(repo)
|
||||
if check_only:
|
||||
current = target.read_text() if target.exists() else ""
|
||||
if current != rendered:
|
||||
changed.append(str(target))
|
||||
else:
|
||||
target.write_text(rendered)
|
||||
print(f"wrote {target}")
|
||||
if check_only:
|
||||
if changed:
|
||||
print("NOT up to date:")
|
||||
for c in changed:
|
||||
print(f" {c}")
|
||||
sys.exit(1)
|
||||
print("all rendered files up to date")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,95 @@
|
||||
# Manifest for the "pypi-wheel-attic" Woodpecker template (template.yaml.py).
|
||||
#
|
||||
# Scope note (oleks/ci-scripts#1): only repos that share the SAME pipeline
|
||||
# shape (single build-and-publish step, inline `attic watch-store` accelerator
|
||||
# optional, PLUGIN_TAGS=false/PLUGIN_DEPTH=1) are templated here. A second,
|
||||
# genuinely different family (asyncpg/cryptography/fastuuid/lightningcss/
|
||||
# nextjs-swc/rollup/onnxruntime/scikit-learn/scipy — "thin parity-lib publish"
|
||||
# with per-repo Redis sccache and a pipeline-doctor gate) was deliberately left
|
||||
# out: those differ in real, load-bearing ways (redis cache wiring, doctor
|
||||
# step, CI_REGISTRY_TOKEN vs REGISTRY_TOKEN), not accidental copy-paste drift.
|
||||
# Forcing them into this template would either lose functionality or bloat the
|
||||
# manifest schema for a family that isn't actually duplicated. Tracked as
|
||||
# follow-up scope, not done here.
|
||||
#
|
||||
# Fields:
|
||||
# dir - path relative to ~/projects/building/
|
||||
# arch - node-pinning label (oleks/ci-scripts#5): amd64 (cross-build
|
||||
# host) or arm64 (native build host)
|
||||
# memory_request / memory_limit - k8s resource sizing
|
||||
# cores - Nix `cores` setting written into /etc/nix/nix.conf
|
||||
# max_jobs - whether to also cap `max-jobs = 1` (small/serial builds)
|
||||
# attic - whether to run the inline `attic watch-store` accelerator
|
||||
|
||||
repos:
|
||||
- dir: s390x/lxml-s390x
|
||||
arch: amd64
|
||||
memory_request: 2Gi
|
||||
memory_limit: 6Gi
|
||||
cores: 2
|
||||
max_jobs: true
|
||||
attic: true
|
||||
- dir: s390x/pillow-s390x
|
||||
arch: amd64
|
||||
memory_request: 6Gi
|
||||
memory_limit: 6Gi
|
||||
cores: 2
|
||||
max_jobs: true
|
||||
attic: true
|
||||
- dir: s390x/psycopg-s390x
|
||||
arch: amd64
|
||||
memory_request: 6Gi
|
||||
memory_limit: 6Gi
|
||||
cores: 2
|
||||
max_jobs: true
|
||||
attic: true
|
||||
- dir: s390x/psycopg2-binary-s390x
|
||||
arch: amd64
|
||||
memory_request: 6Gi
|
||||
memory_limit: 6Gi
|
||||
cores: 2
|
||||
max_jobs: true
|
||||
attic: true
|
||||
- dir: s390x/tiktoken-s390x
|
||||
arch: amd64
|
||||
memory_request: 6Gi
|
||||
memory_limit: 6Gi
|
||||
cores: 2
|
||||
max_jobs: true
|
||||
attic: true
|
||||
redis_cache: true
|
||||
- dir: s390x/duckdb-s390x
|
||||
arch: amd64
|
||||
memory_request: 8Gi
|
||||
memory_limit: 32Gi
|
||||
cores: 4
|
||||
max_jobs: true
|
||||
attic: true
|
||||
- dir: s390x/faiss-cpu-s390x
|
||||
arch: amd64
|
||||
memory_request: 8Gi
|
||||
memory_limit: 32Gi
|
||||
cores: 4
|
||||
max_jobs: true
|
||||
attic: true
|
||||
- dir: s390x/numpy-s390x
|
||||
arch: amd64
|
||||
memory_request: 8Gi
|
||||
memory_limit: 32Gi
|
||||
cores: 4
|
||||
max_jobs: true
|
||||
attic: true
|
||||
- dir: s390x/pymupdf-s390x
|
||||
arch: amd64
|
||||
memory_request: 8Gi
|
||||
memory_limit: 50Gi
|
||||
cores: null
|
||||
max_jobs: false
|
||||
attic: true
|
||||
- dir: s390x/markupsafe-s390x
|
||||
arch: amd64
|
||||
memory_request: 2Gi
|
||||
memory_limit: 6Gi
|
||||
cores: 2
|
||||
max_jobs: true
|
||||
attic: false
|
||||
Reference in New Issue
Block a user