oleks/ci-scripts
1
0
Fork 0
Code Issues 8 Pull Requests Actions Packages Projects Releases Wiki Activity

Consolidate s390x/arm64 cross-build fleet into one shared template #1

New Issue
Closed
opened 2026-07-04 19:08:21 +03:00 by oleks · 1 comment
oleks commented 2026-07-04 19:08:21 +03:00
Owner
Copy Link
Copy Source

23 files under building/s390x/ plus building/arm64/{angie,baml-py,devpi,mempalace} are near-byte-identical (diff between pillow-s390x and lxml-s390x is 3 substantive lines: label key, memory, comment casing). 13 carry the same hand-rolled attic block (attic login && attic watch-store & … kill $ATTIC_PID). Collapse ~1,200 lines into one shared template with per-package overrides (label, memory, package name). Biggest single win.

23 files under building/s390x/ plus building/arm64/{angie,baml-py,devpi,mempalace} are near-byte-identical (diff between pillow-s390x and lxml-s390x is 3 substantive lines: label key, memory, comment casing). 13 carry the same hand-rolled attic block (attic login && attic watch-store & … kill $ATTIC_PID). Collapse ~1,200 lines into one shared template with per-package overrides (label, memory, package name). Biggest single win.
oleks added this to the CI Scripts Optimization milestone 2026-07-04 19:08:21 +03:00
repo-worker commented 2026-07-04 19:24:13 +03:00
Collaborator
Copy Link
Copy Source

Templated the family of repos that are genuinely near-identical (single build-and-publish step, PLUGIN_TAGS=false/depth=1, optional inline attic watch-store accelerator, PyPI wheel publish on tag): manifest.yaml + generate.py in oleks/ci-scripts (7a434fa, on top of 6b0d0d4) render .woodpecker.yaml for 10 repos — lxml, pillow, psycopg, psycopg2-binary, tiktoken, duckdb, faiss-cpu, numpy, pymupdf, markupsafe-s390x.

Regenerated + pushed to each repo's main:

  • lxml-s390x f9e3c55, pillow-s390x 935483f, psycopg-s390x 1ff1397, psycopg2-binary-s390x 600d41a, tiktoken-s390x e3c0e55, duckdb-s390x 3e4798f, faiss-cpu-s390x 73747ba, numpy-s390x 1bd7151, pymupdf-s390x 0e74e55, markupsafe-s390x c31293c.

Verification: render is idempotent (generate.py --check clean after a second run), all 10 regenerated files pass woodpecker-cli lint, and I diffed each against the pre-change original (parsed YAML, not text) confirming semantic equivalence modulo the intentional #5 node-pinning migration and the #6 || true fix. No pipelines fired from the push (these trigger on tag: v*, not push: main).

Deliberately NOT templated (documented in manifest.yaml + README): asyncpg, cryptography, fastuuid, lightningcss, nextjs-swc, rollup, onnxruntime, scikit-learn, scipy run a different "thin parity-lib publish" shape with real per-repo differences (Redis sccache wiring, pipeline-doctor gate, CI_REGISTRY_TOKEN vs REGISTRY_TOKEN) — templating them would either lose functionality or bloat the manifest schema for differences that aren't accidental duplication. geesefs-s390x, sentry-cli-s390x, attic-client-s390x, angie-arm64, devpi-arm64, mempalace are bespoke pipelines (OCI image publish, binary builds, multi-file workflows) — also out of scope. Consider a follow-up issue if the second family should get its own smaller template (redis+doctor toggles).

Closing.

Templated the family of repos that are genuinely near-identical (single build-and-publish step, PLUGIN_TAGS=false/depth=1, optional inline `attic watch-store` accelerator, PyPI wheel publish on tag): `manifest.yaml` + `generate.py` in oleks/ci-scripts (7a434fa, on top of 6b0d0d4) render `.woodpecker.yaml` for 10 repos — lxml, pillow, psycopg, psycopg2-binary, tiktoken, duckdb, faiss-cpu, numpy, pymupdf, markupsafe-s390x. Regenerated + pushed to each repo's main: - lxml-s390x f9e3c55, pillow-s390x 935483f, psycopg-s390x 1ff1397, psycopg2-binary-s390x 600d41a, tiktoken-s390x e3c0e55, duckdb-s390x 3e4798f, faiss-cpu-s390x 73747ba, numpy-s390x 1bd7151, pymupdf-s390x 0e74e55, markupsafe-s390x c31293c. Verification: render is idempotent (`generate.py --check` clean after a second run), all 10 regenerated files pass `woodpecker-cli lint`, and I diffed each against the pre-change original (parsed YAML, not text) confirming semantic equivalence modulo the intentional #5 node-pinning migration and the #6 `|| true` fix. No pipelines fired from the push (these trigger on `tag: v*`, not `push: main`). Deliberately NOT templated (documented in manifest.yaml + README): `asyncpg`, `cryptography`, `fastuuid`, `lightningcss`, `nextjs-swc`, `rollup`, `onnxruntime`, `scikit-learn`, `scipy` run a different "thin parity-lib publish" shape with real per-repo differences (Redis sccache wiring, pipeline-doctor gate, `CI_REGISTRY_TOKEN` vs `REGISTRY_TOKEN`) — templating them would either lose functionality or bloat the manifest schema for differences that aren't accidental duplication. `geesefs-s390x`, `sentry-cli-s390x`, `attic-client-s390x`, `angie-arm64`, `devpi-arm64`, `mempalace` are bespoke pipelines (OCI image publish, binary builds, multi-file workflows) — also out of scope. Consider a follow-up issue if the second family should get its own smaller template (redis+doctor toggles). Closing.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Bug or defect (requires fix)
 epic:modern-ci
p1
No labels
Milestone
No items
No Milestone CI Scripts Optimization
Projects
Clear projects
No projects
Assignees
Clear assignees
repo-worker oleks
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: oleks/ci-scripts#1
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?