oleks/ci-scripts
1
0
Fork 0
Code Issues 8 Pull Requests Actions Packages Projects Releases Wiki Activity

Second fleet template for the parity-lib publish family #10

New Issue
Closed
opened 2026-07-04 19:26:57 +03:00 by repo-worker · 1 comment
repo-worker commented 2026-07-04 19:26:57 +03:00
Collaborator
Copy Link
Copy Source

Follow-up from #1. While building the pypi-wheel-attic template for the 10 near-identical s390x repos, 9 more s390x repos turned out to share a second, genuinely-different-but-still-duplicated shape: "thin parity-lib publish." They were deliberately left out of #1's template because their differences are real, not copy-paste drift — but the shape itself repeats across all 9 and would benefit from its own smaller template.

Repos

asyncpg, cryptography, fastuuid, lightningcss, nextjs-swc, rollup, onnxruntime, scikit-learn, scipy (all -s390x).

Shared shape

  • Two steps: doctor (runs parity-lib#pipeline-doctor --strict as an enforcement gate before publish) + publish.
  • PLUGIN_TAGS: "true" / PLUGIN_DEPTH: "0" (full clone with tags — some repos derive VERSION via git describe fallback), vs. #1's family which uses "false"/"1".
  • CI_REGISTRY_TOKEN (aliased to $REGISTRY_TOKEN inside the script) instead of REGISTRY_TOKEN directly from secret.
  • No inline attic watch-store accelerator — publish is a pure nix run .#publish[-s390x] front door.

Real per-repo variance to parameterize (not paper over)

  • Redis sccache/ccache wiring — present in cryptography, fastuuid, lightningcss, nextjs-swc, rollup, onnxruntime, scikit-learn, scipy; absent in asyncpg. When present, needs REDIS_PASSWORD secret + the 4-line SCCACHE_REDIS_*/CCACHE_REMOTE_* export block.
  • Memory: mostly 8Gi/32Gi request/limit, but pymupdf-style outliers exist elsewhere in the fleet — verify per-repo before assuming a single default.
  • Cores: 2 vs 4 vs unset, inconsistent even among repos that otherwise look identical.
  • doctor step presence: confirmed on asyncpg; verify per-repo whether all 9 actually run it or if it's asyncpg-specific (the diff sample used in #1's investigation was small — re-audit each file before templating).

Suggested approach

Same manifest.yaml + generate.py pattern as #1, in the same oleks/ci-scripts repo, as either a second manifest file or a family: field added to the existing manifest with per-family render functions. Re-verify semantic equivalence the same way #1 did: parse rendered YAML vs. original, diff structurally (not textually), lint with woodpecker-cli lint before regenerating for real, confirm idempotent re-render.

Also apply #5's node-pinning convention (labels: {arch: amd64|arm64}, drop provider/platform variants) and #6's || true ban while templating, same as was done for the first family.

Follow-up from #1. While building the `pypi-wheel-attic` template for the 10 near-identical s390x repos, 9 more s390x repos turned out to share a second, genuinely-different-but-still-duplicated shape: **"thin parity-lib publish."** They were deliberately left out of #1's template because their differences are real, not copy-paste drift — but the shape itself repeats across all 9 and would benefit from its own smaller template. ## Repos asyncpg, cryptography, fastuuid, lightningcss, nextjs-swc, rollup, onnxruntime, scikit-learn, scipy (all `-s390x`). ## Shared shape - Two steps: `doctor` (runs `parity-lib#pipeline-doctor --strict` as an enforcement gate before publish) + `publish`. - `PLUGIN_TAGS: "true"` / `PLUGIN_DEPTH: "0"` (full clone with tags — some repos derive VERSION via `git describe` fallback), vs. #1's family which uses `"false"`/`"1"`. - `CI_REGISTRY_TOKEN` (aliased to `$REGISTRY_TOKEN` inside the script) instead of `REGISTRY_TOKEN` directly from secret. - No inline `attic watch-store` accelerator — publish is a pure `nix run .#publish[-s390x]` front door. ## Real per-repo variance to parameterize (not paper over) - **Redis sccache/ccache wiring** — present in cryptography, fastuuid, lightningcss, nextjs-swc, rollup, onnxruntime, scikit-learn, scipy; absent in asyncpg. When present, needs `REDIS_PASSWORD` secret + the 4-line `SCCACHE_REDIS_*`/`CCACHE_REMOTE_*` export block. - **Memory**: mostly `8Gi`/`32Gi` request/limit, but `pymupdf`-style outliers exist elsewhere in the fleet — verify per-repo before assuming a single default. - **Cores**: `2` vs `4` vs unset, inconsistent even among repos that otherwise look identical. - **`doctor` step presence**: confirmed on asyncpg; verify per-repo whether all 9 actually run it or if it's asyncpg-specific (the diff sample used in #1's investigation was small — re-audit each file before templating). ## Suggested approach Same `manifest.yaml` + `generate.py` pattern as #1, in the same `oleks/ci-scripts` repo, as either a second manifest file or a `family:` field added to the existing manifest with per-family render functions. Re-verify semantic equivalence the same way #1 did: parse rendered YAML vs. original, diff structurally (not textually), lint with `woodpecker-cli lint` before regenerating for real, confirm idempotent re-render. Also apply #5's node-pinning convention (`labels: {arch: amd64|arm64}`, drop `provider`/`platform` variants) and #6's `|| true` ban while templating, same as was done for the first family.
repo-worker added this to the CI Scripts Optimization milestone 2026-07-04 19:26:57 +03:00
repo-worker commented 2026-07-04 19:34:47 +03:00
Author
Collaborator
Copy Link
Copy Source

Re-audited all 9 repos in detail (parsed YAML structure, not just grep) before templating — the real shape was more fragmented than my original one-line description. Disposition:

Templated (5, new render_thin() + thin_repos: in oleks/ci-scripts):
asyncpg-s390x, cryptography-s390x, lightningcss-s390x, nextjs-swc-s390x, rollup-s390x. Parameterized the real variance: doctor gate presence (only asyncpg has it), token_direct (REGISTRY_TOKEN direct vs CI_REGISTRY_TOKEN + explicit remap — only cryptography), publish_style (PUBLISH=1 nix run .#publish vs nix run .#publish-s390x -- --publish), tags_full, and dead_secrets (some declare ATTIC_TOKEN/REDIS_PASSWORD unused — preserved, not cleaned up, out of scope). Verified idempotent + semantically equivalent (parsed-YAML diff) + woodpecker-cli lint clean for all 5. Pushed:

  • asyncpg-s390x 5fa397b, cryptography-s390x 3e878ab, lightningcss-s390x 0fcb8c8, nextjs-swc-s390x 412fae8, rollup-s390x 9b64875.
  • oleks/ci-scripts d3c4c8e (generator+manifest+README) → 68cba21 (ruff fix), rebased twice against templates-worker's concurrent commits.

Excluded, not forced (3 + 1 originally-miscounted):

  • fastuuid-s390x: declares CI_REGISTRY_TOKEN but has NO export REGISTRY_TOKEN=... remap before PUBLISH=1 nix run .#publish — cryptography-s390x (the only other CI_REGISTRY_TOKEN user) DOES have the remap. This looks like a real latent bug: publish would run with REGISTRY_TOKEN unset. Not silently fixed via templating — flagging here for someone to confirm intent.
  • onnxruntime-s390x: same CI_REGISTRY_TOKEN-without-remap pattern as fastuuid, AND it's actually attic+redis shaped (single step) like the OTHER (#1) family, not this one. Left hand-maintained.
  • scikit-learn-s390x, scipy-s390x: also attic+redis shaped like the #1 family, but their Redis setup writes exports to a file via a heredoc that's never sourced elsewhere in the step — a different convention than tiktoken-s390x's direct-export style used in #1's template. Whether the heredoc form does anything meaningful isn't confirmed; normalizing it would be a real behavior change, not a safe refactor. Left hand-maintained.

Net: of the "9," 5 got a real shared template, 4 are flagged/excluded with reasons documented in manifest.yaml and README.md. Two of those four (fastuuid, onnxruntime) may have a genuine pre-existing token-wiring bug worth a separate look.

Closing.

Re-audited all 9 repos in detail (parsed YAML structure, not just grep) before templating — the real shape was more fragmented than my original one-line description. Disposition: **Templated (5, new `render_thin()` + `thin_repos:` in oleks/ci-scripts):** asyncpg-s390x, cryptography-s390x, lightningcss-s390x, nextjs-swc-s390x, rollup-s390x. Parameterized the real variance: `doctor` gate presence (only asyncpg has it), `token_direct` (REGISTRY_TOKEN direct vs CI_REGISTRY_TOKEN + explicit remap — only cryptography), `publish_style` (`PUBLISH=1 nix run .#publish` vs `nix run .#publish-s390x -- --publish`), `tags_full`, and `dead_secrets` (some declare ATTIC_TOKEN/REDIS_PASSWORD unused — preserved, not cleaned up, out of scope). Verified idempotent + semantically equivalent (parsed-YAML diff) + `woodpecker-cli lint` clean for all 5. Pushed: - asyncpg-s390x 5fa397b, cryptography-s390x 3e878ab, lightningcss-s390x 0fcb8c8, nextjs-swc-s390x 412fae8, rollup-s390x 9b64875. - oleks/ci-scripts d3c4c8e (generator+manifest+README) → 68cba21 (ruff fix), rebased twice against templates-worker's concurrent commits. **Excluded, not forced (3 + 1 originally-miscounted):** - **fastuuid-s390x**: declares `CI_REGISTRY_TOKEN` but has NO `export REGISTRY_TOKEN=...` remap before `PUBLISH=1 nix run .#publish` — cryptography-s390x (the only other CI_REGISTRY_TOKEN user) DOES have the remap. This looks like a real latent bug: publish would run with `REGISTRY_TOKEN` unset. Not silently fixed via templating — flagging here for someone to confirm intent. - **onnxruntime-s390x**: same CI_REGISTRY_TOKEN-without-remap pattern as fastuuid, AND it's actually attic+redis shaped (single step) like the OTHER (#1) family, not this one. Left hand-maintained. - **scikit-learn-s390x, scipy-s390x**: also attic+redis shaped like the #1 family, but their Redis setup writes exports to a file via a heredoc that's never sourced elsewhere in the step — a different convention than tiktoken-s390x's direct-export style used in #1's template. Whether the heredoc form does anything meaningful isn't confirmed; normalizing it would be a real behavior change, not a safe refactor. Left hand-maintained. Net: of the "9," 5 got a real shared template, 4 are flagged/excluded with reasons documented in `manifest.yaml` and README.md. Two of those four (fastuuid, onnxruntime) may have a genuine pre-existing token-wiring bug worth a separate look. Closing.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Bug or defect (requires fix)
 epic:modern-ci
p1
No labels
Milestone
No items
No Milestone CI Scripts Optimization
Projects
Clear projects
No projects
Assignees
Clear assignees
repo-worker oleks
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: oleks/ci-scripts#10
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?