cms-plugins/DEPLOYMENT.md

Deployment

How a code change in this repo becomes a running pod at cms-plugins-{staging,production}.kotkanagrilli.fi.

Repos involved

Repo	Lives at	Owns
cms-plugins (this one)	git.oleks.space/oleks/cms-plugins	App source, Dockerfile, Helm chart, Woodpecker pipeline
anton-helm-workloads	git.oleks.space/anton/helm-workloads	The FluxCD HelmReleases for each subdomain on kotkanagrilli.fi. The legacy WP kotkanagrilli/ and the dev hello-kotkan/ releases also live here.
Gitea OCI registry	git.oleks.space/oleks/cms-plugins (image)	Container images pushed by CI

The cms-plugins repo and anton-helm-workloads repo are linked by:

1. The Helm chart path. The HelmRelease in anton-helm-workloads references chart: ./deploy/helm from a GitRepository source that points back at the staging or production branch of cms-plugins. Flux pulls the chart directly from this repo — there is no "publish chart" step in CI.
2. The image policy. A FluxCD ImagePolicy in anton-helm-workloads watches the staging / production floating tag in the Gitea OCI registry and rewrites the resolved digest into helmrelease.yaml, which is what makes helm upgrade see a change when CI retags.

Branch → environment

Three branches, three environments. Promotion is fast-forward only — no PR ceremony.

Branch	Environment	URL	Where it runs
develop	dev	https://cms-plugins.ddev.site/	Local DDEV
staging	staging	https://cms-plugins-staging.kotkanagrilli.fi/	k3s on kotkan
production	production	https://cms-plugins-production.kotkanagrilli.fi/	k3s on kotkan

What happens on a push

A push to staging or production triggers .woodpecker/container.yaml:

1. Spins up a remote buildkit at buildkit-rootless-arm64.infra.svc.cluster.local:1234.
2. Builds the root Dockerfile.
3. Publishes three refs for each build:
   • git.oleks.space/oleks/cms-plugins:0.1.<pipeline> — immutable audit tag.
   • git.oleks.space/oleks/cms-plugins:<branch> — floating channel pointer; this is what FluxCD's ImagePolicy watches.
   • git.oleks.space/oleks/cms-plugins:<branch>-latest — cosmetic; falls back to a real ref if the chart's image.tag is consulted (which the digest-pinning path normally bypasses).

FluxCD then takes over (it polls every minute):

1. The ImagePolicy in anton-helm-workloads/cms-plugins-<env>/image-automation.yaml resolves the new digest behind the <branch> tag.
2. ImageUpdateAutomation writes that digest back into helmrelease.yaml and commits/pushes to anton-helm-workloads:main.
3. The GitRepository in anton-helm-workloads/cms-plugins-<env>/source.yaml syncs the cms-plugins branch and exposes deploy/helm/.
4. The HelmRelease reconciles: helm upgrade sees a new digest in values.image.digest and rolls the Deployment.

End-to-end deploy time: ≈4 min (build ≈3 min + Flux interval ≈1 min).

Local development

cd ~/projects/cms-plugins
ddev start
# → https://cms-plugins.ddev.site/  (admin at /_emdash/admin)

This is not a prod-parity smoke test — DDEV runs npx emdash dev with hot reload; production runs the built node ./dist/server/entry.mjs. The shared artifact is the Astro source under app/. To smoke-test the production image standalone:

docker build -t cms-plugins:dev .
docker run --rm -p 4321:4321 \
  -e EMDASH_ENCRYPTION_KEY=$(openssl rand -hex 32) \
  -v cms-plugins-state:/app/state \
  cms-plugins:dev

Promoting develop → staging

git switch staging
git merge --ff-only develop
git push origin staging

The Woodpecker pipeline triggers automatically. Watch it at the Woodpecker UI; once green, Flux reconciles within ≤1 min.

Promoting staging → production

After staging looks good:

git switch production
git merge --ff-only staging
git push origin production

If the fast-forward fails, something is out of order — investigate before forcing.

Rolling back

The chart pins by digest (set by FluxCD's ImageUpdateAutomation), with <branch>-latest as a fallback tag. To roll back:

1. Identify the previous good build's immutable tag from the Gitea registry — it is 0.1.<N> where <N> is the pipeline number.
2. In anton-helm-workloads/cms-plugins-<env>/helmrelease.yaml, hard-code the previous digest under values.image.digest and remove the {"$imagepolicy": ...} marker (so IUA stops overwriting it). Commit and push.
3. Flux reconciles in ≤1 min.

To resume tracking the floating tag, restore the marker and the auto-update resumes.

Secrets

All secrets are sops-encrypted in anton-helm-workloads:

• anton-helm-workloads/cms-plugins-staging/secrets.yaml
• anton-helm-workloads/cms-plugins-production/secrets.yaml

Each env has two distinct Secrets:

Secret	Namespace	Purpose
cms-plugins-deploy-key	kotkan	SSH deploy key for Flux to clone cms-plugins (one pair shared between staging + production — same key reads both branches). Defined once in the staging overlay; co-located with the GitRepositories in kotkan so the secretRef is same-namespace.
cms-plugins-<env>-secrets	kotkan	Env vars the pod consumes via existingSecret. Required key: EMDASH_ENCRYPTION_KEY.

To rotate a credential:

cd ~/projects/servers/anton/anton-helm-workloads/cms-plugins-staging
sops secrets.yaml         # opens decrypted in $EDITOR
git add . && git commit -m "rotate cms-plugins staging credential"
git push                  # Flux re-applies in ≤1 min

The Deployment does NOT restart on Secret change (k8s envFrom doesn't watch). Force a roll after rotating:

kubectl -n kotkan rollout restart deployment/cms-plugins-staging

First-time setup checklist

This is the once-per-environment dance to bring staging or production online.

A. In this (cms-plugins) repo

1. Push to git.oleks.space/oleks/cms-plugins with three branches: develop, staging, production. The initial commit lives on develop; the other two are fast-forward copies.
2. Add the Woodpecker secrets at Repo → Settings → Secrets in the Woodpecker UI (ci.oleks.space), scoped to event push:

   Secret	Purpose
   gitea_clone_token	Gitea PAT for cloning during the build step.
   registry_token	Gitea PAT with write access to git.oleks.space/oleks/cms-plugins (container packages).

B. In the anton-helm-workloads repo

1. Generate an SSH deploy key (one pair, shared across envs):

   ssh-keygen -t ed25519 -f /tmp/cms-plugins-deploy -N ""
   

2. Upload /tmp/cms-plugins-deploy.pub to the cms-plugins repo on Gitea: Settings → Deploy Keys → "cms-plugins Flux deploy", read-only.

3. Generate one EMDASH_ENCRYPTION_KEY per env (do NOT reuse):

   openssl rand -hex 32   # staging
   openssl rand -hex 32   # production
   

4. Copy this repo's deploy/fleet-overlay/cms-plugins-staging/ and cms-plugins-production/ directories into the root of anton-helm-workloads, then edit each secrets.yaml to put the real key material in, and sops-encrypt:

   cd anton-helm-workloads/cms-plugins-staging
   sops --encrypt --age <recipient> secrets.yaml > secrets.enc.yaml
   mv secrets.enc.yaml secrets.yaml
   

5. Add both directories to anton-helm-workloads/kustomization.yaml:

   resources:
     - hello-kotkan
     - kotkanagrilli
     - max-openclaw
     - terminal-agent
     - cms-plugins-staging        # ← add
     - cms-plugins-production     # ← add
   

6. Ensure a writable GitRepository named anton-workloads-image-automation exists in flux-system for the ImageUpdateAutomation to push digest commits to anton-helm-workloads:main. If it doesn't exist yet, create it (this is a one-time setup for the whole workloads repo). The emdash-kotkanagrilli analogue is oleks-fleet-image-automation.

7. Commit and push anton-helm-workloads. Flux picks it up in ≤1 min.

C. DNS

dig +short cms-plugins-staging.kotkanagrilli.fi
dig +short cms-plugins-production.kotkanagrilli.fi

Both should resolve to the kotkan ingress IP (the same one emdash-staging.kotkanagrilli.fi already uses).

D. First deploy

Go-live prerequisite — floating tag must exist (issue #3). While the HelmReleases are suspend: true (Phase 0) and no pod is running, the floating staging/production tags are referenced by no live workload. gitea-oci-cleanup is fleet-aware and only auto-pins in-use tags, so an unreferenced floating tag is reapable. If it gets reaped before go-live, the ImagePolicy has nothing to resolve and the first deploy can't pin a digest. Mitigation: the floating tags are explicitly pinned in servers/armer/scripts/registry-pins.txt (container/cms-plugins==staging, container/cms-plugins==production) so the cleaner always keeps them. Before the first flux resume, confirm both tags still exist (oleks/-/packages/container/cms-plugins); if not, push the branch to rebuild them first.

1. Re-run/confirm the staging build so git.oleks.space/oleks/cms-plugins:staging exists. Watch Woodpecker.

2. Resume the release and watch Flux reconcile + the rollout:

   flux resume helmrelease cms-plugins-staging -n kotkan
   kubectl -n kotkan get pods -w | grep cms-plugins
   flux get all -n flux-system | grep cms-plugins
   

3. Once cms-plugins-staging.kotkanagrilli.fi returns a 200, fast-forward production to staging and push.

Why this shape

• Single replica, pinned to kotkan. Emdash uses local SQLite (single-writer); horizontal scaling isn't a thing here. The PVC is local-path, so the pod can only run where the volume lives.
• Mutable <branch> tag. The sha-tagged ref (0.1.<pipeline>) gives audit trail; the mutable <branch> tag is what Flux watches. Together they give visibility without forcing an anton-helm-workloads commit per deploy (the ImageUpdateAutomation does that automatically).
• Flux pulls the chart from this repo (no OCI step). Keeps the chart in the same git history as the app code; a chart change rolls with the same release.
• HelmRelease lives in anton-helm-workloads, not in fleet. Matches the convention that the kotkanagrilli.fi subdomain pool (legacy kotkanagrilli/, hello-kotkan/, future subdomains) is reconciled out of the workloads repo, not the personal fleet repo.
• kotkan, not armer. The legacy kotkanagrilli.fi site and the emdash replacement both live on kotkan; colocating reduces DNS / ingress drift while the WP → Emdash story plays out.