cms-plugins/DEPLOYMENT.md

# Deployment

How a code change in this repo becomes a running pod at
`cms-plugins-{staging,production}.kotkanagrilli.fi`.

## Repos involved

| Repo | Lives at | Owns |
|---|---|---|
| **cms-plugins** (this one) | `git.oleks.space/oleks/cms-plugins` | App source, Dockerfile, Helm chart, Woodpecker pipeline |
| **anton-helm-workloads** | `git.oleks.space/anton/helm-workloads` | The FluxCD `HelmRelease`s for each subdomain on `kotkanagrilli.fi`. The legacy WP `kotkanagrilli/` and the dev `hello-kotkan/` releases also live here. |
| Gitea OCI registry | `git.oleks.space/oleks/cms-plugins` (image) | Container images pushed by CI |

The `cms-plugins` repo and `anton-helm-workloads` repo are linked by:

1. **The Helm chart path.** The HelmRelease in `anton-helm-workloads`
   references `chart: ./deploy/helm` from a `GitRepository` source that
   points back at the `staging` or `production` branch of `cms-plugins`.
   Flux pulls the chart directly from this repo — there is no
   "publish chart" step in CI.
2. **The image policy.** A FluxCD `ImagePolicy` in `anton-helm-workloads`
   watches the `staging` / `production` floating tag in the Gitea OCI
   registry and rewrites the resolved digest into `helmrelease.yaml`,
   which is what makes `helm upgrade` see a change when CI retags.

## Branch → environment

Three branches, three environments. Promotion is fast-forward only — no PR
ceremony.

| Branch | Environment | URL | Where it runs |
|---|---|---|---|
| `develop` | dev | `https://cms-plugins.ddev.site/` | Local DDEV |
| `staging` | staging | `https://cms-plugins-staging.kotkanagrilli.fi/` | k3s on `kotkan` |
| `production` | production | `https://cms-plugins-production.kotkanagrilli.fi/` | k3s on `kotkan` |

## What happens on a push

A push to `staging` or `production` triggers `.woodpecker/container.yaml`:

1. Spins up a remote buildkit at `buildkit-rootless-arm64.infra.svc.cluster.local:1234`.
2. Builds the root `Dockerfile`.
3. Publishes **three refs** for each build:
   - `git.oleks.space/oleks/cms-plugins:0.1.<pipeline>` — immutable audit tag.
   - `git.oleks.space/oleks/cms-plugins:<branch>` — floating channel pointer; this is what FluxCD's `ImagePolicy` watches.
   - `git.oleks.space/oleks/cms-plugins:<branch>-latest` — cosmetic; falls back to a real ref if the chart's `image.tag` is consulted (which the digest-pinning path normally bypasses).

FluxCD then takes over (it polls every minute):

1. The `ImagePolicy` in `anton-helm-workloads/cms-plugins-<env>/image-automation.yaml` resolves the new digest behind the `<branch>` tag.
2. `ImageUpdateAutomation` writes that digest back into `helmrelease.yaml` and commits/pushes to `anton-helm-workloads:main`.
3. The `GitRepository` in `anton-helm-workloads/cms-plugins-<env>/source.yaml` syncs the `cms-plugins` branch and exposes `deploy/helm/`.
4. The `HelmRelease` reconciles: `helm upgrade` sees a new digest in `values.image.digest` and rolls the Deployment.

End-to-end deploy time: ≈4 min (build ≈3 min + Flux interval ≈1 min).

## Local development

```bash
cd ~/projects/cms-plugins
ddev start
# → https://cms-plugins.ddev.site/  (admin at /_emdash/admin)
```

This is **not** a prod-parity smoke test — DDEV runs `npx emdash dev` with
hot reload; production runs the built `node ./dist/server/entry.mjs`. The
shared artifact is the Astro source under `app/`. To smoke-test the
production image standalone:

```bash
docker build -t cms-plugins:dev .
docker run --rm -p 4321:4321 \
  -e EMDASH_ENCRYPTION_KEY=$(openssl rand -hex 32) \
  -v cms-plugins-state:/app/state \
  cms-plugins:dev
```

## Promoting develop → staging

```bash
git switch staging
git merge --ff-only develop
git push origin staging
```

The Woodpecker pipeline triggers automatically. Watch it at the Woodpecker
UI; once green, Flux reconciles within ≤1 min.

## Promoting staging → production

After staging looks good:

```bash
git switch production
git merge --ff-only staging
git push origin production
```

If the fast-forward fails, something is out of order — investigate before
forcing.

## Rolling back

The chart pins by **digest** (set by FluxCD's
`ImageUpdateAutomation`), with `<branch>-latest` as a fallback tag. To
roll back:

1. Identify the previous good build's immutable tag from the Gitea
   registry — it is `0.1.<N>` where `<N>` is the pipeline number.
2. In `anton-helm-workloads/cms-plugins-<env>/helmrelease.yaml`, hard-code
   the previous digest under `values.image.digest` and remove the
   `{"$imagepolicy": ...}` marker (so IUA stops overwriting it). Commit
   and push.
3. Flux reconciles in ≤1 min.

To resume tracking the floating tag, restore the marker and the
auto-update resumes.

## Secrets

All secrets are sops-encrypted in `anton-helm-workloads`:

- `anton-helm-workloads/cms-plugins-staging/secrets.yaml`
- `anton-helm-workloads/cms-plugins-production/secrets.yaml`

Each env has two distinct Secrets:

| Secret | Namespace | Purpose |
|---|---|---|
| `cms-plugins-deploy-key` | `kotkan` | SSH deploy key for Flux to clone `cms-plugins` (one pair shared between staging + production — same key reads both branches). Defined once in the staging overlay; co-located with the GitRepositories in `kotkan` so the `secretRef` is same-namespace. |
| `cms-plugins-<env>-secrets` | `kotkan` | Env vars the pod consumes via `existingSecret`. Required key: `EMDASH_ENCRYPTION_KEY`. |

To rotate a credential:

```bash
cd ~/projects/servers/anton/anton-helm-workloads/cms-plugins-staging
sops secrets.yaml         # opens decrypted in $EDITOR
git add . && git commit -m "rotate cms-plugins staging credential"
git push                  # Flux re-applies in ≤1 min
```

The Deployment does NOT restart on Secret change (k8s `envFrom` doesn't
watch). Force a roll after rotating:

```bash
kubectl -n kotkan rollout restart deployment/cms-plugins-staging
```

## First-time setup checklist

This is the once-per-environment dance to bring staging or production
online.

### A. In this (cms-plugins) repo

1. Push to `git.oleks.space/oleks/cms-plugins` with three branches:
   `develop`, `staging`, `production`. The initial commit lives on
   `develop`; the other two are fast-forward copies.
2. Add the Woodpecker secrets at Repo → Settings → Secrets in the
   Woodpecker UI (`ci.oleks.space`), scoped to event `push`:

   | Secret | Purpose |
   |---|---|
   | `gitea_clone_token` | Gitea PAT for cloning during the build step. |
   | `registry_token` | Gitea PAT with write access to `git.oleks.space/oleks/cms-plugins` (container packages). |

### B. In the anton-helm-workloads repo

1. Generate an SSH deploy key (one pair, shared across envs):

   ```bash
   ssh-keygen -t ed25519 -f /tmp/cms-plugins-deploy -N ""
   ```

2. Upload `/tmp/cms-plugins-deploy.pub` to the `cms-plugins` repo on
   Gitea: Settings → Deploy Keys → "cms-plugins Flux deploy",
   **read-only**.
3. Generate one `EMDASH_ENCRYPTION_KEY` per env (do NOT reuse):

   ```bash
   openssl rand -hex 32   # staging
   openssl rand -hex 32   # production
   ```

4. Copy this repo's `deploy/fleet-overlay/cms-plugins-staging/` and
   `cms-plugins-production/` directories into the root of
   `anton-helm-workloads`, then edit each `secrets.yaml` to put the
   real key material in, and sops-encrypt:

   ```bash
   cd anton-helm-workloads/cms-plugins-staging
   sops --encrypt --age <recipient> secrets.yaml > secrets.enc.yaml
   mv secrets.enc.yaml secrets.yaml
   ```

5. Add both directories to `anton-helm-workloads/kustomization.yaml`:

   ```yaml
   resources:
     - hello-kotkan
     - kotkanagrilli
     - max-openclaw
     - terminal-agent
     - cms-plugins-staging        # ← add
     - cms-plugins-production     # ← add
   ```

6. Ensure a writable `GitRepository` named
   `anton-workloads-image-automation` exists in `flux-system` for the
   `ImageUpdateAutomation` to push digest commits to
   `anton-helm-workloads:main`. If it doesn't exist yet, create it
   (this is a one-time setup for the whole workloads repo). The
   `emdash-kotkanagrilli` analogue is `oleks-fleet-image-automation`.
7. Commit and push `anton-helm-workloads`. Flux picks it up in ≤1 min.

### C. DNS

```bash
dig +short cms-plugins-staging.kotkanagrilli.fi
dig +short cms-plugins-production.kotkanagrilli.fi
```

Both should resolve to the `kotkan` ingress IP (the same one
`emdash-staging.kotkanagrilli.fi` already uses).

### D. First deploy

1. Push to `staging` to trigger the first build. Watch Woodpecker.
2. Once the image lands, Flux reconciles. Watch the rollout:

   ```bash
   kubectl -n kotkan get pods -w | grep cms-plugins
   flux get all -n flux-system | grep cms-plugins
   ```

3. Once `cms-plugins-staging.kotkanagrilli.fi` returns a 200, fast-forward
   `production` to `staging` and push.

## Why this shape

- **Single replica, pinned to `kotkan`.** Emdash uses local SQLite
  (single-writer); horizontal scaling isn't a thing here. The PVC is
  `local-path`, so the pod can only run where the volume lives.
- **Mutable `<branch>` tag.** The sha-tagged ref (`0.1.<pipeline>`) gives
  audit trail; the mutable `<branch>` tag is what Flux watches. Together
  they give visibility without forcing an `anton-helm-workloads` commit
  per deploy (the `ImageUpdateAutomation` does that automatically).
- **Flux pulls the chart from this repo (no OCI step).** Keeps the chart
  in the same git history as the app code; a chart change rolls with the
  same release.
- **HelmRelease lives in `anton-helm-workloads`, not in fleet.** Matches
  the convention that the kotkanagrilli.fi subdomain pool (legacy
  `kotkanagrilli/`, `hello-kotkan/`, future subdomains) is reconciled
  out of the workloads repo, not the personal fleet repo.
- **kotkan, not armer.** The legacy `kotkanagrilli.fi` site and the
  emdash replacement both live on `kotkan`; colocating reduces DNS /
  ingress drift while the WP → Emdash story plays out.