README.md

@@ -29,6 +29,20 @@ are excluded from scans — this keeps vendored upstream mirrors' own CI test
 fixtures (e.g. the `woodpecker` repo's own `test-dag`/`test-when`/... corpus)
 out of fleet-wide results. Pass `-all` to include everything.
 
+`-analyze` also flags two Woodpecker footguns that are statically visible in
+a step's `commands:` text, both of which have cost a real pipeline run:
+
+- a stray apostrophe inside a `-c bash -euxc '...'`-style single-quoted
+  shell block — including one inside a `#` comment — closes the quote
+  early and hands the rest of the script to the outer shell, usually
+  surfacing as a baffling "command not found" for a tool that IS
+  installed (oleks/mempalace pipeline 218).
+- an unescaped `${VAR}` in a step command — Woodpecker substitutes braced
+  `${VAR}` in the raw pipeline text at parse time, before the shell runs,
+  so a shell variable meant for the shell needs to be escaped as `$${VAR}`
+  (Woodpecker's own `${CI_*}` builtins and a pipeline's own `matrix:` axis
+  names are the legitimate unescaped exceptions and aren't flagged).
+
 ## Build
 
 ```sh

SPEC.md

@@ -154,6 +154,67 @@ whoever ran this, not part of pipetree's own tracker):
   skipping" instead of an error. Any other status/network failure still
   surfaces as a real error, unchanged.
 
+## Shipped (v0.4, footgun detection)
+
+`-analyze` now also flags two statically-detectable Woodpecker footguns
+inside step `commands:` text (`internal/analyze/footguns.go`), on top of
+the existing shared-image/broken-depends_on checks. Both were motivated by
+a real pipeline failure (oleks/mempalace, 2026-08-03) and grounded in
+reading actual incident commits, not guessed:
+
+- **StrayApostrophe**: the common `nix shell ... -c bash -euxc
+  '...multi-line script...'` idiom wraps the whole script in ONE
+  single-quoted shell argument. Bash gives single quotes no escape
+  mechanism other than the close-escape-reopen trick (`'`, `\`, `''`), so
+  ANY literal `'` inside the block — including inside a `#` comment —
+  closes the quote early and hands everything after it to the outer
+  shell to re-tokenize. Live incident: oleks/mempalace pipeline 218
+  failed `exit 127` ("patch: command not found", though patch WAS
+  installed) because a comment added in `3fb2b02` contained the word
+  "Woodpecker's" — root-caused and fixed in commit `54899ec`
+  (oleks/mempalace#87/#90). The exact same idiom had already been bitten
+  once before (commit `e409e70`, oleks/mempalace#68).
+
+  Detection: find the idiom's opening line (`-c '`/`-euxc '` etc. at the
+  end of a line) and its dedicated closing line (a line that trims to
+  exactly `'`); any additional unescaped `'` strictly between them is
+  flagged. Commands that don't match this precise shape (no dedicated
+  closing line) are left unchecked rather than guessed at, to avoid false
+  positives on differently-structured single-quoted commands.
+
+- **UnescapedWoodpeckerVar**: Woodpecker substitutes braced `${VAR}` in
+  the raw pipeline YAML text at parse time, before the shell ever runs —
+  including inside comments, since the substitution pass has no concept
+  of what's a comment. A shell variable meant for the shell (e.g.
+  `${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}`) gets mangled (undefined ->
+  empty/malformed text), typically surfacing as an "unbound variable"
+  failure under `set -u`. Fix: escape as `$${VAR}` so Woodpecker's
+  substitution collapses the `$$` to a literal `$`, leaving `${VAR}`
+  intact for the shell (commit `33b880d`, oleks/mempalace#44).
+
+  Detection: any `${NAME...}` not immediately preceded by a second `$`
+  (the `$${...}` escape) and whose name isn't one of Woodpecker's own
+  `CI_*` metadata builtins or the pipeline's own `matrix:` axis names —
+  Woodpecker's matrix feature works BY substituting `${AXIS_NAME}`
+  throughout the YAML with each axis value before parsing, so that's a
+  second class of legitimate unescaped use alongside `CI_*`.
+
+**False positive found and fixed during fleet validation**: the first
+fleet-wide run flagged `${TARGET_ARCH}` in `oleks/ii-researcher`,
+`oleks/csi-s3`, and `oleks/common-chronicle` (18 hits total) — all three
+declare `matrix: {TARGET_ARCH: [...]}` and reference `${TARGET_ARCH}`
+unescaped in the step name and commands, which is Woodpecker's matrix
+feature working exactly as documented, not a mistake. Excluding a
+pipeline's own matrix axis names (alongside `CI_*`) dropped this to 0
+false positives; the check still correctly flags real footguns (fleet run
+after the fix: 0 stray apostrophes, 8 genuine unescaped-var findings in
+`oleks/deals-site`, `oleks/element-web-patched`, and
+`oleks/terminal-agent` — shell variables set via step `environment:` or
+local assignment, not Woodpecker metadata). Verified `oleks/mempalace`
+(both the exact pipeline-218-broken commit and its fix) and
+`oleks/oracle-adb-backend` (currently clean) as positive/negative
+fixtures.
+
 ## External validation: `woodpecker-cli`
 
 The official CLI (`woodpecker-cli`, v3.16.0, matches `ci.oleks.space`) is

internal/analyze/analyze.go

@@ -1,7 +1,11 @@
 // Package analyze looks across an already-scanned set of projects for
 // structural smells that don't show up looking at any one pipeline file
 // in isolation: step images worth consolidating, and depends_on
-// references that name something that doesn't exist.
+// references that name something that doesn't exist. It also runs two
+// single-pipeline-local footgun checks (see footguns.go) alongside the
+// cross-pipeline ones, since -analyze is already the closest thing this
+// tool has to a lint mode and both footguns are things a human skims past
+// without noticing.
 //
 // Both depends_on checks below are grounded in reading Woodpecker's own
 // source (git.oleks.space/oleks/woodpecker, pipeline/frontend/yaml/
@@ -68,6 +72,8 @@ type Report struct {
 	SharedImages            []SharedImage
 	BrokenStepDependsOn     []BrokenStepDependsOn
 	BrokenPipelineDependsOn []BrokenPipelineDependsOn
+	StrayApostrophes        []StrayApostrophe
+	UnescapedVars           []UnescapedWoodpeckerVar
 }
 
 // Run analyzes projects and returns a Report, each section sorted for
@@ -77,8 +83,12 @@ func Run(projects []*model.Project) Report {
 	imageRefs := map[string][]PipelineRef{}
 	var brokenStep []BrokenStepDependsOn
 	var brokenPipeline []BrokenPipelineDependsOn
+	var strayApostrophes []StrayApostrophe
+	var unescapedVars []UnescapedWoodpeckerVar
 
 	for _, proj := range projects {
+		runFootguns(proj, &strayApostrophes, &unescapedVars)
+
 		pipelineNames := make(map[string]bool, len(proj.Pipelines))
 		for _, pl := range proj.Pipelines {
 			pipelineNames[pl.Name] = true
@@ -147,5 +157,13 @@ func Run(projects []*model.Project) Report {
 		return brokenPipeline[i].Pipeline < brokenPipeline[j].Pipeline
 	})
 
-	return Report{SharedImages: shared, BrokenStepDependsOn: brokenStep, BrokenPipelineDependsOn: brokenPipeline}
+	sortFootguns(strayApostrophes, unescapedVars)
+
+	return Report{
+		SharedImages:            shared,
+		BrokenStepDependsOn:     brokenStep,
+		BrokenPipelineDependsOn: brokenPipeline,
+		StrayApostrophes:        strayApostrophes,
+		UnescapedVars:           unescapedVars,
+	}
 }

internal/analyze/footguns.go

@@ -0,0 +1,237 @@
+// Footgun detection for two Woodpecker CI gotchas that are statically
+// visible in a step's `commands:` text and have both cost a real pipeline
+// run (oleks/mempalace, 2026-08-03 and originally 2026-07-31):
+//
+//  1. StrayApostrophe: the common `nix shell ... -c bash -euxc '...multi-
+//     line script...'` idiom wraps the whole script in ONE single-quoted
+//     shell argument. Bash gives single quotes no escape mechanism at all
+//     (other than the close-escape-reopen trick, written as a single
+//     quote, backslash, then two single quotes) - so ANY literal
+//     `'` inside the block, including inside a `#` comment, closes the
+//     quote early and hands everything after it to the outer shell to
+//     re-tokenize, usually surfacing as a baffling "command not found" for
+//     a tool that IS installed. Live incident: oleks/mempalace pipeline
+//     218 failed exit 127 ("patch: command not found", patch WAS
+//     installed) because a comment added in 3fb2b02 contained the word
+//     "Woodpecker's" - root-caused and fixed in commit 54899ec
+//     (oleks/mempalace#87/#90). The same class of bug had already bitten
+//     this exact idiom once before (commit e409e70, oleks/mempalace#68:
+//     "test_mcp_http_transport.py's" in a comment) - which is exactly why
+//     it's worth detecting statically rather than trusting people to
+//     remember not to use apostrophes in comments forever.
+//
+//  2. UnescapedWoodpeckerVar: Woodpecker substitutes braced `${VAR}`
+//     references in the raw pipeline YAML text itself, at parse time,
+//     before the shell ever runs - including inside comments, since the
+//     substitution pass has no idea what's a comment. A shell variable
+//     meant for the shell (e.g. `${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}`)
+//     gets mangled (undefined -> empty/malformed text), typically
+//     surfacing as an "unbound variable" failure under `set -u`. The fix
+//     is to escape it as `$$` so Woodpecker's substitution collapses that
+//     to a literal `$`, leaving `${VAR}` intact for the shell - see
+//     oleks/mempalace commit 33b880d (oleks/mempalace#44). Woodpecker's
+//     own builtin metadata vars (`${CI_*}`) are the one legitimate
+//     unescaped use and are not flagged.
+package analyze
+
+import (
+	"regexp"
+	"sort"
+	"strings"
+
+	"git.oleks.space/oleks/pipetree/internal/model"
+)
+
+// StrayApostrophe is one line inside a `-c '...'`-style single-quoted
+// shell block whose apostrophe isn't the block's own close-escape-reopen
+// idiom (a single quote, backslash, then two single quotes) - it closes
+// the quote early, corrupting everything after it in the script.
+type StrayApostrophe struct {
+	Project  string
+	Pipeline string
+	Step     string
+	Line     int    // 1-based line number within the command's text (not the file)
+	Context  string // the offending line, trimmed
+}
+
+// UnescapedWoodpeckerVar is a `${VAR}` in a step command that Woodpecker
+// will substitute at parse time (before the shell runs) because it isn't
+// escaped as `$${VAR}` and isn't one of Woodpecker's own `CI_*` builtins.
+type UnescapedWoodpeckerVar struct {
+	Project  string
+	Pipeline string
+	Step     string
+	Line     int
+	Var      string
+	Context  string
+}
+
+// openQuoteRe matches a line (right-trimmed) ending in a `-c '`/`-euxc '`
+// style flag that opens a single-quoted shell argument - the idiom is
+// `... -c bash -euxc '` as the last thing on the line, with the script
+// body starting on the next line.
+var openQuoteRe = regexp.MustCompile(`-\S*c\s+'$`)
+
+// dollarBraceRe matches `${NAME...}` - NAME must start with a letter or
+// underscore (excludes positional/special params like ${1} or ${@}, which
+// Woodpecker's metadata substitution doesn't touch anyway) and may be
+// followed by any parameter-expansion suffix (`:-default`, `:+alt`, ...).
+var dollarBraceRe = regexp.MustCompile(`\$\{([A-Za-z_][A-Za-z0-9_]*)[^}]*\}`)
+
+// woodpeckerBuiltinVar reports whether name is one of Woodpecker's own
+// substituted variables, for which an unescaped `${...}` is the correct,
+// intended usage rather than a footgun:
+//   - namespaced CI_* metadata vars (pipeline/frontend/metadata package);
+//   - this pipeline's own `matrix:` axis names - Woodpecker's matrix
+//     feature works BY replacing `${AXIS_NAME}` throughout the YAML text
+//     with each axis value before parsing, so e.g. `matrix: {TARGET_ARCH:
+//     [...]}` + `${TARGET_ARCH}` in a step name/command is the documented,
+//     correct way to use it, not a mistake (confirmed against real fleet
+//     pipelines: oleks/ii-researcher, oleks/csi-s3, oleks/common-chronicle
+//     all use this pattern deliberately).
+func woodpeckerBuiltinVar(name string, matrixVars map[string]bool) bool {
+	return strings.HasPrefix(name, "CI_") || matrixVars[name]
+}
+
+// findStrayApostrophes scans one command string for the stray-apostrophe
+// footgun. It looks for the `-c '` idiom's opening line, then the nearest
+// following line that is - once trimmed - exactly `'` (the idiom's own
+// dedicated closing line). Any additional, non-escaped `'` strictly
+// between those two lines closes the shell's quote early. Commands with no
+// such dedicated closing line are left alone rather than guessed at, to
+// avoid false positives on differently-shaped single-quoted commands.
+func findStrayApostrophes(cmd string) []struct {
+	Line    int
+	Context string
+} {
+	var hits []struct {
+		Line    int
+		Context string
+	}
+	lines := strings.Split(cmd, "\n")
+	for i, raw := range lines {
+		if !openQuoteRe.MatchString(strings.TrimRight(raw, " \t\r")) {
+			continue
+		}
+		closeIdx := -1
+		for j := i + 1; j < len(lines); j++ {
+			if strings.TrimSpace(lines[j]) == "'" {
+				closeIdx = j
+				break
+			}
+		}
+		if closeIdx == -1 {
+			// No dedicated closing line found for this opening - the
+			// script doesn't match the idiom precisely enough to check
+			// reliably, so skip rather than risk a false positive.
+			continue
+		}
+		for k := i + 1; k < closeIdx; k++ {
+			// Strip the legitimate `'\''` close-escape-reopen idiom before
+			// looking for a stray quote.
+			stripped := strings.ReplaceAll(lines[k], `'\''`, "")
+			if strings.Contains(stripped, "'") {
+				hits = append(hits, struct {
+					Line    int
+					Context string
+				}{Line: k + 1, Context: strings.TrimSpace(lines[k])})
+			}
+		}
+	}
+	return hits
+}
+
+// findUnescapedVars scans one command string for `${VAR}` references that
+// Woodpecker will substitute at parse time: any `${...}` not immediately
+// preceded by a second `$` (the `$${...}` escape that hands the expansion
+// to the shell) and whose var name isn't one of Woodpecker's own `CI_*`
+// builtins or this pipeline's own matrix axis names (see
+// woodpeckerBuiltinVar). matrixVars may be nil.
+func findUnescapedVars(cmd string, matrixVars map[string]bool) []struct {
+	Line    int
+	Var     string
+	Context string
+} {
+	var hits []struct {
+		Line    int
+		Var     string
+		Context string
+	}
+	lines := strings.Split(cmd, "\n")
+	for i, line := range lines {
+		trimmed := strings.TrimSpace(line)
+		for _, loc := range dollarBraceRe.FindAllStringSubmatchIndex(line, -1) {
+			start := loc[0]
+			name := line[loc[2]:loc[3]]
+			if start > 0 && line[start-1] == '$' {
+				continue // $${...} - correctly escaped for the shell
+			}
+			if woodpeckerBuiltinVar(name, matrixVars) {
+				continue // Woodpecker's own builtin metadata or matrix var
+			}
+			hits = append(hits, struct {
+				Line    int
+				Var     string
+				Context string
+			}{Line: i + 1, Var: name, Context: trimmed})
+		}
+	}
+	return hits
+}
+
+// runFootguns appends the two footgun checks' findings (for one project's
+// pipelines) onto the accumulators passed in.
+func runFootguns(proj *model.Project, apos *[]StrayApostrophe, vars *[]UnescapedWoodpeckerVar) {
+	for _, pl := range proj.Pipelines {
+		var matrixVars map[string]bool
+		if len(pl.Matrix) > 0 {
+			matrixVars = make(map[string]bool, len(pl.Matrix))
+			for axis := range pl.Matrix {
+				matrixVars[axis] = true
+			}
+		}
+		for _, s := range pl.Steps {
+			for _, cmd := range s.Commands {
+				for _, h := range findStrayApostrophes(cmd) {
+					*apos = append(*apos, StrayApostrophe{
+						Project: proj.Name, Pipeline: pl.Name, Step: s.Name,
+						Line: h.Line, Context: h.Context,
+					})
+				}
+				for _, h := range findUnescapedVars(cmd, matrixVars) {
+					*vars = append(*vars, UnescapedWoodpeckerVar{
+						Project: proj.Name, Pipeline: pl.Name, Step: s.Name,
+						Line: h.Line, Var: h.Var, Context: h.Context,
+					})
+				}
+			}
+		}
+	}
+}
+
+func sortFootguns(apos []StrayApostrophe, vars []UnescapedWoodpeckerVar) {
+	sort.Slice(apos, func(i, j int) bool {
+		if apos[i].Project != apos[j].Project {
+			return apos[i].Project < apos[j].Project
+		}
+		if apos[i].Pipeline != apos[j].Pipeline {
+			return apos[i].Pipeline < apos[j].Pipeline
+		}
+		if apos[i].Step != apos[j].Step {
+			return apos[i].Step < apos[j].Step
+		}
+		return apos[i].Line < apos[j].Line
+	})
+	sort.Slice(vars, func(i, j int) bool {
+		if vars[i].Project != vars[j].Project {
+			return vars[i].Project < vars[j].Project
+		}
+		if vars[i].Pipeline != vars[j].Pipeline {
+			return vars[i].Pipeline < vars[j].Pipeline
+		}
+		if vars[i].Step != vars[j].Step {
+			return vars[i].Step < vars[j].Step
+		}
+		return vars[i].Line < vars[j].Line
+	})
+}

internal/analyze/footguns_test.go

@@ -0,0 +1,235 @@
+package analyze
+
+import (
+	"strings"
+	"testing"
+
+	"git.oleks.space/oleks/pipetree/internal/model"
+	"git.oleks.space/oleks/pipetree/internal/parse"
+)
+
+// loadProject parses one pipeline file (a testdata fixture) into a
+// single-pipeline model.Project, the same shape buildProjects (main.go)
+// produces for a real scan.
+func loadProject(t *testing.T, name, file string) *model.Project {
+	t.Helper()
+	pl, err := parse.File(file)
+	if err != nil {
+		t.Fatalf("parse.File(%s): %v", file, err)
+	}
+	pl.Name = "default"
+	return &model.Project{Name: name, Path: file, Pipelines: []*model.Pipeline{pl}}
+}
+
+// TestStrayApostrophe_PositiveCase reproduces the exact shape of the bug
+// that broke oleks/mempalace pipeline 218 on 2026-08-03 (commit 54899ec^,
+// fixed by 54899ec, oleks/mempalace#87/#90): a comment inside the
+// `-c bash -euxc '...'` block containing the word "Woodpecker's" closes
+// the quote early, dropping the rest of the script to the outer shell -
+// the pipeline failed with a baffling "patch: command not found" (exit
+// 127) even though patch was installed, just not on the fallback shell's
+// PATH. Verifies pipetree catches it.
+func TestStrayApostrophe_PositiveCase(t *testing.T) {
+	proj := loadProject(t, "mempalace", "testdata/mempalace-stray-apostrophe.yaml")
+	report := Run([]*model.Project{proj})
+
+	if len(report.StrayApostrophes) == 0 {
+		t.Fatal("expected at least one stray apostrophe finding, got none")
+	}
+	found := false
+	for _, a := range report.StrayApostrophes {
+		if a.Step != "pytest-ruff" {
+			t.Errorf("finding on unexpected step %q", a.Step)
+		}
+		found = found || strings.Contains(a.Context, "Woodpecker's")
+	}
+	if !found {
+		t.Errorf("expected a finding pointing at the offending line (\"...Woodpecker's...\"), got: %+v", report.StrayApostrophes)
+	}
+}
+
+// TestUnescapedVar_PositiveCase reproduces the shape of the bug fixed in
+// oleks/mempalace commit 33b880d (oleks/mempalace#44): an unescaped
+// `${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}` that Woodpecker substitutes at
+// parse time, before the shell runs. The same file also legitimately uses
+// `${CI_COMMIT_TAG:-v3.5.0}` (a real Woodpecker builtin), which must NOT
+// be flagged.
+func TestUnescapedVar_PositiveCase(t *testing.T) {
+	proj := loadProject(t, "mempalace", "testdata/mempalace-unescaped-var.yaml")
+	report := Run([]*model.Project{proj})
+
+	var gotLDLibraryPath, gotCIBuiltin bool
+	for _, u := range report.UnescapedVars {
+		if u.Var == "LD_LIBRARY_PATH" {
+			gotLDLibraryPath = true
+		}
+		if u.Var == "CI_COMMIT_TAG" {
+			gotCIBuiltin = true
+		}
+	}
+	if !gotLDLibraryPath {
+		t.Errorf("expected LD_LIBRARY_PATH to be flagged as an unescaped shell var, got: %+v", report.UnescapedVars)
+	}
+	if gotCIBuiltin {
+		t.Errorf("CI_COMMIT_TAG is a legitimate Woodpecker builtin and must not be flagged, got: %+v", report.UnescapedVars)
+	}
+	// This fixture's single-quoted shell block has no apostrophe footgun.
+	if len(report.StrayApostrophes) != 0 {
+		t.Errorf("expected no stray-apostrophe findings in this fixture, got: %+v", report.StrayApostrophes)
+	}
+}
+
+// TestFootguns_NegativeCase_KnownGoodFixture is the current (post-fix),
+// live oleks/mempalace .woodpecker/test.yaml: correctly-escaped $${...}
+// var references, a legitimate ${CI_COMMIT_TAG} builtin use outside any
+// single-quoted block, and an apostrophe-free -c bash -euxc '...' block.
+// Regression guard for false positives - this exact file was flagged as
+// "known-good post-fix" and a hit here would need fixing before shipping.
+func TestFootguns_NegativeCase_KnownGoodFixture(t *testing.T) {
+	proj := loadProject(t, "mempalace", "testdata/mempalace-fixed.yaml")
+	report := Run([]*model.Project{proj})
+
+	if len(report.StrayApostrophes) != 0 {
+		t.Errorf("false positive: expected no stray-apostrophe findings on the known-good fixture, got: %+v", report.StrayApostrophes)
+	}
+	if len(report.UnescapedVars) != 0 {
+		t.Errorf("false positive: expected no unescaped-var findings on the known-good fixture, got: %+v", report.UnescapedVars)
+	}
+}
+
+// TestFootguns_NegativeCase_MatrixVar guards against the false positive
+// found while validating against the real fleet: oleks/ii-researcher (and
+// equivalently oleks/csi-s3, oleks/common-chronicle) declares `matrix:
+// {TARGET_ARCH: [...]}` and legitimately references `${TARGET_ARCH}`
+// unescaped in the step name and commands - that's Woodpecker's own
+// matrix-substitution feature working as intended, not a footgun, and
+// must not be flagged.
+func TestFootguns_NegativeCase_MatrixVar(t *testing.T) {
+	proj := loadProject(t, "ii-researcher", "testdata/matrix-fixture.yaml")
+	report := Run([]*model.Project{proj})
+
+	for _, u := range report.UnescapedVars {
+		if u.Var == "TARGET_ARCH" {
+			t.Errorf("false positive: TARGET_ARCH is this pipeline's own matrix axis, must not be flagged: %+v", u)
+		}
+	}
+}
+
+// TestFindStrayApostrophes_Unit exercises the line-level detector directly
+// against small synthetic commands, independent of YAML parsing.
+func TestFindStrayApostrophes_Unit(t *testing.T) {
+	cases := []struct {
+		name string
+		cmd  string
+		want int
+	}{
+		{
+			name: "clean block, no apostrophe",
+			cmd: "nix shell nixpkgs#bash -c bash -euxc '\n" +
+				"  echo hello\n" +
+				"  echo world\n" +
+				"'",
+			want: 0,
+		},
+		{
+			name: "apostrophe in a comment ends the quote early",
+			cmd: "nix shell nixpkgs#bash -c bash -euxc '\n" +
+				"  # this is Woodpecker's escape hatch\n" +
+				"  echo hello\n" +
+				"'",
+			want: 1,
+		},
+		{
+			name: "legitimate close-escape-reopen idiom is not flagged",
+			cmd: "nix shell nixpkgs#bash -c bash -euxc '\n" +
+				"  echo '\\''it'\\''s fine'\\''\n" +
+				"'",
+			want: 0,
+		},
+		{
+			name: "no dedicated closing line - not checked (avoid false positive)",
+			cmd:  "nix shell nixpkgs#bash -c bash -euxc 'echo hi'",
+			want: 0,
+		},
+		{
+			name: "no single-quoted -c block at all",
+			cmd:  "echo \"it's a plain command, no -c '...' idiom here\"",
+			want: 0,
+		},
+	}
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			got := findStrayApostrophes(tc.cmd)
+			if len(got) != tc.want {
+				t.Errorf("findStrayApostrophes(%q) = %d hits %+v, want %d", tc.cmd, len(got), got, tc.want)
+			}
+		})
+	}
+}
+
+// TestFindUnescapedVars_Unit exercises the ${VAR} detector directly.
+func TestFindUnescapedVars_Unit(t *testing.T) {
+	cases := []struct {
+		name       string
+		cmd        string
+		matrixVars map[string]bool
+		want       []string
+	}{
+		{
+			name: "correctly escaped shell var",
+			cmd:  `export LD_LIBRARY_PATH="$CCLIB/lib$${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"`,
+			want: nil,
+		},
+		{
+			name: "unescaped shell var",
+			cmd:  `export LD_LIBRARY_PATH="$CCLIB/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"`,
+			want: []string{"LD_LIBRARY_PATH"},
+		},
+		{
+			name: "legitimate unescaped Woodpecker builtin",
+			cmd:  `VERSION=$(printf "%s" "${CI_COMMIT_TAG:-v3.5.0}")`,
+			want: nil,
+		},
+		{
+			name: "unescaped var mentioned inside a comment still bites",
+			cmd:  "# don't forget ${MY_SECRET} needs escaping\necho hi",
+			want: []string{"MY_SECRET"},
+		},
+		{
+			name: "bare $VAR and $(...) are unaffected",
+			cmd:  `echo $HOME && VER=$(date +%s)`,
+			want: nil,
+		},
+		{
+			// Woodpecker's matrix feature works BY substituting ${AXIS}
+			// throughout the YAML with each axis value before parsing -
+			// confirmed live in oleks/ii-researcher, oleks/csi-s3,
+			// oleks/common-chronicle. This is the correct, intended use,
+			// not a footgun, so a pipeline's own matrix axis names must
+			// not be flagged.
+			name:       "matrix axis var is not a footgun",
+			cmd:        `ci/local.sh --arch "${TARGET_ARCH}"`,
+			matrixVars: map[string]bool{"TARGET_ARCH": true},
+			want:       nil,
+		},
+		{
+			name:       "non-matrix var still flagged even with a matrix in scope",
+			cmd:        `echo "${OTHER_VAR}"`,
+			matrixVars: map[string]bool{"TARGET_ARCH": true},
+			want:       []string{"OTHER_VAR"},
+		},
+	}
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			got := findUnescapedVars(tc.cmd, tc.matrixVars)
+			if len(got) != len(tc.want) {
+				t.Fatalf("findUnescapedVars(%q) = %+v, want vars %v", tc.cmd, got, tc.want)
+			}
+			for i, w := range tc.want {
+				if got[i].Var != w {
+					t.Errorf("hit %d: var = %q, want %q", i, got[i].Var, w)
+				}
+			}
+		})
+	}
+}

internal/analyze/testdata/matrix-fixture.yaml

@@ -0,0 +1,78 @@
+labels:
+  arch: amd64
+
+clone:
+  - name: clone
+    image: woodpeckerci/plugin-git
+    environment:
+      CI_NETRC_MACHINE: git.oleks.space
+      CI_NETRC_USERNAME: oleks
+      CI_NETRC_PASSWORD:
+        from_secret: gitea_clone_token
+      PLUGIN_TAGS: "false"
+      PLUGIN_DEPTH: "1"
+
+# DISABLED: replaced by .woodpecker/arm64.yaml and .woodpecker/amd64.yaml
+# which run on native arch nodes.
+when:
+  - event: manual
+    evaluate: "false"
+
+# Matrix: uncomment additional arches to enable multi-arch builds
+# Remote buildkit workers do the native-arch build per target.
+matrix:
+  TARGET_ARCH:
+    # - s390x
+    - arm64
+    # - amd64
+
+steps:
+  - name: build-api-${TARGET_ARCH}
+    image: git.oleks.space/oleks/nix-ci:latest
+    environment:
+      REGISTRY_TOKEN:
+        from_secret: registry_token
+    commands:
+      - echo "▸ arch=$(uname -m)"
+      - |
+        TAG=$(echo "$CI_COMMIT_TAG" | sed 's/^v//')
+        CHECK=$(curl -sf -H "Authorization: token $REGISTRY_TOKEN" \
+          "https://git.oleks.space/api/v1/packages/oleks?type=container&q=ii-researcher/api&limit=50" || echo "")
+        if echo "$CHECK" | grep -q "\"version\":\"$TAG-${TARGET_ARCH}\""; then
+          echo "Image ii-researcher/api:$TAG-${TARGET_ARCH} already exists, skipping build"
+          exit 0
+        fi
+      - echo "$REGISTRY_TOKEN" | docker login git.oleks.space -u oleks --password-stdin
+      - |
+        BUILDER_HOST="buildkit-rootless-${TARGET_ARCH}.infra.svc.cluster.local"
+        BUILDER_PORT="1234"
+        echo "Waiting for builder..."
+        for i in $(seq 1 30); do
+          if echo >/dev/tcp/$BUILDER_HOST/$BUILDER_PORT 2>/dev/null; then
+            echo "Builder ready"; break
+          fi
+          [ "$i" -eq 30 ] && echo "Builder not available" && exit 1
+          sleep 10
+        done
+        docker buildx create --name ${TARGET_ARCH}-api --driver remote "tcp://$BUILDER_HOST:$BUILDER_PORT"
+        TAG=$(echo "$CI_COMMIT_TAG" | sed 's/^v//')
+        docker buildx build \
+          --builder ${TARGET_ARCH}-api \
+          --platform linux/${TARGET_ARCH} \
+          --build-arg REGISTRY_TOKEN="$REGISTRY_TOKEN" \
+          -f Dockerfile.api \
+          -t "git.oleks.space/oleks/ii-researcher/api:$TAG-${TARGET_ARCH}" \
+          --push .
+    backend_options:
+      kubernetes:
+        nodeSelector:
+          provider: digitalocean
+        resources:
+          requests:
+            memory: 6Gi
+          limits:
+            memory: 6Gi
+        labels:
+          commit-tag: "${CI_COMMIT_TAG}"
+          commit-branch: "${CI_COMMIT_BRANCH}"
+          pipeline-number: "${CI_PIPELINE_NUMBER}"

internal/analyze/testdata/mempalace-fixed.yaml

@@ -0,0 +1,135 @@
+# CI gate for the patched mempalace tree (oleks/mempalace#44).
+#
+# The image build applies an ordered stack of downstream patches to the upstream
+# v3.5.0 tarball (see flake.nix `patches = [ ... ]`), but the build itself does
+# NOT run the test suite — so a behavior patch that breaks an upstream test, or
+# pushes a function over ruff's C901 budget, would ship silently. This job
+# reconstructs the exact patched tree and runs pytest + ruff so patch drift
+# fails CI on push/PR, BEFORE a release tag builds an image.
+#
+# DRY: the patch list + order are read straight out of flake.nix (the single
+# source of truth), so this can never fall out of sync with what the image builds.
+# arch: arm64 — mempalace is arm64-only; an amd64 label mis-scheduled this onto
+# agents that report labels:null and the long clone got context-canceled
+# (oleks/mempalace#41 CI clone-cancel root cause).
+labels:
+  arch: arm64
+
+# push+pull_request together double-run every commit on a branch with an open
+# PR (oleks/cluster#364's trigger case) -- push is scoped to main (post-merge
+# validation) so pull_request alone covers feature branches.
+when:
+  - event: pull_request
+  - event: push
+    branch: main
+  - event: manual
+
+steps:
+  - name: pytest-ruff
+    image: git.oleks.space/oleks/nix-ci:latest
+    environment:
+      # ci/setup.sh writes a git.oleks.space netrc from this — without it,
+      # evaluating this flake's private git+https inputs (fleet-pins,
+      # heatwave-backend, parity-lib) to resolve `.#devShells.<system>.test`
+      # fails the same way flake-check.yaml's GITEA_CLONE_TOKEN comment
+      # describes (oleks/mempalace#57).
+      GITEA_CLONE_TOKEN:
+        from_secret: gitea_clone_token
+    commands:
+      - echo "▸ arch=$(uname -m)"
+      - sh ci/setup.sh
+      # `nix develop .#test` now evaluates this repo's own flake.nix (unlike the
+      # old bare `nix shell nixpkgs#...`, which never touched it), and flake.nix
+      # unconditionally `import`s the gitignored version.nix — so it has to exist
+      # (and be `git add -f`'d, matching flake-check.yaml/arm64.yaml: a local git
+      # flake's pure eval only sees git-tracked/staged files) before any output
+      # can be evaluated, even one that doesn't otherwise care about VERSION.
+      - VERSION=$(echo "${CI_COMMIT_TAG:-v3.5.0}" | sed 's/^v//; s/-[0-9]*$//')
+      - printf '"%s"\n' "$VERSION" > version.nix && git add -f version.nix
+      # oleks/mempalace#68/#73/#325 fix: narrow the test shell to lightweight direct
+      # deps only (uv, ruff, python3, curl, tar, gzip, sed, grep, patch, coreutils),
+      # pinned to THIS repo's nixpkgs (fleet/nixpkgs-ci) via flake.nix's input pins.
+      # This avoids evaluating the full flake.nix (which pulls in image-build
+      # infrastructure + deep transitive closures), and restores the fast,
+      # reproducible cache locality from pre-#73. The #68 PATH fix (pinned uv/ruff)
+      # is preserved: nix shell resolves them against the fleet pin, not live
+      # nixpkgs-unstable. oleks/cluster#325 (slow attic cache) no longer dominates
+      # the setup phase because we're not pulling full image-build closures anymore.
+      # Resolved out here, where `nix` is still on PATH (inside `nix shell` it is
+      # not). Exported so the apostrophe-free block below can consume it.
+      - export CXX_LIB_DIR="$(nix eval --raw nixpkgs#stdenv.cc.cc.lib.outPath)/lib"
+      - |
+        # Read nixpkgs from flake.lock to ensure we use the fleet-pinned version,
+        # matching what the full flake would use. Avoids nixpkgs-unstable drift.
+        nix shell \
+          "nixpkgs#uv" \
+          "nixpkgs#ruff" \
+          "nixpkgs#python312" \
+          "nixpkgs#stdenv.cc.cc.lib" \
+          "nixpkgs#curl" \
+          "nixpkgs#gnutar" \
+          "nixpkgs#gzip" \
+          "nixpkgs#gnused" \
+          "nixpkgs#gnugrep" \
+          "nixpkgs#patch" \
+          "nixpkgs#coreutils" \
+          -c bash -euxc '
+            # Explicit bash, not the base images bare /bin/sh (dash/busybox,
+            # unknown which) — see oleks/mempalace#68 for the debugging this
+            # replaced. All tools above come from the pinned nixpkgs, so
+            # they are guaranteed present on PATH without a retry/poll loop.
+            UV_BIN=$(command -v uv)
+            RUFF_BIN=$(command -v ruff)
+            echo "resolved: uv=$UV_BIN ruff=$RUFF_BIN"
+            # Use the nix-provided CPython, NOT a uv-downloaded standalone one: the
+            # nix-ci image has no nix-ld, so uv-fetched interpreters cannot find their
+            # dynamic loader ("Python interpreter not found"). only-system + never
+            # makes uv build the venv from python3.12 on PATH.
+            export UV_PYTHON_DOWNLOADS=never UV_PYTHON_PREFERENCE=only-system
+
+            # numpy/chromadb arrive as manylinux wheels that dlopen libstdc++.so.6,
+            # which the nix-ci image does not provide (no nix-ld either). devShells.test
+            # sets this, but a6f9b5c stopped CI from using that shell when it narrowed
+            # the step to a bare `nix shell` for cache locality (#73/#325) -- and dropped
+            # stdenv.cc.cc.lib with it, regressing the #190/PR#77 fix (88cfa1b) and
+            # leaving main red (oleks/mempalace#87/#90). Re-added here rather than
+            # reverting to `nix develop .#test`, which would pull back the image-build
+            # closures a6f9b5c deliberately removed.
+            #
+            # NOTE: this whole block is a single-quoted `bash -euxc ...` argument, so
+            # an apostrophe anywhere in it -- including in a comment -- terminates the
+            # quote early and silently drops the rest of the script back to /bin/sh
+            # (seen live: pipeline 218 failed with "patch: command not found").
+            # Keep this block apostrophe-free.
+            #
+            # LIB_DIR is resolved OUTSIDE this shell and passed in via the environment:
+            # `nix shell` puts only the listed packages on PATH, and nix itself is not
+            # one of them, so `nix eval` in here would be command-not-found.
+            export LD_LIBRARY_PATH="$${CXX_LIB_DIR}$${LD_LIBRARY_PATH:+:$$LD_LIBRARY_PATH}"
+
+            # version.nix is gitignored; take it from the release tag when present
+            # (v3.5.0-7 -> 3.5.0), else the current default (bump with version.nix).
+            VERSION=$(printf "%s" "$${CI_COMMIT_TAG:-v3.5.0}" | sed "s/^v//; s/-[0-9]*$//")
+            echo "mempalace version=$VERSION"
+
+            curl -fsSL "https://github.com/milla-jovovich/mempalace/archive/refs/tags/v$VERSION.tar.gz" | tar xz
+            cd "mempalace-$VERSION"
+
+            # Apply the patch stack in the EXACT order flake.nix declares it,
+            # parsed from the `patches = [ ... ];` block (no hand-maintained copy).
+            PATCHES=$(sed -n "/patches = \[/,/\];/p" ../flake.nix | grep -oE "mempalace-[a-z0-9-]+\.patch")
+            echo "applying:"; echo "$PATCHES"
+            for p in $PATCHES; do echo "+ $p"; patch -p1 < "../$p"; done
+
+            # uv pulls the test deps (real chromadb, prometheus_client, pyyaml,
+            # ruff) into an ephemeral env; the package imports from the patched
+            # tree via PYTHONPATH — same invocation a developer runs locally.
+            # mcp/starlette/uvicorn/httpx (oleks/mempalace#27): the streamable-HTTP
+            # transport test file exercises the real SDK client/server over a real
+            # loopback socket, matching the real-socket approach the legacy HTTP
+            # transport test file (test_mcp_http_transport.py) already uses.
+            PYTHONPATH=. "$UV_BIN" run --python python3.12 --with pytest --with chromadb --with pyyaml --with prometheus_client \
+              --with "mcp>=1.29,<2" --with starlette --with uvicorn --with httpx \
+              pytest tests/test_mcp_server.py tests/test_mcp_streamable_http.py -q -p no:cacheprovider
+            "$RUFF_BIN" check mempalace/mcp_server.py mempalace/mcp_streamable_http.py
+          '

internal/analyze/testdata/mempalace-stray-apostrophe.yaml

@@ -0,0 +1,123 @@
+# CI gate for the patched mempalace tree (oleks/mempalace#44).
+#
+# The image build applies an ordered stack of downstream patches to the upstream
+# v3.5.0 tarball (see flake.nix `patches = [ ... ]`), but the build itself does
+# NOT run the test suite — so a behavior patch that breaks an upstream test, or
+# pushes a function over ruff's C901 budget, would ship silently. This job
+# reconstructs the exact patched tree and runs pytest + ruff so patch drift
+# fails CI on push/PR, BEFORE a release tag builds an image.
+#
+# DRY: the patch list + order are read straight out of flake.nix (the single
+# source of truth), so this can never fall out of sync with what the image builds.
+# arch: arm64 — mempalace is arm64-only; an amd64 label mis-scheduled this onto
+# agents that report labels:null and the long clone got context-canceled
+# (oleks/mempalace#41 CI clone-cancel root cause).
+labels:
+  arch: arm64
+
+# push+pull_request together double-run every commit on a branch with an open
+# PR (oleks/cluster#364's trigger case) -- push is scoped to main (post-merge
+# validation) so pull_request alone covers feature branches.
+when:
+  - event: pull_request
+  - event: push
+    branch: main
+  - event: manual
+
+steps:
+  - name: pytest-ruff
+    image: git.oleks.space/oleks/nix-ci:latest
+    environment:
+      # ci/setup.sh writes a git.oleks.space netrc from this — without it,
+      # evaluating this flake's private git+https inputs (fleet-pins,
+      # heatwave-backend, parity-lib) to resolve `.#devShells.<system>.test`
+      # fails the same way flake-check.yaml's GITEA_CLONE_TOKEN comment
+      # describes (oleks/mempalace#57).
+      GITEA_CLONE_TOKEN:
+        from_secret: gitea_clone_token
+    commands:
+      - echo "▸ arch=$(uname -m)"
+      - sh ci/setup.sh
+      # `nix develop .#test` now evaluates this repo's own flake.nix (unlike the
+      # old bare `nix shell nixpkgs#...`, which never touched it), and flake.nix
+      # unconditionally `import`s the gitignored version.nix — so it has to exist
+      # (and be `git add -f`'d, matching flake-check.yaml/arm64.yaml: a local git
+      # flake's pure eval only sees git-tracked/staged files) before any output
+      # can be evaluated, even one that doesn't otherwise care about VERSION.
+      - VERSION=$(echo "${CI_COMMIT_TAG:-v3.5.0}" | sed 's/^v//; s/-[0-9]*$//')
+      - printf '"%s"\n' "$VERSION" > version.nix && git add -f version.nix
+      # oleks/mempalace#68/#73/#325 fix: narrow the test shell to lightweight direct
+      # deps only (uv, ruff, python3, curl, tar, gzip, sed, grep, patch, coreutils),
+      # pinned to THIS repo's nixpkgs (fleet/nixpkgs-ci) via flake.nix's input pins.
+      # This avoids evaluating the full flake.nix (which pulls in image-build
+      # infrastructure + deep transitive closures), and restores the fast,
+      # reproducible cache locality from pre-#73. The #68 PATH fix (pinned uv/ruff)
+      # is preserved: nix shell resolves them against the fleet pin, not live
+      # nixpkgs-unstable. oleks/cluster#325 (slow attic cache) no longer dominates
+      # the setup phase because we're not pulling full image-build closures anymore.
+      - |
+        # Read nixpkgs from flake.lock to ensure we use the fleet-pinned version,
+        # matching what the full flake would use. Avoids nixpkgs-unstable drift.
+        nix shell \
+          "nixpkgs#uv" \
+          "nixpkgs#ruff" \
+          "nixpkgs#python312" \
+          "nixpkgs#stdenv.cc.cc.lib" \
+          "nixpkgs#curl" \
+          "nixpkgs#gnutar" \
+          "nixpkgs#gzip" \
+          "nixpkgs#gnused" \
+          "nixpkgs#gnugrep" \
+          "nixpkgs#patch" \
+          "nixpkgs#coreutils" \
+          -c bash -euxc '
+            # Explicit bash, not the base images bare /bin/sh (dash/busybox,
+            # unknown which) — see oleks/mempalace#68 for the debugging this
+            # replaced. All tools above come from the pinned nixpkgs, so
+            # they are guaranteed present on PATH without a retry/poll loop.
+            UV_BIN=$(command -v uv)
+            RUFF_BIN=$(command -v ruff)
+            echo "resolved: uv=$UV_BIN ruff=$RUFF_BIN"
+            # Use the nix-provided CPython, NOT a uv-downloaded standalone one: the
+            # nix-ci image has no nix-ld, so uv-fetched interpreters cannot find their
+            # dynamic loader ("Python interpreter not found"). only-system + never
+            # makes uv build the venv from python3.12 on PATH.
+            export UV_PYTHON_DOWNLOADS=never UV_PYTHON_PREFERENCE=only-system
+
+            # numpy/chromadb arrive as manylinux wheels that dlopen libstdc++.so.6,
+            # which the nix-ci image does not provide (no nix-ld either). devShells.test
+            # sets this, but a6f9b5c stopped CI from using that shell when it narrowed
+            # the step to a bare `nix shell` for cache locality (#73/#325) -- and dropped
+            # stdenv.cc.cc.lib with it, regressing the #190/PR#77 fix (88cfa1b) and
+            # leaving main red (oleks/mempalace#87/#90). Re-add it here rather than
+            # reverting to `nix develop .#test`, which would pull the image-build
+            # closures a6f9b5c deliberately removed. $$ escapes Woodpecker's own
+            # substitution so the shell owns the expansion (same as 33b880d).
+            export LD_LIBRARY_PATH="$$(nix eval --raw nixpkgs#stdenv.cc.cc.lib.outPath)/lib$${LD_LIBRARY_PATH:+:$$LD_LIBRARY_PATH}"
+
+            # version.nix is gitignored; take it from the release tag when present
+            # (v3.5.0-7 -> 3.5.0), else the current default (bump with version.nix).
+            VERSION=$(printf "%s" "$${CI_COMMIT_TAG:-v3.5.0}" | sed "s/^v//; s/-[0-9]*$//")
+            echo "mempalace version=$VERSION"
+
+            curl -fsSL "https://github.com/milla-jovovich/mempalace/archive/refs/tags/v$VERSION.tar.gz" | tar xz
+            cd "mempalace-$VERSION"
+
+            # Apply the patch stack in the EXACT order flake.nix declares it,
+            # parsed from the `patches = [ ... ];` block (no hand-maintained copy).
+            PATCHES=$(sed -n "/patches = \[/,/\];/p" ../flake.nix | grep -oE "mempalace-[a-z0-9-]+\.patch")
+            echo "applying:"; echo "$PATCHES"
+            for p in $PATCHES; do echo "+ $p"; patch -p1 < "../$p"; done
+
+            # uv pulls the test deps (real chromadb, prometheus_client, pyyaml,
+            # ruff) into an ephemeral env; the package imports from the patched
+            # tree via PYTHONPATH — same invocation a developer runs locally.
+            # mcp/starlette/uvicorn/httpx (oleks/mempalace#27): the streamable-HTTP
+            # transport test file exercises the real SDK client/server over a real
+            # loopback socket, matching the real-socket approach the legacy HTTP
+            # transport test file (test_mcp_http_transport.py) already uses.
+            PYTHONPATH=. "$UV_BIN" run --python python3.12 --with pytest --with chromadb --with pyyaml --with prometheus_client \
+              --with "mcp>=1.29,<2" --with starlette --with uvicorn --with httpx \
+              pytest tests/test_mcp_server.py tests/test_mcp_streamable_http.py -q -p no:cacheprovider
+            "$RUFF_BIN" check mempalace/mcp_server.py mempalace/mcp_streamable_http.py
+          '

internal/analyze/testdata/mempalace-unescaped-var.yaml

@@ -0,0 +1,63 @@
+# CI gate for the patched mempalace tree (oleks/mempalace#44).
+#
+# The image build applies an ordered stack of downstream patches to the upstream
+# v3.5.0 tarball (see flake.nix `patches = [ ... ]`), but the build itself does
+# NOT run the test suite — so a behavior patch that breaks an upstream test, or
+# pushes a function over ruff's C901 budget, would ship silently. This job
+# reconstructs the exact patched tree and runs pytest + ruff so patch drift
+# fails CI on push/PR, BEFORE a release tag builds an image.
+#
+# DRY: the patch list + order are read straight out of flake.nix (the single
+# source of truth), so this can never fall out of sync with what the image builds.
+# arch: arm64 — mempalace is arm64-only; an amd64 label mis-scheduled this onto
+# agents that report labels:null and the long clone got context-canceled
+# (oleks/mempalace#41 CI clone-cancel root cause).
+labels:
+  arch: arm64
+
+when:
+  - event: [push, pull_request, manual]
+
+steps:
+  - name: pytest-ruff
+    image: git.oleks.space/oleks/nix-ci:latest
+    commands:
+      - echo "▸ arch=$(uname -m)"
+      # Everything runs inside one nix shell so the exact tool set is guaranteed
+      # regardless of the base image contents.
+      - |
+        nix shell nixpkgs#uv nixpkgs#python312 nixpkgs#curl nixpkgs#gnutar nixpkgs#gzip \
+                  nixpkgs#gnused nixpkgs#gnugrep nixpkgs#patch nixpkgs#coreutils \
+          -c sh -euxc '
+            # Use the nix-provided CPython, NOT a uv-downloaded standalone one: the
+            # nix-ci image has no nix-ld, so uv-fetched interpreters cannot find their
+            # dynamic loader ("Python interpreter not found"). only-system + never
+            # makes uv build the venv from python3.12 on PATH.
+            export UV_PYTHON_DOWNLOADS=never UV_PYTHON_PREFERENCE=only-system
+            # manylinux wheel .so files (pydantic-core, chromadb deps) are dlopened by
+            # the interpreter and need libstdc++/libgcc on the loader path (no /usr/lib
+            # on NixOS).
+            CCLIB=$(nix build --no-link --print-out-paths nixpkgs#stdenv.cc.cc.lib)
+            export LD_LIBRARY_PATH="$CCLIB/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
+
+            # version.nix is gitignored; take it from the release tag when present
+            # (v3.5.0-7 -> 3.5.0), else the current default (bump with version.nix).
+            VERSION=$(printf "%s" "${CI_COMMIT_TAG:-v3.5.0}" | sed "s/^v//; s/-[0-9]*$//")
+            echo "mempalace version=$VERSION"
+
+            curl -fsSL "https://github.com/milla-jovovich/mempalace/archive/refs/tags/v$VERSION.tar.gz" | tar xz
+            cd "mempalace-$VERSION"
+
+            # Apply the patch stack in the EXACT order flake.nix declares it,
+            # parsed from the `patches = [ ... ];` block (no hand-maintained copy).
+            PATCHES=$(sed -n "/patches = \[/,/\];/p" ../flake.nix | grep -oE "mempalace-[a-z0-9-]+\.patch")
+            echo "applying:"; echo "$PATCHES"
+            for p in $PATCHES; do echo "+ $p"; patch -p1 < "../$p"; done
+
+            # uv pulls the test deps (real chromadb, prometheus_client, pyyaml,
+            # ruff) into an ephemeral env; the package imports from the patched
+            # tree via PYTHONPATH — same invocation a developer runs locally.
+            PYTHONPATH=. uv run --python python3.12 --with pytest --with chromadb --with pyyaml --with prometheus_client \
+              pytest tests/test_mcp_server.py -q -p no:cacheprovider
+            uv run --python python3.12 --with ruff ruff check mempalace/mcp_server.py
+          '

internal/render/analyze.go

@@ -26,4 +26,14 @@ func Analyze(w io.Writer, report analyze.Report) {
 	for _, b := range report.BrokenPipelineDependsOn {
 		_, _ = fmt.Fprintf(w, "  %s / %s depends_on %q (not found)\n", b.Project, b.Pipeline, b.MissingName)
 	}
+
+	_, _ = fmt.Fprintf(w, "\n%d stray apostrophe(s) inside a single-quoted shell block (closes the quote early, drops the rest of the script to the outer shell):\n", len(report.StrayApostrophes))
+	for _, a := range report.StrayApostrophes {
+		_, _ = fmt.Fprintf(w, "  %s / %s / step %s, line %d: %s\n", a.Project, a.Pipeline, a.Step, a.Line, a.Context)
+	}
+
+	_, _ = fmt.Fprintf(w, "\n%d unescaped ${VAR} in step commands (Woodpecker substitutes these at parse time, before the shell runs - escape as $${VAR} to hand it to the shell):\n", len(report.UnescapedVars))
+	for _, u := range report.UnescapedVars {
+		_, _ = fmt.Fprintf(w, "  %s / %s / step %s, line %d: ${%s} in %s\n", u.Project, u.Pipeline, u.Step, u.Line, u.Var, u.Context)
+	}
 }