Container-registry blob GC purged blobs of still-referenced image tags → dangling manifests / MANIFEST_UNKNOWN (2026-07 incident) #25

New Issue

2026-07-04T19:28:52+03:00

oleks commented

2026-07-04 19:28:52 +03:00

Incident (2026-07-03/04)

The Gitea container registry served dangling manifests for many image tags: the tag/manifest row existed but the underlying config/layer blobs were gone, so skopeo inspect (config fetch) and docker pull failed with MANIFEST_UNKNOWN / not found, while skopeo inspect --raw (manifest only) still "succeeded".

Confirmed simultaneously across unrelated repos:

oleks/mempalace: every pre-3.5.0 tag dead — 3.3.6, 3.3.6-arm64, 3.4.x-arm64, latest, latest-arm64 — only the newest 3.5.0-arm64 intact.
oleks/nix-ci: latest (amd64) dead → broke every downstream CI step (trying and failing to pull image).

This is a recurrence — the same failure hit mempalace:3.3.6-arm64 on 2026-06-25 (fixed then by delete-dangling-version + clean re-push; see the fleet HelmRelease comment / oleks/mempalace#12).

Impact

A bad mempalace deploy could not be rolled back (no pullable prior image) — forced a risky forward-fix with the memory backend down ~1 day.
Fleet-wide CI outage (nix-ci base image unpullable).
Only recovered because the running images were still cached in nodes' containerd.

Suspected cause

The registry/package GC (Gitea cron.cleanup_packages / container blob GC, or a retention rule) is removing blobs that are still referenced by live manifests — an orphaned-blob sweep that under-counts references (shared layers, per-arch sub-manifests of a multi-arch index, or <ver> vs <ver>-<arch> tags sharing blobs), or a retention policy pruning tags too aggressively.

Ask

Audit the container-registry GC / package-cleanup job (schedule + reference-counting logic) — it must not delete blobs reachable from any existing tag/manifest (including multi-arch index children).
Add tag/version retention protection for latest, the last N released versions, and any currently-deployed digest.
Consider verifying blob integrity after GC and alerting on dangling manifests.

Related: oleks/mempalace#18 (protect key tags / digest-pin fleet), oleks/mempalace#12 (prior instance), oleks/nixos-ci#2 (nix-ci:latest casualty), oleks/mempalace#14 (incident that exposed it). Gitea runs as the Nix fork on armer.

## Incident (2026-07-03/04) The Gitea container registry served **dangling manifests** for many image tags: the tag/manifest row existed but the underlying **config/layer blobs were gone**, so `skopeo inspect` (config fetch) and `docker pull` failed with `MANIFEST_UNKNOWN` / `not found`, while `skopeo inspect --raw` (manifest only) still "succeeded". Confirmed simultaneously across unrelated repos: - `oleks/mempalace`: **every** pre-3.5.0 tag dead — `3.3.6`, `3.3.6-arm64`, `3.4.x-arm64`, `latest`, `latest-arm64` — only the newest `3.5.0-arm64` intact. - `oleks/nix-ci`: `latest` (amd64) dead → broke every downstream CI step (`trying and failing to pull image`). This is a **recurrence** — the same failure hit `mempalace:3.3.6-arm64` on 2026-06-25 (fixed then by delete-dangling-version + clean re-push; see the fleet HelmRelease comment / oleks/mempalace#12). ## Impact - A bad mempalace deploy could **not be rolled back** (no pullable prior image) — forced a risky forward-fix with the memory backend down ~1 day. - Fleet-wide CI outage (nix-ci base image unpullable). - Only recovered because the running images were still cached in nodes' containerd. ## Suspected cause The registry/package GC (Gitea `cron.cleanup_packages` / container blob GC, or a retention rule) is removing blobs that are **still referenced by live manifests** — an orphaned-blob sweep that under-counts references (shared layers, per-arch sub-manifests of a multi-arch index, or `<ver>` vs `<ver>-<arch>` tags sharing blobs), or a retention policy pruning tags too aggressively. ## Ask - Audit the container-registry GC / package-cleanup job (schedule + reference-counting logic) — it must not delete blobs reachable from any existing tag/manifest (including multi-arch index children). - Add tag/version **retention protection** for `latest`, the last N released versions, and any currently-deployed digest. - Consider verifying blob integrity after GC and alerting on dangling manifests. Related: oleks/mempalace#18 (protect key tags / digest-pin fleet), oleks/mempalace#12 (prior instance), oleks/nixos-ci#2 (nix-ci:latest casualty), oleks/mempalace#14 (incident that exposed it). Gitea runs as the Nix fork on armer.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: oleks/gitea#25