ADR-0008: Immutable, verified, reversible deploys
- Status
- accepted (principles) · superseded (hosted pipeline)
- Decided
- 2026, for the original hosted deployment
- Applies to this repository
- The principles do; the hosted single-revision deploy pipeline is not part of this repository. Releases publish signed images and Helm charts to GHCR and you deploy them (see CI/CD Pipelines).
Context
The hosted deployment ran each service as a single active revision on a managed container platform. An early outage lasted about 40 minutes because no previous image existed to roll back to. The rules below came out of that.
Decision
- Immutable versions. Deploy a pinned version, never
latest. A re-push of the same tag does not reliably roll anything. - Core first, frontends after. Deploy and verify the API before the consoles that call it; stop on failure.
- Verify health, then continue. A deploy is done when the new instance is healthy, not when the command returns.
- Keep the previous version to roll back to. Record what was running before every update.
How this maps to self-hosting
| Principle | Compose | Helm |
|---|---|---|
| Immutable versions | Pin VEGADUTA_VERSION in deploy/compose/.env | --version <chart>; image tags come from the chart |
| Verify health | scripts/vegaduta upgrade pulls, recreates and waits for every service to be healthy | Rolling update with maxUnavailable: 0 (guarded); the migrate Job runs pre-upgrade and a failure rolls no pods |
| Roll back | Set the previous VEGADUTA_VERSION and run upgrade again. Database migrations are forward-only; restore from backup if one must be undone. | helm rollback (same caveat for migrations) |
Evidence & references
deploy/README.md(Upgrade),deploy/helm/README.md(Upgrade),deploy/helm/charts/core/values.yaml(strategy).- Rollback & Recovery.