Docs/Deployment/Rollback & Recovery

Rollback & Recovery

Rolling back is a version swap back to the previous release, not a restart. This page collects the rollback paths for the Compose and Helm deployments, the database recovery pointers, and the list of things that must not be done. Commands are quoted from repo docs; none were executed here.

Do not power-cycle a live stack to "fix" a hang

A host reset is a power-yank of the tenant database, and it destroys the evidence of why the system hung. Avoid host reboots, instance resets and docker compose down on a stack people depend on unless you have decided to take it offline. Collect evidence first, then decide.

Unverified page

The rollback recipes below follow deploy/README.md and deploy/helm/README.md; they were not run during authoring. The Helm charts are a preview and have not been run end to end against published images.

Rollback target: The version (VEGADUTA_VERSION or chart version) recorded before the upgrade. Never rely on latest
Compose upgrade: scripts/vegaduta upgrade --version X.Y.Z backs up .env, pulls, recreates and waits for health
Helm upgrade: helm upgrade with a pinned chart version; helm history / helm rollback apply

First: collect evidence, do not touch

scripts/vegaduta status              # service state + health
scripts/vegaduta logs core           # follow one service
scripts/vegaduta doctor              # docker/compose versions, RAM, disk, ports, secrets, URLs

D=example.com   # your BASE_DOMAIN; locally use the localhost ports instead
for u in https://api.$D/actuator/health https://app.$D/ \
         https://auth.$D/realms/agentic-ai/.well-known/openid-configuration \
         https://admin.$D/; do
  echo "$(curl -s -o /dev/null -w '%{http_code}' -m 20 "$u")  $u"; done

Then ask what changed (upgrade, config edit, secret rotation, realm import). An escalation bundle should contain: environment, HTTP codes, running image versions, last change, 20 log lines, what you ruled out.

Which rollback do I need?

Situation	Action
An upgrade made things worse	Version swap back to the recorded previous release (below)
core never becomes healthy after an upgrade	Read its logs for the cause (a Spring context-start failure or a Flyway error is typical). Version swap back if it is not a quick config fix
Config is wrong (env var, redirect URI, switch)	Fix `deploy/compose/.env` or Helm values and recreate the affected service; check the issuer settings still match `PUBLIC_AUTH_URL`
Stored credentials suddenly undecryptable	`APP_ENCRYPTION_KEY` mismatch: restore the key the data was written with (from your backup of `.env` or secret store), never re-encrypt
Migration went wrong	Rolling the image back does not roll the schema back. See "Migrations" below
Data lost or corrupted	Database restore path below (human decision)

Version rollback

Compose

scripts/vegaduta upgrade --version <previous-version>

The previous version must still be pullable from GHCR (release images are signed and versioned). If only a frontend changed, roll only that image back and leave core alone. Verify with scripts/vegaduta status and the health sweep above before believing any 200. For images built from source (--build), keep the previous image tagged before rebuilding, for example docker tag <image>:latest <image>:prev, otherwise there is nothing to return to.

Helm

helm history vegaduta -n vegaduta
helm rollback vegaduta <revision> -n vegaduta

Or re-run helm upgrade with the previous pinned chart version. Note that the migration hook Job has already run for the newer version; see below.

Config, secrets and Keycloak

Secrets. Keep the previous value of every rotated secret written down until the new one is verified. For APP_ENCRYPTION_KEY, always retain the previous value as the rollback.
Keycloak. Do not re-import a realm to "roll back": import is a no-op on an existing realm, and re-importing the committed JSON on a live realm can disable the social IdPs, locking out federated users who have no password. Back up first and use live updates; see Keycloak Provisioning.

Migrations: the image rolls back, the schema does not

Flyway runs inside core at startup (on Helm, in a migration hook Job); there is no down path. Plan for it in the release, not in the rollback: expand, migrate, contract (never ship destructive DDL in the same release as the code that stops using the old column), and never edit an applied migration (checksum mismatch). A duplicate version number makes core crash-loop; the only repair is a human-run edit of public.flyway_schema_history after taking a backup, which is not a routine step. Flyway versions share one namespace across the controlplane/ and tenant/ directories; scripts/ci/flyway-lint.sh (part of make gates) guards this before merge. Read the release notes before upgrading.

Database recovery pointers

What	How
Postgres (Compose)	`scripts/vegaduta compose exec -T postgres pg_dumpall -U agentic \| gzip > dump.sql.gz`; restore with `gunzip -c dump.sql.gz \| scripts/vegaduta compose exec -T postgres psql -U agentic -d postgres` (`deploy/README.md`)
Keycloak database (Full)	`pg_dump` of `keycloak-db`; in Lite, Keycloak lives in the `keycloak-data` volume
Files	The `core-storage` (Lite) or `seaweedfs-data` (Full) volume, or your external S3 bucket
Windows scheduled backups	`scripts/backup/pg-backup.ps1`, `register-backup-task.ps1`, `verify-restore.ps1`
Kubernetes	Your managed PostgreSQL's backups, or the CloudNativePG cluster's backup configuration if you use the bundled chart

Restoring into a stock postgres:16 image silently loses pgvector columns; restore into an image with pgvector and Apache AGE (the release postgres image).
After any restore, APP_ENCRYPTION_KEY must be the key the data was written with, or every stored provider key and channel secret is undecryptable. Back up deploy/compose/.env together with the data.

Full detail: Backup & Restore.

What NOT to do

Do not	Because
Reset or reboot the host, or `docker compose down`, on a stack people depend on while diagnosing	A reset power-yanks the database and destroys the evidence needed to diagnose the stall
Restart or reboot to "see if it helps" while collecting evidence	Same rule; gather logs, container states and memory first
Track `latest` on a deployment you depend on	A re-run can silently change the running version; pin a release
Re-encrypt data under a new `APP_ENCRYPTION_KEY` to "fix" decryption	The plaintext is gone; point back at the original key
Edit an applied Flyway migration, or edit `flyway_schema_history` unattended	Checksum failure; any repair must be human-run with a backup
Re-import the Keycloak realm JSON into a live realm	No-op for existing realms; can disable social login where it does import
`docker image prune -a` before the new version is verified	Keep a rollback image until the upgrade is verified

Evidence & references

deploy/README.md (operations, backup, upgrade); deploy/helm/README.md; docs/installation.md.
scripts/backup/; scripts/ci/flyway-lint.sh.