Docs/Security & Compliance/Secrets & Key Management

Secrets & Key Management

Scope: Where secrets live per deployment type, how tenant-supplied credentials are encrypted at rest, rotation mechanics, and the traps that have bitten before
Rule of this page: Names and locations only. No values, fragments or connection strings appear here or should ever be added.
Related: Security Architecture, ADR-0010, ADR-0005, Threat Model

1. Where secrets live

Deployment	Store	Notes	Evidence
Docker Compose (Lite / Full)	`deploy/compose/.env` (gitignored)	Generated by `scripts/vegaduta secrets` with random values; existing values are never overwritten. Compose refuses to start without the required ones (`${...:?}`), including `APP_ENCRYPTION_KEY`.	`deploy/compose/.env.example`, `deploy/compose/docker-compose.yml`, `.gitignore`
Kubernetes (Helm)	Kubernetes Secrets	`global.core.appSecret.existingSecret` holds `APP_ENCRYPTION_KEY` (and any other core env such as SMTP or OAuth secrets) for core, core-jobs, core-worker and the migrate Job; left empty, the chart generates `<release>-core-app` once on `helm install`. Components have their own `existingSecret` values. Use an external-secrets operator if you keep secrets in a vault.	`deploy/helm/charts/vegaduta/values.yaml`
Local core process	Your shell environment (or `core/.env.local`, gitignored)	Must carry the same `APP_ENCRYPTION_KEY` as the database it points at.	`scripts/dev-up.sh` header, `.gitignore`

The original hosted deployment kept these values in a cloud key vault; that setup is not part of this repository (ADR-0010).

2. Secret inventory (names only)

Variable	Consumers	Consequence if wrong or lost
`APP_ENCRYPTION_KEY` (+ `APP_ENCRYPTION_KEY_PREVIOUS` during rotation)	core, core-jobs, core-worker	Every stored provider key and channel secret becomes undecryptable
`KEYCLOAK_ADMIN_PASSWORD`, `KEYCLOAK_CLIENT_SECRET`, `KEYCLOAK_DB_PASSWORD`	Keycloak, core	Login, admin API (signup), Keycloak boot
`POSTGRES_PASSWORD`	postgres, pgbouncer, core	Core cannot reach Postgres
`VALKEY_PASSWORD`	valkey, core (as `REDIS_PASSWORD`)	Sessions, rate limiting, caches
`SIDECAR_SHARED_SECRET`, `MCP_STDIO_SIDECAR_SHARED_SECRET`, `TOOL_SANDBOX_INTERNAL_TOKEN`, `CODE_INTEL_INTERNAL_TOKEN`, `APP_BUILDER_*`, `RL_TRAINER_INTERNAL_TOKEN`	core plus the named service	Ingestion/transcription; STDIO MCP; custom code tools; code intelligence; app builder
`STORAGE_ACCESS_KEY`, `STORAGE_SECRET_KEY`	core, seaweedfs	Uploads and artifacts
`LANGFUSE_*`	Langfuse web/worker, core (OTEL auth header)	Tracing
`PLATFORM_MASTER_KEY`, `PLATFORM_ALERTS_WEBHOOK_SECRET`	core, Alertmanager	Tenant hard-delete control; alert webhook
`LIVEKIT_API_SECRET`, `TEMPORAL_DB_PASSWORD`, `SMTP_PASSWORD`, `KEYCLOAK_SMTP_PASSWORD`, social `*_CLIENT_SECRET`	respective services	Voice/video; workflow persistence; transactional email; social login

3. Encryption of tenant-supplied credentials

One class, one audit question. Every stored provider API key, channel credential (Twilio, Meta, Slack, Discord, Telegram, email, X), OAuth refresh token and MCP server env goes through EncryptionService (AES-256-GCM, fresh random 12-byte IV per call, 128-bit tag; stored as base64 of IV plus ciphertext). "Is X encrypted?" is answered by grepping for that type.
Key source. APP_ENCRYPTION_KEY, base64 of exactly 32 bytes. Startup fails fast if missing, invalid base64, or the wrong length.
Dual-key rotation. encrypt() always uses the current key; decrypt() tries the current key then APP_ENCRYPTION_KEY_PREVIOUS. No ciphertext-format change and no migration of the roughly fifteen credential tables was needed.
Envelope encryption exists but is not wired. EnvelopeEncryptionService with KeyManagementProvider (per-secret data keys wrapped by a key-encryption key) is implemented, with only LocalKeyManagementProvider, which wraps under the same APP_ENCRYPTION_KEY. Its javadoc states it is intentionally not used by the existing credential tables. There is no cloud-KMS provider in the repository. So today's blast radius of a master-key compromise is: all tenant credentials.
Hashed, not encrypted. Demo tokens, MCP grant keys and webhook tokens are high-entropy and stored as SHA-256 digests (DemoTokens, McpApiKeyHasher, WebhookTokenHasher); an internal SAST review judged that appropriate for high-entropy secrets.

Standing rule: one key everywhere core boots

APP_ENCRYPTION_KEY must be identical everywhere core boots: deploy/compose/.env, a locally run core, and your Kubernetes Secret. A different key does not error on write; it makes every previously stored credential undecryptable (GCM tag failure). Never regenerate it on an existing database, and restore it together with any database backup.

4. Rotation runbook (encryption key)

Generate a new 32-byte base64 key.
Set APP_ENCRYPTION_KEY_PREVIOUS to the current key; set APP_ENCRYPTION_KEY to the new key; restart core.
New and updated secrets encrypt under the new key; old rows still decrypt via the fallback.
There is no bulk re-encryption job (documented scope limit). Re-save each credential through its normal update endpoint, then remove APP_ENCRYPTION_KEY_PREVIOUS and restart to retire the old key.

Rotation of other secrets (Keycloak client secret, shared secrets, DB passwords): change the value in deploy/compose/.env or the Kubernetes Secret, update the other side where the secret is shared (for example the Keycloak client and KEYCLOAK_CLIENT_SECRET; the realm reconcile re-syncs it), and recreate the consumers. No rotation schedule is prescribed by the repository.

5. Separation rules seen in the code

Sandboxes get no core secrets: mcp-stdio-runner is designed with no secrets and its own shared secret; the compose comments state the relay secret is deliberately not APP_ENCRYPTION_KEY. See Security Architecture §7.
Internal-only URLs stay internal; browser-facing and issuer URLs never point at localhost.
Operator guidance: never paste a secret into a document, chat, commit or log.

Open items in this area (status only)

Envelope/KMS encryption not wired; no cloud KMS provider.
No documented rotation cadence or evidence of past rotations of the encryption key.
The plaintext deploy/compose/.env file on a Compose host is an operator risk: restrict its permissions and its backups; see the Threat Model.

Evidence & references

core/crypto/src/main/java/com/agenticai/core/crypto/EncryptionService.java, EnvelopeEncryptionService.java, KeyManagementProvider.java, LocalKeyManagementProvider.java.
deploy/compose/.env.example; deploy/helm/charts/vegaduta/values.yaml (existingSecret values).
deploy/compose/docker-compose.yml (APP_ENCRYPTION_KEY required; mcp-stdio secret comments), .gitignore.
Internal SAST review (crypto and hashing; not published).