Deployment shapes and pools¶

One image, several ways to start it. A shape says what a pod is for; a pool splits one shape into groups of pods with different jobs. Both are chosen by Spring profile, so nothing here is a different build. The reasoning, and what is deliberately not split, is in ADR-0014.

The shapes¶

Shape	Profile	Runs	REST	Replicas
API	`shape-api` (only when `core-jobs` is deployed)	Chat, streaming, every controller; the scheduled work listed under "Also run in the API shape" below	yes	Many. Needs the replica contract
Jobs (main pod)	`shape-jobs`, `jobs` (+ `APP_SCHEDULING_OFFLOADED_POOLS` when pools are on)	Every sweep and poller the API hands over, minus the pools that run elsewhere	no (actuator only)	One, or a few for failover
Jobs, `ingest` pool	`shape-jobs`, `shape-jobs-ingest`, `jobs`	Only document ingestion and re-embedding	no	One (failover only)
Jobs, `webhooks` pool	`shape-jobs`, `shape-jobs-pool`, `jobs`, `APP_SCHEDULING_POOL=webhooks`	Only webhook delivery and the workflow event outbox relay	no	One (failover only)
Jobs, `intake` pool	same, `APP_SCHEDULING_POOL=intake`	Only the inbound email and X DM pollers, plus the per-JVM MCP, browser and sandbox sweeps their agent turns need (see "Per-JVM by design")	no	One (failover only)
Jobs, `billing` pool	same, `APP_SCHEDULING_POOL=billing`	Only the usage rollup, spend-counter reconcile, payment reconciliation and comp-grant sweeps	no	One (failover only)
Worker	`worker`, `temporal`	Temporal workflow task queue	no	Scale with workflow load
Migrate	(hook Job)	Flyway for the control plane and every tenant schema	no	Runs to completion

With no shape profile (the Compose default) one process runs everything. Nothing below applies to it.

On the full Compose stack the jobs profile adds core-jobs, but Compose cannot see which profiles are active, so the API only gets shape-api when you set CORE_SHAPE_PROFILE=,shape-api next to it (in deploy/compose/.env). Without that the API keeps running every sweep and poller beside core-jobs (advisory locks keep each tick single-winner, so nothing runs twice, but the API JVM carries the load the jobs pod was added to take). Set it only together with the jobs profile: with shape-api and no jobs pod nothing runs the background work.

The chat hot path never leaves the API pod. Pools and shapes split background work only.

Who runs which scheduled bean¶

The three tables below and the counts in the sentences around them are generated by scripts/docs/gen-shapes.py from the @Scheduled sources under core/ and the profile files in core/app-config/src/main/resources (application-shape-*.yml, including app.scheduling.pools); do not edit what sits between the gen-shapes markers by hand. After changing a scheduled bean or a shape profile, run python3 scripts/docs/gen-shapes.py; make gates runs it with --check. The generator only checks the tables against the sources and profile files (see "What is not proven"); JobsPoolsCoverEveryJobTest is the test that a pool split drops no job, and ScheduledJobsAreLockedOrAnnotatedTest keeps the last column honest. The per-JVM reasons are taken from @AllowUnlockedSchedule(reason = ...) with a trailing cross-reference ("see the javadoc above", "See docs/...") removed.

Owned by the jobs family (the API does not run them)¶

Class	Module	API shape	Main jobs pod	Own pool, when enabled	Cross-replica
`AbandonedSignupCleanupJob`	tenancy	-	yes	-	advisory lock
`AgentExperimentService`	agents-turn	-	yes	-	advisory lock
`AgentJobApprovalSweep`	agents-jobs	-	yes	-	advisory lock
`AgentJobLeaseSweep`	agents-jobs	-	yes	-	advisory lock
`AgentJobRunner`	agents-jobs	-	yes	-	advisory lock
`AgentScheduleTicker`	agents-jobs	-	yes	-	advisory lock
`AuditSealSweep`	platform	-	yes	-	advisory lock
`ChannelSessionResolutionSweep`	app	-	yes	-	advisory lock
`CompGrantExpiryReverter`	tenancy	-	yes, unless its pool is on	billing	advisory lock
`DailyQuotaResetScheduler`	agents	-	yes	-	advisory lock
`DemoSessionExpiryJob`	tenancy	-	yes	-	advisory lock
`EmailInboundPollingJob`	omnichannel	-	yes, unless its pool is on	intake	advisory lock
`KnowledgeIngestionJobRunner`	knowledge	-	yes, unless its pool is on	ingest	advisory lock, or per-job claiming when `KNOWLEDGE_INGESTION_CLAIMING_ENABLED`
`KnowledgeReembedJobRunner`	knowledge	-	yes, unless its pool is on	ingest	advisory lock
`LlmSpendCounterReconciler`	chat	-	yes, unless its pool is on	billing	advisory lock
`LlmUsageRollupScheduler`	platform	-	yes, unless its pool is on	billing	advisory lock
`ModelCapabilitySummaryGenerationService`	chat	-	yes	-	advisory lock
`ProviderModelCatalogService`	providers	-	yes	-	advisory lock
`RetryFailedActionsJob`	approval	-	yes	-	advisory lock
`VideoGenerationJobRunner`	imagegen	-	yes	-	advisory lock
`WebhookDeliveryRelay`	eventbus	-	yes, unless its pool is on	webhooks	advisory lock
`WorkflowEventOutboxRelay`	eventbus	-	yes, unless its pool is on	webhooks	advisory lock
`WorkflowRunRecoveryJob`	workflow	-	yes	-	advisory lock
`WorkflowScheduler`	workflow	-	yes	-	advisory lock
`XDmPollingJob`	omnichannel-social	-	yes, unless its pool is on	intake	advisory lock

Each pool is optional and independent, so any combination can be on. A pool's jobs run in that pool's pod and nowhere else in the jobs family; with the pool off, the main jobs pod runs them exactly as before. The pools are defined once, in app.scheduling.pools (application-shape-jobs.yml), and the main pod learns which are elsewhere from APP_SCHEDULING_OFFLOADED_POOLS, set by the chart from the same switches that create the pool Deployments, so the two cannot disagree. JobsPoolsCoverEveryJobTest checks all 16 combinations of the four pools at matcher level: every job the single jobs pod runs runs in exactly one pod (plus the intake pod, for the per-JVM lifecycle classes it lists). SchedulingRoleTest binds the same properties through SchedulingRoleRegistrar and a real environment. No pod has been started.

What a pool pod runs, exactly: the pool's own classes, plus (for intake only) the seven per-JVM lifecycle classes listed in app.scheduling.pool-lifecycle. The intake pollers run whole agent turns (agentChatService.chat, workflow triggers), which can open MCP connections, browser sessions and sandbox artifacts in the pod's own JVM, so that pod keeps the sweeps that renew and reap them. The ingest, webhooks and billing pods run none of the per-JVM classes: their jobs run no agent turn, and the ingest pool never did. Moving a job that runs agent turns into a pool means adding a pool-lifecycle entry for that pool.

Boot-time listeners are narrower than that. A pool pod skips the listeners in skip-listener-classes (AgentJobBootReconciliation, DailyQuotaResetScheduler, the eval and simulation executors, the demo Telegram bot, LocalLlmPoolBootstrap). Listeners not on that list still run on every pool pod at boot, for example DynamicMcpConnectionManager.reconnectEnabledOnStartup (reconnects every tenant's enabled MCP registrations), SandboxArtifactRetention.onReady and OfficialMcpRegistryClient.warmOnStartup. Enabling the three new pools therefore adds three more pods doing that boot work.

A pool pod does not inherit core-jobs.env, extraEnv or secretEnv (only global.core.* and the pool's own values). A setting an operator put on core-jobs is absent in the pool for the jobs that move.

Not a pool: eval runs. EvalRunExecutor and SimulationRunExecutor run inside the API JVM on a thread pool started by a REST call; there is no poller to hand to another pod, and the boot sweep that fails their orphans depends on that. Moving them needs a durable queue and claiming (a change to a data path), so no core-eval pool exists. The only scheduled eval class, QualityRunOrphanSweep, is a cheap per-tenant UPDATE that ticks in the API shape.

Also run in the API shape¶

These tick in the API pod as well. All 22 take the Postgres advisory lock, so one pod wins each tick; they still borrow a pinned database connection in a chat pod, and each can move to the jobs family once its owner confirms it depends on no API-pod state. AbuseSignalService is the exception: it writes into the API JVM's throttle map, which the API's request filters read, so it stays on the API pods.

Class	Module	API shape	Main jobs pod	Own pool, when enabled	Cross-replica
`AbuseSignalService`	platform	yes	-	-	advisory lock
`AgentSentinelSweep`	agents-jobs	yes	yes	-	advisory lock
`AgentSupervisorSweep`	app	yes	yes	-	advisory lock
`ApprovalExpiryJob`	approval	yes	yes	-	advisory lock
`BriefTrustWatcher`	brief	yes	yes	-	advisory lock
`ChatBranchRetentionPurge`	agents-turn	yes	yes	-	advisory lock
`CompGrantReminderJob`	tenancy	yes	yes, unless its pool is on	billing	advisory lock
`DataRetentionPurge`	platform	yes	yes	-	advisory lock
`MemoryReviewSweep`	agents-jobs	yes	yes	-	advisory lock
`PaymentReconciliationJob`	payments	yes	yes, unless its pool is on	billing	advisory lock
`QualityRunOrphanSweep`	quality	yes	yes	-	advisory lock
`RetentionPurgeJob`	app	yes	yes	-	advisory lock
`RoiBudgetAlertSweep`	app	yes	yes	-	advisory lock
`RunDataCleanupJob`	workflow	yes	yes	-	advisory lock
`ScheduledTaskTicker`	agents-jobs	yes	yes	-	advisory lock
`StalePendingUpgradeCleanup`	tenancy	yes	yes, unless its pool is on	billing	advisory lock
`StepJournalRetentionPurge`	agents-jobs	yes	yes	-	advisory lock
`StudyReviewReminderSweep`	app	yes	yes	-	advisory lock
`TurnJournalRetentionPurge`	agents-turn	yes	yes	-	advisory lock
`VectorSchemaInitializer`	knowledge	yes	yes	-	advisory lock
`WorkflowPollTriggerTicker`	workflow	yes	yes	-	advisory lock
`WorkflowWaitWakeJob`	workflow	yes	yes	-	advisory lock

Per-JVM by design¶

These describe the state of the JVM they run in, so every replica runs its own copy and none takes a lock. Each carries @AllowUnlockedSchedule(reason = ...).

Class	Module	Why every replica runs it
`AbuseDetectionService`	app	Evicts expired buckets from this JVM's own in-memory rateLimitBuckets map.
`BrowserTakeoverService`	agents-browser	Acts on this JVM's own in-memory browser sessions; every replica must run its own sweep.
`BrowserUseSessionManager`	agents-browser	Closes this JVM's own browser sessions (the local sessions map pinned to the executor thread); every replica must reap its own.
`DynamicMcpConnectionManager`	agents-tools	Pings this JVM's own liveClients (subprocess/HTTP clients are per-JVM).
`ImpersonationService`	platform	Flushes THIS JVM's in-memory view-as request counters; every replica drains its own.
`LandingEventBuffer`	platform	Flushes THIS JVM's own in-memory event queue to Redis; every replica must drain its own buffer.
`McpOrphanAdopter`	agents-tools	Coordinated by the Redis ownership lease (adopt takes mcp:owner:* before connecting), not by a per-tick lock; exclusive-clients mode needs every replica to look for orphans.
`RedisMcpOwnershipGate`	agents-tools	Renews THIS replica's own leases and replica registration; must run on every replica by design.
`SandboxArtifactRetention`	agents-tools	Drains this JVM's own in-memory queue of artifacts it stored (pending); every replica deletes the artifacts it tracked.

The ingest, webhooks and billing pools skip all of these: their jobs hold none of this state. The intake pool keeps 7 of them (every row except AbuseDetectionService and ImpersonationService). The intake pollers run agent turns, which is why that pool keeps per-JVM sweeps; the other pools' jobs run none.

Choosing a topology¶

You want	Do
The simplest thing	Defaults: API plus one jobs pod (`global.components.coreJobs=true`)
Large uploads must not slow agent jobs	`global.components.coreIngest=true`
A slow customer endpoint must not delay agent jobs	`global.components.coreWebhooks=true`
A slow mailbox or provider API must not delay agent jobs	`global.components.coreIntake=true`
Tenant fan-out sweeps (usage rollup, reconciliation) must not hold jobs-pod connections	`global.components.coreBilling=true`
More chat capacity	More API replicas: Running the API on more than one replica
Capacity that follows load	Add `deploy/helm/examples/values-autoscaling.yaml`
Workflows on Temporal	`global.temporal.enabled=true` (adds the worker)

Running the API on more than one replica¶

The chart refuses to render core.replicaCount > 1, an HPA or KEDA until every core.replicaReadiness flag is true. In that mode it also sets APP_REPLICA_MODE=multi and moves every per-pod store to Valkey:

Store	Switch the chart sets	Why it matters
A2A task store	`APP_A2A_TASK_STORE`	`GetTask` and `CancelTask` find the task on any pod
Abuse throttle	`APP_ABUSE_THROTTLE_STORE`	a throttle set on one pod applies on all
MCP registry	`APP_MCP_REGISTRY_STORE`	one pod owns each MCP connection; the rest route to it
Data-analysis sandbox files	`APP_SANDBOX_DATA_ANALYSIS_STORE`	a follow-up turn on another pod finds the uploads
E2B run quota	`APP_SANDBOX_PLATFORM_E2B_QUOTA_STORE`	the daily cap is not multiplied by the replica count
Resumable chat streams	`APP_CHAT_RESUMABLE_STREAMS_STORE`	a reconnect on another pod replays the turn
Guardrail procedure state	`APP_GUARDRAILS_PROCEDURES_SESSION_CALLS_STORE`	ordering rules (`REQUIRE_TOOL_BEFORE`) hold across pods
Studio approvals	`APP_DEMO_STUDIO_APPROVAL_STORE`	an approve or reject reaches the pod running the demo
Tool effect journal	`TOOL_EFFECT_JOURNAL_STORE`	a retried write or send is recognised as a repeat
Eval boot sweep	`APP_QUALITY_BOOT_ORPHAN_SWEEP=stale`	a booting pod no longer fails runs another pod is executing

You must also supply the A2A signing key pair (APP_A2A_SIGNING_PRIVATE_KEY, _PUBLIC_KEY) in the app Secret and set core.replicaReadiness.a2aSigningKeysPinned=true; the chart cannot do that for you. ReplicaStoreCoverageTest fails the build if a store with a Valkey switch is not on this list.

Set core.replicaStrict=true to make core refuse to start on a fatal finding (for example MCP OAuth enabled, whose state is per JVM). Without it the same findings are logged as warnings.

A pod that is removed lets its open chat streams finish only for app.shutdown.drain-seconds (default 45). For long agent turns use the preset, which sets 300 seconds and a matching terminationGracePeriodSeconds:

helm upgrade --install vegaduta deploy/helm/charts/vegaduta -f values.yaml \
  -f deploy/helm/examples/values-scale-out.yaml

Autoscaling¶

CPU is a poor signal for an API that mostly waits on model tokens. Scale on the requests a pod is serving right now:

http_server_requests_active_seconds_count

Spring Boot exports it with no code of ours. deploy/helm/examples/values-autoscaling.yaml shows two routes: an HPA Pods metric through prometheus-adapter, or a KEDA ScaledObject with a Prometheus trigger. Both keep CPU as a floor. Scale-in is slow by default (one pod per five minutes after ten quiet minutes) because removing a pod ends the streams still open on it after the drain window. Both routes were rendered and schema-checked, not run on a cluster.

The database connection budget¶

Every core pod holds up to its Hikari pool, and a replica count or an autoscaler multiplies it.

Pod	Default pool
API	12 (15 when `core-jobs` is not deployed)
Jobs	10
Ingest, webhooks pools	6 each
Intake, billing pools	8 each (locked jobs pin up to 2 connections each, `PgAdvisoryJobLock`; intake also runs agent turns)
Worker	8
Migrate Job	10 (while the old pods still hold theirs)

The umbrella chart adds the worst case (the autoscaler's maximum for the API) and fails the render if it exceeds the limit minus a reserve of 20. The limit is postgres.parameters.max_connections for the bundled cluster (chart default 200) or global.postgresql.maxConnections for an external server. A pooler in front must run in session mode: tenants are Postgres schemas selected with SET search_path, and transaction pooling would run queries in the wrong tenant's schema.

Adding a pool or moving a job¶

Decide who owns the class and say so in the profile files: add it to shape-api's skip list to move it off the chat pods, and list it under one pool in app.scheduling.pools (application-shape-jobs.yml). A new pool also needs: a name in JobsPoolsCoverEveryJobTest.POOLS, a chart (copy core-webhooks), an entry in vegaduta.corePools (charts/vegaduta/templates/_guards.tpl) and in vegaduta-lib.coreEnv (_core.tpl), its global.components switch, and its name in every allowFrom list that names core-ingest.
Keep JobsPoolsCoverEveryJobTest green. It tells you if a job would run nowhere, or if a profile names a class that does not exist.
Run python3 scripts/docs/gen-shapes.py to regenerate the ownership tables (--check fails make gates when they are stale), add a pool's row to the shapes table by hand, and commit the result.
If the job is not safe to run twice, it needs the advisory lock or an @AllowUnlockedSchedule reason (ScheduledJobsAreLockedOrAnnotatedTest).
A new in-memory store needs a Valkey switch, a finding in ReplicaModeSafetyCheck and an env var in the Helm library chart (ReplicaStoreCoverageTest).

What is not proven¶

The three new pools (core-webhooks, core-intake, core-billing) have been rendered, schema-checked and covered by guard cases, and the scheduling partition has been unit-tested. No pool pod has been started, on a cluster or in Compose, and nothing has been measured: no latency, throughput or isolation number exists. In particular the intake pod's per-JVM sweeps have not been observed running an agent turn.
The pool resource defaults (core-webhooks and core-intake 250m CPU request, 768Mi request, 1536Mi limit; core-billing 100m, 768Mi, 1536Mi) are estimates from the size of a full core context, not measurements.
A pool relieves the main jobs pod only. PaymentReconciliationJob, CompGrantReminderJob and StalePendingUpgradeCleanup still tick in the API shape as well; the billing pool takes only their copy in the jobs family.
Pools isolate; they do not parallelise. Every poller is advisory-locked, so a second replica in a pool is failover.
scripts/docs/gen-shapes.py reads the YAML profile lists and the Java sources textually (line-anchored @Scheduled, brace matching, as ScheduledJobsAreLockedOrAnnotatedTest does). It does not start Spring, so it does not evaluate @Profile or @ConditionalOn* (a listed class may still not be registered in a given deployment), does not see beans registered by @Bean methods outside a class with its own @Scheduled, and does not see properties set outside these files (Helm values, environment variables).
A class with both locked and @AllowUnlockedSchedule methods is listed with the lock and a note that some methods are per-JVM; a per-JVM class with several reasons shows the first.
The shapes table at the top, the "Per-JVM by design" and pool prose and the connection budget are written by hand. The generator only fails when a pool in app.scheduling.pools has no **Jobs,pool** row in the shapes table.