Docs/Decisions/ADR-0003 Dynamic ChatClients

ADR-0003: Per-request Spring AI ChatClient construction (BYOK)

Status: accepted
Decided: 2026-07; re-confirmed 2026-08-15
Deciders: Platform Engineering
Scope: core/app (CoreApplication.java), core/chat, core/providers

Context

The platform is BYOK and multi-provider: each tenant (or individual agent) activates its own provider credential — Anthropic, OpenAI, Google GenAI, OpenRouter, Grok, Groq, Mistral, DeepSeek, Azure OpenAI, AWS Bedrock, Ollama, Meta, Sarvam and the platform-hosted pool — stored encrypted in the database and decrypted at request time. There is no single platform API key for these providers; spring.ai.openai.api-key is never configured at all.

Spring AI's default model is the opposite: each provider starter registers auto-configurations that eagerly build a singleton ChatModel from a static application.yml key at context refresh. On this platform those beans can never be constructed — and would be wrong even if they could be, because "the OpenAI key" is a per-request value, not an environment variable.

Two real failures forced the full exclusion set:

Excluding only the two chat auto-configurations missed that spring-ai-starter-model-openai registers six auto-configuration classes (chat, embedding, image, audio-speech, audio-transcription, moderation) and the Google GenAI starter registers three — each eagerly building a static model bean. OpenAiAudioSpeechAutoConfiguration was the one that actually crashed startup: "OpenAI API key must be set."
ToolCallbackConverterAutoConfiguration was briefly un-excluded on 2026-08-15 after decompiling the pinned spring-ai-autoconfigure-mcp-server-common-1.1.7.jar disproved the original crash theory. Verified on a running stack: startup was fine, but tools/list on the inbound MCP server returned all ~60 tools from every ToolCallback bean in the context — including filesystem read/write/delete and full remote browser control — to any caller holding a core/mcpserver grant key. A real capability leak, confirmed live; the exclusion was restored immediately.

Decision

CoreApplication excludes eleven Spring AI auto-configurations, and no singleton provider client exists anywhere:

AnthropicChatAutoConfiguration; all six OpenAI classes (OpenAiChat, OpenAiEmbedding, OpenAiImage, OpenAiAudioSpeech, OpenAiAudioTranscription, OpenAiModeration); all three Google GenAI classes (GoogleGenAiChat, GoogleGenAiEmbeddingConnection, GoogleGenAiTextEmbedding); and ToolCallbackConverterAutoConfiguration.
ChatClientRegistry (core/chat) maps ProviderName → AgentChatClientFactory; each call site resolves the caller's credential and the registry dispatches to one of 14 per-provider factories (AnthropicAgentChatClientFactory, GeminiAgentChatClientFactory, OpenRouterAgentChatClientFactory, …).
The factory builds a fresh Api + ChatModel + ChatClient per call from the decrypted key. BYOK plaintext keys are decrypted per call and are never cached as plaintext — the credential row is the only persistent form, under APP_ENCRYPTION_KEY.

flowchart LR call["Chat turn / quick task /
workflow node / assist"] --> reg["ChatClientRegistry"] reg --> fac["*AgentChatClientFactory
(per ProviderName)"] cred["ProviderCredentialService
decrypt per call"] --> fac fac --> cc["new ChatClient
+ LlmMetricsAdvisor
+ shared DefaultAgentAdvisors"]

Alternatives considered

Option	Why it lost
Spring AI auto-configuration + static keys	Structurally incapable of expressing "key is chosen per request, per tenant." Also crashes at startup because the keys it wants don't exist.
Singleton client pool, one per provider, platform key	Defeats BYOK: tenant spend and tenant quotas would be untrackable, and a platform key makes the platform the billing party for every tenant's usage.
Cached per-credential clients now	Deliberately deferred, documented in `AnthropicAgentChatClientFactory`: "worth caching (keyed by credential id + updatedAt, invalidated on key rotation) once this is under real load — not done here to avoid a caching bug masquerading as a feature this early." Reconstruction cost is acceptable at current load.

Consequences

Positive

Uniform instrumentation: because every ChatClient in the app funnels through ChatClientRegistry → the factories, registering LlmMetricsAdvisor once per constructed client covers every call site — agent chat, quick task, guardrails, reflexion, graph memory, workflow graph generation, Ask-AI assist — with zero per-site instrumentation.
No plaintext key ever sits in a long-lived bean; a heap dump shows at most the key in flight for the request being served.
Key rotation is trivially safe: the next call reads the current credential row; there is nothing to invalidate.
Provider capability quirks live next to the client they affect (e.g. Anthropic's "can't send both temperature and top_p", and the claude-sonnet-5 temperature-deprecation prefix list).

Negative / costs

Client construction cost on every LLM call — accepted for now; the documented future is a cache keyed by credential id + updatedAt so rotation invalidates automatically.
The exclusion list is load-bearing and counter-intuitive: removing ToolCallbackConverterAutoConfiguration from it re-opens the whole-context tool leak — the 5 curated MCP-server tools are only reachable through that same bean, so the exclusion also means core/mcpserver exposes nothing until its own narrow conversion is built (tracked follow-up, not a config toggle).
New provider starters must be audited for how many auto-configuration classes they register; a partial exclusion is a startup crash or a capability leak, not a compile error.

Do not remove

ToolCallbackConverterAutoConfiguration was un-excluded once and verified live to leak every tool in the context. Do not remove that exclusion again until core/mcpserver has a narrow conversion that does not route through aggregateToolCallbacks.

Evidence & references

core/app/src/main/java/com/agenticai/core/CoreApplication.java — the exclusion block and the dated javadoc history (incl. the 2026-08-15 live verification).
core/chat/.../ChatClientRegistry.java, core/chat/.../*AgentChatClientFactory.java (14 factories), core/chat/.../LlmMetricsAdvisor.java.
core/providers/.../ProviderCredentialService.java, ProviderCredentialResolver.java — encrypted credentials resolved per call.
AnthropicAgentChatClientFactory javadoc — the deferred caching design (key by credential id + updatedAt).