AI / LLM Layer

The AI layer is the heart of the platform: a Spring AI 1.1.7 stack that talks to fourteen LLM providers under two credential regimes (tenant BYOK and a platform-hosted pool), wraps every call in metrics, budget, guardrail and tracing advisors, and deliberately disables almost everything Spring AI would auto-configure. Everything on this page was verified against core/chat/, core/providers/, core/agents/, core/mcp/, core/mcpserver/ and core/pom.xml.

1. Build and dependency surface

Parent

spring-boot-starter-parent 3.4.4 (core/pom.xml)

Java

Release 21 (virtual threads enabled); local toolchain compiles on JDK 24

Spring AI BOM

spring-ai-bom 1.1.7, imported in dependencyManagement

Chat module starters

spring-ai-starter-model-anthropic, -openai, -google-genai

Agents module

spring-ai-starter-mcp-client (outbound MCP tool calls)

MCP server module

spring-ai-starter-mcp-server-webmvc (inbound MCP hosting)

AWS Bedrock

AWS SDK v2 2.28.29 with a hand-rolled AwsBedrockChatModel — no starter

HTTP layer

LlmHttpClientFactory supplies RestClient/WebClient builders with timeouts — Spring AI's defaults have none

2. Provider registry

ProviderName (core/providers/.../ProviderName.java) is the single source of truth for what the platform can call. The PLATFORM_HOSTED value is different in kind — it is a marker that means "resolve through the hosted pool," not a provider with its own API.

Provider	Enum	Client implementation	Quirks — verified in the factories
Anthropic	ANTHROPIC	AnthropicAgentChatClientFactory (Spring AI Anthropic)	Prompt caching via AnthropicProviderCachingStrategy; cache tokens recorded to llm_call_log
OpenAI	OPENAI	OpenAIAgentChatClientFactory	Baseline OpenAI-compatible path
Azure OpenAI	AZURE_OPENAI	AzureOpenAIAgentChatClientFactory	Hosted pool uses the newer /openai/v1/ bearer-token surface, not the classic deployment+api-key protocol
Google Vertex AI	GOOGLE_VERTEX	VertexAiChatClientFactory	Requires a single merged system message (SystemMessageMergingAdvisor); needs Google Application Default Credentials
AWS Bedrock	AWS_BEDROCK	AwsBedrockAgentChatClientFactory + hand-rolled AwsBedrockChatModel	AWS SDK v2 direct — no Spring AI Bedrock starter
Google Gemini (API key)	GEMINI	GeminiAgentChatClientFactory (google-genai)	GeminiMixedContentToolCallChatModel + GeminiSchemaSelfHealer; -latest aliases can silently roll generations — pin versions
xAI Grok	GROK	GrokAgentChatClientFactory (OpenAI-compatible)	Grok 4.x rejects stop and penalty params
OpenRouter	OPENROUTER	OpenRouterAgentChatClientFactory (OpenAI-compatible)	Custom baseUrl on shared OpenAiApi
DeepSeek	DEEPSEEK	DeepSeekAgentChatClientFactory (OpenAI-compatible)	Custom baseUrl
Groq	GROQ	GroqAgentChatClientFactory (OpenAI-compatible)	Custom baseUrl
Mistral	MISTRAL	MistralAgentChatClientFactory (OpenAI-compatible)	422s on stream_options but appends usage itself — excluded from the streamUsage flag
Meta (Llama / Muse Spark)	META	MetaAgentChatClientFactory (OpenAI-compatible)	Rejects stop / stream_options with HTTP 400; *-contributor models train on traffic
Sarvam AI	SARVAM	SarvamAgentChatClientFactory	Uses api-subscription-key header via NoopApiKey; own credential table
Ollama	OLLAMA	OllamaAgentChatClientFactory (OpenAI-compatible)	Self-hosted / local models
Hosted pool marker	PLATFORM_HOSTED	HostedLlmChatClientFactory → FreeTierChatClientService	Not a provider — resolves to a control-plane pool row at call time

3. Credential model — BYOK and the hosted pool

There is exactly one encryption chokepoint: EncryptionService (core/crypto), AES-256-GCM, keyed by APP_ENCRYPTION_KEY (a base64-encoded 32-byte key). Rotation is supported via APP_ENCRYPTION_KEY_PREVIOUS. If the key differs between any two places core boots, every stored credential becomes undecryptable.

Credential kind	Storage	Resolution
Tenant BYOK LLM keys	Per-tenant LlmProviderCredential rows in the tenant's own schema, AES-256-GCM encrypted	ProviderCredentialResolver, per call — never cached, plaintext never logged
Hosted pool keys	platform_hosted_llm_provider rows in the control plane (public)	Managed only via POST /api/platform/hosted-llm-providers — the endpoint encrypts on write
Sarvam / ElevenLabs / Jev keys	Own credential tables (not LlmProviderCredential)	Own services — e.g. ProviderCredentialResolver branches SARVAM to SarvamCredentialService

4. Request path — how a turn becomes a provider call

flowchart TD
  A["Inbound turn
(web / channel / job)"] --> B["ChatClientResolver"]
  B -->|"FREE tenant or usePlatformLlm"| C["FreeTierChatClientService
hosted-pool fallback chain"]
  B -->|"BYOK tenant"| D["ProviderCredentialResolver
decrypts tenant key"]
  C --> E["ChatClientRegistry
ProviderName → factory"]
  D --> E
  E --> F["*AgentChatClientFactory
builds ChatClient this call"]
  F --> G["Advisors: LlmGatewayService pre-flight
LlmMetricsAdvisor + DefaultAgentAdvisor"]
  G --> H["GuardedToolCallback chain
ceiling → HITL → validate → sanitize"]
  H --> I["Provider API"]
  I --> J["llm_call_log row
+ OTLP span to Langfuse"]

4.1 The registry and factories

ChatClientRegistry dispatches ProviderName to one factory class per provider (e.g. AnthropicAgentChatClientFactory). Every factory-built client gets the same advisor stack:

• LlmMetricsAdvisor — implements both CallAdvisor and StreamAdvisor. Records latency and call counts, writes the durable llm_call_log row, computes cost via LlmPricingCatalog, and restores TenantContext across Reactor thread hops on streamed calls.
• DefaultAgentAdvisor beans — includes RlTrajectoryAdvisor (per-turn trajectory recording for the RL subsystem).
• Hosted-only advisors — FreeTierTokenTrackingAdvisor and PerUserBudgetTrackingAdvisor run only on pool calls.

Two shared pieces of plumbing deserve note: ResilientToolCallingManagerFactory gives every client a tool-calling manager in which a malformed tool call cannot kill a turn, and AnthropicProviderCachingStrategy (with Gemini, OpenAI, Azure and Mistral siblings behind ProviderCachingRegistry) manages prompt caching per provider.

4.2 The hosted-pool fallback chain

When ChatClientResolver selects the pool (FREE tenant or usePlatformLlm), FreeTierChatClientService walks an ordered chain of platform_hosted_llm_provider rows ranked by HOSTED_POOL_ORDER_BY_COST, observed latency, and admin-assigned priority. HostedProviderHealth circuit breakers skip rows that are failing. A wholly failed chain surfaces to the user as a generic chat failure — when debugging, check all rows, not just priority 0.

5. The LLM gateway — pre-flight governance

Before any provider call, LlmGatewayService (core/chat/gateway/) runs the pre-flight chain: rate limiter → budget service → content policy → PII redactor → audit service. Its state lives in five tables: LlmGatewayPolicy, LlmGatewayUserPolicy, LlmGatewayUsageLedger, LlmGatewayAuditLog and LlmGatewayPiiAuditLog. A refusal or budget denial happens before the call — which is why "no llm_call_log row" is a valid debugging signal, not a bug.

6. Reasoning depth and the tool pipeline

Depth	Behaviour
FAST	Default. Direct streamed call, no extra verification passes.
CAREFUL	Self-consistency check over the answer.
RIGOROUS	CAREFUL plus reflexion and an independent ToolCallVerifierService pass.
PLANNED	Writes a task plan first; not combined with the CAREFUL/RIGOROUS quality passes in the same turn.

Non-FAST depths run through BufferedTurnRunner (they cannot stream mid-verification).

Tool resolution and the guarded pipeline

AgentToolProvider.resolveToolsFor assembles the toolset per turn from resolvers: integrations, knowledge (gated by the agent's knowledge profile and flag), browser, the MCP registry, commerce (payment links), and the imagegen / documentgen / sdlc / codeintel / google / sarvam generators, plus call_agent, a2a and progress tools. Every tool is wrapped in GuardedToolCallback inside LoggingToolCallback, and executes in this order:

1. Step ceiling — a shared AtomicInteger bounds tool calls per turn.
2. Optional RIGOROUS pre-verify — the independent verifier can reject a call before it runs.
3. HITL gate — tools matching mcp_tool.* policy go through ApprovalPolicyService + ApprovalReplayGrant.
4. ToolCallOutputValidator — validates the result shape.
5. sanitizeToolOutput — strips content the model should not see verbatim.

7. Session memory

Conversation memory is Redis-backed: MessageHistoryAdvisor rehydrates prior turns, persisted through RedisMessageStore and RedisChatSessionStore, with SessionSummarizer compacting long sessions. This is why Redis loss degrades continuity but never loses durable data — the system of record is Postgres.

8. MCP — outbound and inbound

Direction	Module	Mechanism	Controls
Outbound (agent → external MCP servers)	core/mcp	DB-backed registry with encrypted env vars; per-JVM DynamicMcpConnectionManager plus RedisMcpOwnershipRegistry for cross-replica ownership. Dynamic STDIO runs through the mcp-stdio-proxy Node sidecar (MCP_STDIO_SIDECAR_* envs).	McpStdioCommandGuard, McpRemoteUrlGuard; zero-credential stdio servers preconfigured: filesystem, memory, sequential-thinking, puppeteer, @playwright/mcp@0.0.79
Inbound (external MCP client → this platform)	core/mcpserver (Spring profile mcpserver)	STREAMABLE HTTP endpoint at /mcp, server name vegaduta-mcp-server	McpApiKeyAuthFilter + access grants + rate limiter; only the curated tools call_agent, call_workflow, search_knowledge are exposed

9. Supporting subsystems

Autonomous jobs

Scheduled/event agent runs are durable QUEUED rows with run-now-mode enqueue on the API pods, cron schedules, event triggers, approval parking, boot-time reconciliation of orphaned rows, and a structured output contract enforced by JobResultSchemaValidator. Progress notes stream back to the session.

Jev — the calibrated decision model

JevDecisionService wraps a TypeSafe-calibrated yes/no model used for two hot decisions: AgentRelevanceSelector (which agent should reply on a shared channel) and QueryComplexityClassifier (per-turn cost routing — deterministic checks first, Jev as rescue-only, fail-toward-primary). Jev uses its own BYOK credential and tenant policy, with a per-agent threshold; precedence is agent > tenant > platform flag.

Knowledge / RAG

Per-tenant pgvector vector_store plus hierarchical knowledge_document tables. The Python sidecar does chunking, embedding, OCR, transcription and reranking. KnowledgeSearchService runs hybrid retrieval — vector similarity + Postgres full-text search fused by Reciprocal Rank Fusion (RRF_K=60) — capped at rag.max-results=20 / rag.max-chars=8000 per turn. Ingestion jobs and Notion / SharePoint / web-crawl connectors feed the base; graphmemory adds Apache AGE multi-hop Cypher recall as a tool.

10. Tracing and the durable call record

Core exports OTLP spans to Langfuse:

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT = http://<langfuse-web>/api/public/otel/v1/traces
LANGFUSE_OTEL_AUTH_HEADER          = <Basic …>   (a secret: env var or Kubernetes Secret)
TRACING_SAMPLING_PROBABILITY       = 0.1           (10% sampling by default)

llm_call_log.trace_id is the Langfuse trace id — the join key between cost accounting (SQL) and prompt/tool detail (Langfuse). Spring AI's built-in observations light up via micrometer-tracing-bridge-otel. Tracing fails silently: a wrong endpoint or stale header drops spans but never breaks calls, so absence of traces is not evidence of absence of calls.

llm_call_log schema

One row per real LLM call, success or failure, in the tenant's own schema (assembled across migrations V52, V82, V93, V138, V195):

Column	Meaning
id	uuid
agent_id, session_id	which agent, which conversation
provider_name, model	who served it, which model string
prompt_tokens, completion_tokens, total_tokens	usage
cache_creation_tokens, cache_read_tokens	prompt-caching accounting (V195)
estimated_cost_usd	NUMERIC(12,6) — estimate from LlmPricingCatalog; reconcile against provider billing before quoting
success	boolean
error_message, exception_type, stack_trace	failure detail (V82)
latency_ms, started_at	timing
trace_id, span_id	join key to Langfuse (V138)
request_id	the req= value from the application log line

Queries worth memorising

SET search_path TO <tenant_schema>;

-- last 20 calls, newest first
SELECT started_at, provider_name, model, success, latency_ms,
       total_tokens, estimated_cost_usd, left(error_message, 80) AS err
FROM llm_call_log ORDER BY started_at DESC LIMIT 20;

-- today's failures, grouped by cause
SELECT provider_name, model, exception_type, count(*), max(started_at)
FROM llm_call_log
WHERE success = false AND started_at > now() - interval '24 hours'
GROUP BY 1,2,3 ORDER BY 4 DESC;

-- spend and volume by model, last 7 days
SELECT model, count(*) AS calls,
       sum(total_tokens) AS tokens,
       round(sum(estimated_cost_usd), 4) AS usd
FROM llm_call_log
WHERE started_at > now() - interval '7 days'
GROUP BY 1 ORDER BY usd DESC NULLS LAST;

-- p50 / p95 latency by provider
SELECT provider_name,
       percentile_cont(0.5)  WITHIN GROUP (ORDER BY latency_ms) AS p50,
       percentile_cont(0.95) WITHIN GROUP (ORDER BY latency_ms) AS p95,
       count(*)
FROM llm_call_log WHERE started_at > now() - interval '24 hours'
GROUP BY 1;

-- one conversation, in order
SELECT started_at, model, success, total_tokens, trace_id
FROM llm_call_log WHERE session_id = '<session>' ORDER BY started_at;

11. Known provider quirks — quick reference

Provider / area	Behaviour	Handling
Meta Model API	400 on stop / stream_options	Params stripped in MetaAgentChatClientFactory
Mistral	422 on stream_options, but appends usage itself	Excluded from streamUsage(true)
Grok 4.x	Rejects stop and penalties	Params stripped in GrokAgentChatClientFactory
Sarvam	Non-standard auth (api-subscription-key)	NoopApiKey + custom header interceptor
Vertex	Requires single merged system message	SystemMessageMergingAdvisor
Gemini	-latest aliases roll generations silently (Gemini 3 requires echoed thought_signature)	Pin explicit model versions — never -latest — in the hosted pool
Spring AI HTTP	Default clients have no timeouts	LlmHttpClientFactory injects timeout-configured builders