Docs/Operations/Observability

Observability: Logs, Metrics & Traces

Every signal the platform emits, where it lands in a self-hosted install, and how to read it. The Compose stack ships an opt-in Prometheus / Grafana / Loki / Promtail / Alertmanager set (profile observability) and Langfuse for per-LLM-call traces (profile tracing). On Kubernetes the charts emit Prometheus Operator objects and export traces to your OpenTelemetry endpoint.

Where each signal lives

Signal	Store	Read it with
Container stdout/stderr	Docker logs; Loki (via Promtail) with profile `observability`, retention 14 days (`deploy/loki/loki-config.yaml`)	`scripts/vegaduta logs core`; Grafana Explore
Metrics	Prometheus, retention `PROMETHEUS_RETENTION` (default 15d); scrapes `core:8080/actuator/prometheus`, `valkey-exporter`, Temporal, SeaweedFS	Grafana dashboards in `deploy/grafana/provisioning/dashboards`
Alerts	Prometheus rules in `deploy/prometheus/rules/` (shared with Helm), routed by Alertmanager	Alertmanager on `127.0.0.1:9093`; core stores webhook alerts (below)
LLM traces, prompts, tokens, latency	Langfuse (own Postgres + ClickHouse), profile `tracing`	Langfuse UI on `127.0.0.1:3002`
Durable app records — `llm_call_log`, audit, MCP registrations, jobs	Postgres (per-tenant schema)	`psql` via `scripts/vegaduta compose exec postgres`

Blind spots — read before concluding "nothing happened"

Prompt/completion text is deliberately not in app logs. It is in Langfuse, sampled (default 10 %).
Secrets are redacted in MCP failure messages before logging — a blank-looking credential in an error is the redaction, not an empty config.
Browser-side errors never reach the server; use the browser console.

Reading a core log line

Core's console pattern (application.yml):

%d{yyyy-MM-dd'T'HH:mm:ss.SSSXXX} %-5level [%thread] [req=%X{requestId:-} tenant=%X{tenantId:-} user=%X{userId:-}] %logger{36} - %msg%n

req= — request id; grep it to pull every line one request touched.
tenant= — the tenant schema; empty means platform-level work or a background job that iterates tenants.
[thread] — tomcat-handler-* is a real request, scheduling-* a scheduled job, mcp-* MCP connection management.

scripts/vegaduta status                               # service state + health
scripts/vegaduta logs core                            # follow one service
scripts/vegaduta compose logs --since 30m core | grep -i error
scripts/vegaduta compose logs core | grep -E 'Migrating schema|Successfully applied'   # Flyway progress

Container exit code 137 is usually an OOM kill (core runs with -XX:+ExitOnOutOfMemoryError and exits rather than thrash); raise CORE_MEM_LIMIT or check scripts/vegaduta doctor. 143 is a normal stop.

Metrics and alerts

Enable with scripts/vegaduta up --full --profile observability. Grafana is on 127.0.0.1:3003 (admin password GRAFANA_ADMIN_PASSWORD in deploy/compose/.env) with dashboards for operations, LLM business metrics, alerts overview and user activity. Prometheus scrapes core over the internal compose network, so blocking /actuator at your edge does not break monitoring.

Rule files: vegaduta-alerts.yml (core down/absent, LLM error rate, Hikari pool exhaustion, JVM heap/GC, 5xx rate, workflow outbox backlog, Redis down/memory, governance fail-open, job-lock anomalies), vegaduta-platform-business-metrics.yml and vegaduta-latency-recording.yml. Alertmanager's one wired receiver is a webhook to core (POST /webhook/alerts/alertmanager, bearer PLATFORM_ALERTS_WEBHOOK_SECRET), which stores each alert in the control plane. Optional email via ALERT_EMAIL_TO + ALERT_SMTP_*; add Slack or PagerDuty receivers in deploy/alertmanager/alertmanager.yml for real paging.

Health endpoints

curl -s http://localhost:8080/actuator/health            # aggregate
curl -s http://localhost:8080/actuator/health/liveness   # is the JVM alive
curl -s http://localhost:8080/actuator/health/readiness  # can it serve (DB, Redis, tenant migrations)
curl -s http://localhost:8090/health                     # code-intel (profile code-intel)
curl -s http://localhost:8091/health                     # app-builder (profile builder)

Exposed actuator endpoints are exactly health, info and prometheus. The mail health indicator is disabled on purpose (an unconfigured SMTP would drag the aggregate to DOWN); enable it with MANAGEMENT_HEALTH_MAIL_ENABLED=true only where SMTP is always on.

Keep core's port behind a proxy

Core permits /actuator/prometheus without a JWT so Prometheus can scrape it. ActuatorEdgeGuardFilter returns 404 for that path when the request carries a proxy forwarding header (as every request through Caddy or an Ingress does), unless it presents the optional scrape token. So never publish core's port 8080 to the internet directly (Compose binds it to 127.0.0.1 by default); an edge rule blocking /actuator/* except /actuator/health is still good defence in depth. See Threat Model.

Langfuse LLM tracing

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT = http://langfuse-web:3000/api/public/otel/v1/traces
LANGFUSE_OTEL_AUTH_HEADER          = <Basic …>   (secret, from deploy/compose/.env)
TRACING_SAMPLING_PROBABILITY       = 0.1           (10% sampling)

A missing trace may simply not have been sampled. Set TRACING_SAMPLING_PROBABILITY=1.0 temporarily when chasing a bug, then put it back.
The join key: llm_call_log.trace_id is the Langfuse trace id — SQL gives cost/error detail, Langfuse gives the prompt and tool calls.
It fails silently. A wrong endpoint or stale auth header drops spans while the platform keeps working — absence of traces is not evidence of absence of calls.