Docs/Testing & Quality/Test Strategy

Test Strategy

How the platform is tested, layer by layer, and where each layer runs. The strategy is defined by the configuration and workflows in the repository rather than by a single test-plan document (none exists under docs/testing). Detail pages: End-to-End Suites and Quality Gates & CI Checks.

What runs automatically

ci.yml (summary check CI OK) and static-gates.yml run on every pull request; e2e.yml runs nightly and on demand. Each layer also has a local make target (make gates, make test, make lint, make compose-validate, make helm-lint).

Layers

flowchart TB L1["Static gates: Flyway lint, price and file-size ratchets, skip expiry, bean/entity-name guards, gitleaks"] L2["Unit tests: Maven surefire (core), vitest (web, web-admin), pytest (sidecar), script harness (code-intel)"] L3["Boot check: full jar against a throwaway Postgres before any prod image"] L4["End-to-end: Playwright against a real stack, no mocking"] L5["Eval: golden-dataset runner in core/quality, run by hand"] L6["Load: k6, staging only"] L1 --> L2 --> L3 --> L4 L4 --> L5 L4 --> L6

Layer	Tooling as configured	Where it lives
Static gates	Shell scripts and gitleaks	`scripts/ci/`, `.github/workflows/static-gates.yml`, `make gates`
Core unit / integration	Maven reactor, surefire with a documented JVM-flag mitigation for random fork crashes on JDK 21+; maintainers build locally with JDK 24	Modules under `core/` (563 `*Test.java` files at authoring time; a file count, not a test count)
Web unit	Vitest 4 + React Testing Library, jsdom; includes only `app/*/.test.{ts,tsx}`	`web/vitest.config.mts`, `web/vitest.setup.ts`; `npm test` runs `vitest run`
Python services	Ruff lint; `pytest` for sidecar; code-intel uses a script harness, not pytest	`services/sidecar/tests/`, `services/code-intel/tests/`; `make test-python`, `ci.yml` “python” job
Lint + build	`npm run lint` and `npm run build` for `web/` and `web-admin/`	`ci.yml` jobs `web`, `web-admin`; `make lint`
End-to-end	Playwright against a real running stack	`tests/e2e/` (run by `e2e.yml`) and `web/e2e/`; see E2E Suites
Agent eval	Golden-dataset runner `com.agenticai.core.quality.eval.EvalRunner`; no workflow runs it	`core/quality/` (see eval-harness.md)
Infra validation	helm lint/template and kubeconform; `docker compose config` on every compose file	`ci.yml` jobs `helm` and `compose`; `make helm-lint`, `make compose-validate`
Load / latency	k6 profile with a documented acceptance rule; staging only	`tests/load/`

Principles visible in the configuration

Real stack, no mocking at the E2E tier. The Playwright suite header states there is no mocking layer anywhere; a fresh tenant is signed up per test through POST /api/signup.
Tests cannot see boot-time failures. Several recorded incidents (duplicate bean names, missing @Autowired, Hibernate validation) passed compile and unit tests and failed at Spring context start. The mitigations are the bean-name gate and booting the jar: a full-reactor jar boot against a throwaway deploy-postgres before building images, with all migrations applied and Started CoreApplication logged. The rule: boot the jar before any prod image (see Build, Test & Verify).
Ratchets over big-bang fixes. File size and hardcoded prices are frozen at a baseline that may only go down; see Quality Gates.
Skips must expire. Tracked-gap skips carry an expires:YYYY-MM-DD token enforced by scripts/ci/skip-expiry.sh; opt-in credential skips do not.
Load tests never target production. The k6 runbook forbids the production host unless a human explicitly approves ALLOW_PROD=1, and requires an allow-listed load-test tenant so no billed LLM turns occur.
Prod verification is a separate step. Distinguish compiled, tested, deployed-to-staging and deployed-to-prod. A production e2e probe should use a dedicated tenant kept for that purpose.

Environment prerequisites for the heavy tiers

E2E needs Postgres (pgvector), Valkey, Keycloak, sidecar, core (profiles security,knowledge,mcp,e2e) and the web console; the e2e profile raises the signup rate limit (default 5 per IP per window).
The CI core job starts Postgres with pgvector and Apache AGE for the Maven verify; e2e.yml builds the compose stack with throwaway CI-only secrets.
Local Maven: JDK 21+ for core (maintainers use 24), one Maven build at a time; the parent POM's coverage profile is opt-in.

Not documented in the repository

Coverage percentages, total test counts per module, pass rates, flaky-test statistics and a coverage threshold are not recorded. JaCoCo and vitest coverage are advisory with no threshold. The eval golden-dataset pass criteria are not summarised on this page; see core/quality/eval-harness.md.

Evidence & references

.github/workflows/ci.yml, static-gates.yml, e2e.yml, codeql.yml (on: blocks and job lists); Makefile.
web/vitest.config.mts, web/package.json; web/e2e/README.md; core/pom.xml (surefire and coverage profile).
tests/load/README.md (k6 acceptance rule, ALLOW_PROD).