Eval Harness
Scale evaluation across cohorts, Guardian versions, policy versions, and model swaps. Run a parallel eval matrix, compare safety scores side-by-side, and gate promotions on reproducible, signed results.
What it is
Eval Harness is the orchestration layer on top of Evals. Where Evals runs one suite against one target, the Harness runs an eval matrix — the same suite × multiple Guardian versions × multiple policy versions × multiple underlying models — in parallel, then compares the resulting safety scores side-by-side. It is how you answer "did swapping GPT-5 for Claude regress our PII Guardian?" with a signed, reproducible delta instead of a hunch.
Why it matters
- Model swaps are the highest-risk, most-frequent change. The Harness makes them a measured, gateable event instead of a hope-and-pray.
- Cohort-aware. Evaluate against production-derived cohorts, not just synthetic fixtures, so the score reflects real traffic.
- Reproducible matrices. Every cell of the matrix is a signed run; the whole matrix is itself an artifact you can re-run and diff.
- Promotion gate. A Guardian only promotes when its score clears your threshold across the matrix — automated, auditable, and reversible.
How it works
- Define the matrix — suite, Guardian versions, policy versions, models, cohorts.
- Run in parallel — each cell is a signed, reproducible eval run.
- Compare — side-by-side safety scores with per-control and per-cohort breakdowns.
- Gate — promote only when the matrix clears your thresholds; otherwise, the deltas point at the failing cell.
Get started
- Define your cohorts and suites in the Evals API.
- Drive the matrix via the Eval Harness API.
- Wire the promotion gate into your CI / Guardian lifecycle.
→ Evals — the per-run measurement surface. → Eval Harness API — matrices, cohorts, promotion gates. → Guardians — what you're evaluating.