Evals
You shipped an agent. Everyone wants a number. Prove it is safe to ship, with a signed score a regulator can re-run, not a dashboard number that moves when the GPU gets busy.
Status: Stable · GA Companion docs: Evals onboarding · Red Team · Evals API · Eval Harness Source of truth: the backend Evals onboarding guide. This page is the buyer-side pitch; the SDK onboarding is the developer how-to.
The problem
You shipped an agent. Your board wants to know it is safe. Your auditor wants evidence you tested it. Your customers want an SLA. So you ran an eval. It scored 91. The judge was a frontier LLM, and the score moved three points when you re-ran it on a busy afternoon. Which 91 is the real one?
You cannot reproduce last quarter's run. You cannot prove the model under test was the one you claim. You cannot hand a regulator anything more durable than a screenshot of a dashboard. Every eval platform on the market is built for the pre-ship moment: a batch, a chart, a number that nobody can re-derive. The score is a vibe with a decimal point.
The wedge
Trinitite's judge returns the same bytes every time. The same model that powers our runtime governance also scores your agent, and it is batch-invariant. The same prompt, seed, model, and batch shape return the same bytes regardless of server load, time of day, or which GPU ran it. So an Eval Receipt is replayable bit for bit by a regulator or a counterparty.
Test your agent with our judge, the deliverable is a signed receipt
Same-answer proof
Run the pinned input in three environments
Pinned input
prompt:refund-policy · seed:42 · model:judge-v3- Pool Amorning · shared host
not run - Pool Bafternoon · busy host
not run - Pool Cevening · fresh host
not run
The deliverable
The deliverable is a signed eh_ receipt. A screenshot of a dashboard is not evidence. An eh_… id, re-verifiable by anyone with the rubric and the trajectories, is.
The receipt is the thing. Merkle-rooted over every per-item verdict, KMS-signed by Trinitite,
and replayable by anyone. A regulator can re-run the judge and get the same bytes. That is
what converts "trust us, the agent is safe" into "re-run receipt eh_… and you will get the
same answer."
- Merkle-rooted. Every per-item verdict is a leaf. Tampering with any one is detectable.
- KMS-signed. Trinitite's signing key attests to the bytes. The verify path re-walks the signature.
- Replayable. A regulator or a counterparty with the receipt can re-run the judge over the same rubric, trajectories, and judge config and get the same bytes. That is the wedge, surfaced as an artifact.
- Comparable. Two runs align by
scenario_id, run a paired statistical test on the pass/fail flips, and return the deltas. "Did the new prompt regress" is a signed answer, not an eyeball.
Three ways to drive your agent
Pick how the agent under test gets exercised. One line each. The full how-to lives in the SDK onboarding.
| Mode | How the agent runs | When to pick it |
|---|---|---|
| Submitted | You post finished transcripts. We judge them. | First run. Cheapest, hermetic, CI-friendly. |
| Proxy capture | The agent runs through a Trinitite perimeter in eval mode. Real captured traffic is judged. | Production-shape testing. Judge what your agent actually did. |
| Persona sim | A digital human drives the agent multi-turn, adversarial by default. | Generative testing and red-team. Automated adversarial coverage. |
Point the same judge at MITRE ATLAS red-team probes and the run crosswalks to a separate signed ATLAS attestation, in addition to the eh_ receipt. One adversarial run yields both.
The value, by role
| Role | What this unlocks |
|---|---|
| Head of AI / CTO | A reproducible quality gate on the agent you ship. Re-run on every prompt or model change and diff signed verdicts, not noise. |
| AI product owner | Find your agent's best prompt and model config with a reproducible search, instead of climbing against a judge that moves under you. |
| ML / platform engineer | A versioned dataset and signed receipt loop that drops into CI. Run the suite, assert the score, compare against the last green run. |
| CISO / Compliance | Your agent's behavioral compliance is a signed value, mapped to controls. Evidence, not a dashboard screenshot. |
| Vendor management | Hold a third-party agent to a signed score you can re-verify, not the vendor's marketing benchmark. |
| Internal audit | Reproducible, signed test evidence drawn against the real agent, re-verifiable by an external partner. |
Go build it
The four-line hero, the JSON envelope field by field, what a strong rubric looks like, and the composable top (persona_sim red-team, continuous evals, regulation-to-eval, reproducible prompt optimization) live in the developer onboarding.
→ Evals onboarding: the four-line hero to a signed receipt. → Red Team: named adversarial patterns plus downloadable fixtures. → Evals API: the full HTTP contract. → Eval Harness: scale evals across cohorts and model swaps.