Evals API

Base path: /v1/evals Auth: Session token or API key — see Authentication. Permissions: evals:create, evals:read, evals:run.

The Evals API tests your agent (the Agent-Under-Test) and scores it with Trinitite's deterministic SLM judge, minting a signed, replayable Eval Receipt. See the Evals product page.

Don't confuse this with the Eval Harness API, which records signed receipts for benchmark suites run against a model. An Eval Run internally mints its receipt through the same eh_… primitive.

Concepts

• Eval (eval_…) — a definition: exercise_mode + scenario_source + evaluator_guardian + rubric + agent_under_test.
• Eval Run (evr_…) — one execution; produces per-item verdicts, an aggregate score, and a signed receipt.
• exercise_mode — submitted (judge supplied trajectories), proxy_capture (judge real perimeter traffic), persona_sim (drive the agent with a non-deterministic persona; adversarial = red-team).
• Judge — IInferencePort.govern({ verdict_profile: 'evaluator' }), deterministic, temperature 0. Verdict is passed | failed + violated_controls[] + confidence, plus the multi-dimensional scorecard (rubric_scores[] + overall_score).

Create an eval

POST /v1/evals — evals:create

{
  "name": "refund-agent-safety",
  "exercise_mode": "persona_sim",
  "scenario_source": "ts_01J9X…",
  "evaluator_guardian": "gov_evaluator_v2",
  "rubric": { "…": "…" },
  "agent_under_test": { "endpoint": "https://your.agent/v1/chat", "model": "…" }
}

Returns 201 Created with the eval_… id.

List & fetch

• GET /v1/evals — evals:read. Cursor-paginated list.
• GET /v1/evals/:id — fetch one definition.

Start a run

POST /v1/evals/:id/runs — evals:run

Queues a run. Returns 202 Accepted with { run_id, status: "queued" }. Poll GET /v1/evals/runs/:run_id for terminal state (completed / failed / cancelled).

Fetch & compare runs

• GET /v1/evals/runs/:run_id — the run, its per-item verdicts, aggregate score, and signed eh_… receipt id.
• GET /v1/evals/runs/compare?a=:run_a&b=:run_b — side-by-side delta between two runs (delegates to the Eval Harness comparison endpoint).

Response types

{
  "run_id": "evr_01J9X…",
  "eval_id": "eval_01J9X…",
  "status": "completed",
  "started_at": "…",
  "completed_at": "…",
  "items": [
    { "scenario_id": "scn_…", "verdict": "failed", "violated_controls": ["…"], "confidence": 0.93 }
  ],
  "scorecard": { "overall_score": 0.872, "rubric_scores": [ { "…": "…" } ] },
  "receipt_id": "eh_01J9X…"
}

Eval datasets

GET / POST /v1/evals/datasets — manage reusable scenario datasets (Phase 5 surface).

Errors

Surface-specific codes layer on the shared error envelope:

HTTP	error.code	When
403	forbidden	Missing evals:run permission.
409	conflict	Run already terminal.
422	unprocessable_entity	Agent-Under-Test endpoint unreachable or mis-specified.

---

→ Evals — the product overview. → Eval Harness API — benchmark receipts & comparison. → Verdict vocabulary — the judge's verdict surface.