Eval Harness API

Base path: /v1/audit/eval-harness Auth: Session token or API key — see Authentication. Permissions: audit:run (record), audit:read (all reads).

The suite-agnostic receipt envelope. A runner supplies the per-item inputs/outputs/verdicts for a benchmark or eval suite run against a specific model reference; the service mints a Merkle-rooted, KMS-signed EvalHarnessReceipt an external auditor can re-verify and re-run. See the Eval Harness product page.

Different from the Evals API (/v1/evals): that one exercises a customer's agent and internally mints eh_… receipts through this same primitive. This one records a benchmark run against a model.

Surface map

Record a run

POST /v1/audit/eval-harness/record — audit:run

{
  "suite": "MMLU",
  "model_reference": "openai/gpt-5-2026",
  "guardian_version": "gov_evaluator_v2",
  "policy_hash": "sha256:7f3a…",
  "items": [
    { "input": "…", "output": "…", "verdict": "passed", "label": "correct" }
  ],
  "metadata": { "…": "…" }
}

Returns 201 Created with the eh_… receipt id and the Merkle root.

Verify a receipt

GET /v1/audit/eval-harness/receipts/:id/verify — audit:read

Recomputes the Merkle hash and re-checks the KMS signature; returns { verified: true, recomputed_root, matched_signature }. This is the auditor-facing verification call.

Fetch & list receipts

• GET /v1/audit/eval-harness/receipts/:id — full receipt with per-item entries.
• GET /v1/audit/eval-harness/receipts — cursor-paginated list, filterable by suite, model_reference, guardian_version.

Compare receipts (champion / challenger)

GET /v1/audit/eval-harness/compare?a=:eh_a&b=:eh_b — audit:read

Side-by-side delta between two receipts: per-suite and per-item verdict diffs, score delta, and the model_reference / guardian_version / policy_hash differences. This is how you answer "did swapping models regress the benchmark score?" with a signed, reproducible delta.

Response types

{
  "receipt_id": "eh_01J9X…",
  "suite": "MMLU",
  "model_reference": "openai/gpt-5-2026",
  "guardian_version": "gov_evaluator_v2",
  "policy_hash": "sha256:7f3a…",
  "items": [ { "input": "…", "output": "…", "verdict": "passed" } ],
  "merkle_root": "e3b0c4…",
  "signature": { "kms_key_id": "…", "sig": "…" },
  "created_at": "2026-05-01T12:00:00Z"
}

Errors

HTTP	error.code	When
403	forbidden	Missing audit:run / audit:read.
404	not_found	Receipt id does not exist.
409	conflict	Compare requested on non-comparable receipts (different suite).

---

→ Eval Harness — the product overview. → Evals API — evaluate your own agent. → Errors, rate limits & conventions — shared contract.