Eval receipts, replay, and comparison

A completed Eval Run normally gives you a signed eh_ receipt tied to the rubric, judged items, and result set. Keep the run ID for operations and the receipt ID for evidence. receipt_id is optional because a run can complete even if receipt signing fails.

Status: Beta Companion docs: Exercise modes · Continuous evals · Evals onboarding Source of truth: Evals API. Receipt verification uses the Eval Harness API.

Know the identifiers

These IDs serve different jobs:

Prefix	Object	Use it for
eval_	Eval definition	Reuse a rubric, agent descriptor, exercise mode, and scenario source.
evr_	Eval Run	Fetch one execution and compare it with another run.
eh_	Eval receipt	Verify the signed evidence for a completed run.
evm_	Eval Monitor	Read or pause a recurring capture schedule.
ema_	Monitor receipt	Prove one rolled monitor window and its drift result.
evow_	Winner receipt	Identify the signed winner from an optimization run.

eh_, ema_, and evow_ are receipt families. The other prefixes identify resources and executions.

Read the receipt from a run

In submitted mode, the run completes in the initial call:

from trinitite import Trinitite

tr = Trinitite(env="prod")

ev = tr.eval(
    "support-agent-release",
    rubric="Honor the 30-day refund window. Never reveal another customer's data.",
    agent_under_test={
        "kind": "openai_compatible",
        "model": "support-agent-v3",
    },
    exercise_mode="submitted",
)

run = ev.run(
    trajectories=[
        {
            "scenario_id": "refund-in-window",
            "messages": [
                {"role": "user", "content": "Refund my order from last week."},
                {"role": "assistant", "content": "Your order is eligible for a refund."},
            ],
        }
    ]
)

print(run.run_id)       # evr_...
print(run.receipt_id)   # eh_...
print(run.eval_score)   # aggregate score from 0 to 100
print(run.pass_rate)    # passed scenarios divided by scenario count

The typed accessors read the completed run envelope:

• run.run_status reads status.
• run.eval_score reads aggregate.score.
• run.pass_rate derives aggregate.pass_count / scenario_count.
• run.receipt_id reads the signed eh_ receipt ID.

For an open proxy_capture run, finalize() returns the completed response. Read its body directly, or call run.get() afterward to refresh the handle:

final = run.finalize()
receipt_id = final.body.get("receipt_id")

run.get()
print(run.run_status, run.receipt_id)

A persona_sim run auto-finalizes after its background driver finishes. Call run.wait(), which refreshes the handle before returning:

run.wait()
print(run.run_status, run.receipt_id)

What the receipt proves

Each completed run records per-scenario verdicts and an aggregate. The receipt binds that result set with Merkle roots and a signature.

The receipt lets a verifier detect changes to a result item and check that the signed envelope is valid. Verification is available at:

GET /v1/audit/eval-harness/receipts/:receipt_id/verify

Verification recomputes the attestation hash from the stored receipt and checks its signature. It does not rerun the judge.

Replay is separate. To reproduce a judged result, use the same rubric, fixed trajectories, and judge configuration. Replay does not call the Agent-Under-Test again and does not regenerate a persona conversation.

That distinction matters:

• A submitted transcript is already fixed.
• A captured transcript is fixed after the window closes.
• A persona-generated transcript can vary when generated again, but the receipt remains replayable for the transcript that the run judged.

Compare completed runs

Use run IDs, not receipt IDs, with tr.eval.compare:

comparison = tr.eval.compare(
    a="evr_release_12",
    b="evr_release_13",
)

deltas = comparison.body["deltas"]
statistical = comparison.body["statistical"]

print(deltas["score_delta"])
print(deltas["regressed_scenarios"])
print(deltas["improved_scenarios"])
print(statistical["regression"])

The endpoint treats a as the champion and b as the challenger. Numeric deltas are calculated as b - a.

Comparison behavior is precise:

• Both runs must be completed.
• Scenarios are paired by scenario_id.
• regressed_scenarios passed in A and did not pass in B.
• improved_scenarios did not pass in A and passed in B.
• statistical.regression is true only when the result is significant and regressions outnumber improvements.
• statistical.small_n is true when the discordant pair count is below 20.
• deterministic, items_root_match, and answers_root_match can be null when a receipt is absent.
• Receipt-derived accuracy, refusal-rate, and error-rate deltas can also be null. The run score and paired verdict comparison still come from the completed Eval Runs.

Make comparisons useful

Keep the paired input stable:

1. Give every case a durable scenario_id.
2. Use the same approved dataset items for both runs.
3. Pin dataset_version so later edits do not change the comparison set.
4. Change the agent configuration you intend to test.
5. Compare the completed run IDs.

The paired result answers whether behavior changed on shared cases. It does not treat unmatched scenarios as paired evidence.

Next steps

• Read Exercise modes to choose how transcripts reach the judge.
• Read Continuous evals to create recurring eh_ and ema_ evidence.
• Open the Evals API for the full comparison envelope.