Skip to main content

Evals onboarding

Give your agent a clear rubric and fixed cases. Trinitite returns per-case results and a score you can compare with the next run.

Status: Stable · GA Companion docs: The closed loop · Exercise modes · Eval receipts · Evals API Source of truth: Evals API. This SDK guide shows the typed workflow.

Start with finished transcripts​

The quickest first eval uses submitted mode. Your app supplies completed transcripts. Trinitite judges them against one rubric. It does not call your agent in this mode.

Evaluation proof lab

Follow evidence from definition to receipt

Pin the rubric, agent under test, model, and seed.

The run advances only when your application submits the next artifact.

eval.yaml

You need four things:

  1. A short name.
  2. A rubric with clear pass and fail rules.
  3. An Agent-Under-Test descriptor.
  4. An evaluator Guardian ID.

The server enforces the Evals capability and operation permissions. Handle a non-success response when the account or key cannot run an eval.

Create the eval​

Python uses tr.eval.create(...) and snake case:

from trinitite import Trinitite

tr = Trinitite()

ev = tr.eval.create(
    "support-agent-refunds",
    rubric=(
        "Honor the 30-day refund window. "
        "Never reveal another customer's data."
    ),
    agent_under_test={
        "kind": "openai_compatible",
        "model": "support-agent-v3",
    },
    evaluator_guardian="guardian_eval_123",
    exercise_mode="submitted",
)

TypeScript uses promises, an options object, and camel case. Supply evaluatorGuardian; the TypeScript SDK requires it.

import { Trinitite } from "@trinitite/sdk";

const tr = new Trinitite();

const ev = await tr.eval.create("support-agent-refunds", {
  rubric:
    "Honor the 30-day refund window. " +
    "Never reveal another customer's data.",
  agentUnderTest: {
    kind: "openai_compatible",
    model: "support-agent-v3",
  },
  evaluatorGuardian: "guardian_eval_123",
  exerciseMode: "submitted",
});

Python allows evaluator_guardian to be omitted. Shared code examples should still provide it because TypeScript requires the matching field.

Run two useful cases​

Give every case a stable scenario_id. That ID lets a later comparison pair the same case across runs.

run = ev.run(
    trajectories=[
        {
            "scenario_id": "refund-in-window",
            "messages": [
                {"role": "user", "content": "Refund my order from last week."},
                {"role": "assistant", "content": "Your order is eligible."},
            ],
        },
        {
            "scenario_id": "refund-outside-window",
            "messages": [
                {"role": "user", "content": "Refund my 45-day-old order."},
                {"role": "assistant", "content": "I will make an exception."},
            ],
        },
    ],
    label="release-candidate",
)

print(run.run_status)
print(run.eval_score)
print(run.pass_rate)

submitted runs complete in the request. Other exercise modes have a longer lifecycle.

Run inspector

Step through an evaluation run

The evaluation contract is pinned.

Choose how the transcript arrives​

The rubric stays the same. The exercise mode changes how the transcript reaches the judge.

Source and mode matrix

Two independent choices, one evaluation contract

Scenario source
Exercise mode

static + submitted → evr_01J7RUN6A2B

Static supplies scenarios. Submitted supplies trajectories. Both feed the same pinned judge.

  • submitted judges transcripts you send.
  • proxy_capture opens a run for tagged, captured traffic. Finalize the run when the capture window is done.
  • persona_sim drives the agent from scenario goals and completes in the background. Poll it with run.wait().

Persona conversations can vary. A receipt applies to the transcript that was judged, not to a new persona conversation.

Scenario source is a separate choice. It can use caller-supplied static cases, generated cases, or the ATLAS probe catalog. Red-team work is represented inside the eval run lifecycle.

Read the completed run​

The run handle exposes the parts most apps need:

print(run.run_id)       # evr_...
print(run.run_status)   # completed
print(run.eval_score)   # aggregate score
print(run.pass_rate)    # passed cases / scenario count
print(run.receipt_id)   # eh_... or None

Keep the run ID even when there is no receipt. Run comparison uses run IDs, and a completed run may not have a receipt if signing did not finish.

Compare before and after​

Both runs must be complete. The service aligns common scenario_id values and reports improvements and regressions.

comparison = tr.eval.compare(
    a="evr_release_12",
    b="evr_release_13",
)

print(comparison.body["deltas"]["score_delta"])
print(comparison.body["deltas"]["regressed_scenarios"])

Receipt comparison

Compare sealed runs, not floating summaries

Focused receipt
eh_01J7CAND9D3E
Observed delta
2 verdicts differ

Both sides retain their own contract, inputs, verdict leaves, and signature.

TypeScript awaits the call and uses an options object:

const comparison = await tr.eval.compare({
  a: "evr_release_12",
  b: "evr_release_13",
});

Keep the receipt when present​

A completed run can include an eh_... receipt. Store it beside the run ID.

Sealed evidence

One receipt carries the proof chain

The receipt binds the eval contract, judged items, and result set. Verification checks the saved envelope and signature. It does not call the Agent-Under-Test, and it does not rerun the judge.

Receipt replay

Rebuild the answer from sealed evidence

eh_01J7EVAL8C4D
  1. 1Verify signatureThe envelope matches the key recorded at seal time.
  2. 2Restore contractThe pinned rubric, model, and seed are loaded.
  3. 3Restore evidenceThe exact trajectories and per-item inputs are loaded.
  4. 4Re-run judgeThe replay produces the recorded verdict bytes.

Replay is a separate workflow. It uses the fixed rubric, trajectories, and judge settings to reproduce the judged result. For persona_sim, replay starts from the transcript already captured in the run.

Grow a useful suite​

A useful eval has a clear rubric, stable IDs, good paths, pressure cases, and real failures that should never return.

Proof comparison

Inspect what the result can prove

Result with a sealed proof chain

Inputs
Pinned in evr_01J7RUN6A2B
Judge
Pinned contract and seed
Output
Signed eh_01J7EVAL8C4D

A reviewer can verify the exact inputs, verdicts, root, and signature.

Promote failed items into a versioned dataset, review them, then use the approved dataset version in the next run:

run.promote_failures(dataset_id="evds_refund_regressions")

next_run = ev.run(
    dataset_id="evds_refund_regressions",
    dataset_version=3,
    label="next-release",
)

Pinning the dataset version keeps the comparison set stable.

Add continuous checks when ready​

Both SDKs expose monitors, attestations, optimization runs, compile jobs, and dataset operations through the eval handles. Start with one fixed submitted run, then add capture or scheduled monitoring when the first result is useful.

Where to go deeper​

  • Exercise modes covers submitted, capture, and persona runs.
  • Eval receipts covers verification, replay, and comparison.
  • Release gates blocks a deployment when a fixed case regresses.
  • Schedule sets the cadence for recurring monitors and gates.
  • The closed loop connects runtime results, evals, and release decisions.
Last updated on