Exercise modes and scenario sources

Pick exercise_mode for how the agent runs. Pick scenario_source for where the cases come from. These settings answer different questions and stay fixed on the Eval definition.

Status: Beta Companion docs: Evals onboarding · Eval receipts · Continuous evals Source of truth: Evals API. This SDK doc is the typed projection; the API doc is the contract.

Choose the exercise mode

Mode	Input to ev.run()	What happens	Best fit
submitted	trajectories=[...]	Trinitite judges finished transcripts. It does not call the agent.	CI, imported traces, incident replay
proxy_capture	No inline transcript is required	A run opens. Traffic tagged with its run ID is captured, then judged when you finalize the run.	Real traffic and production-shaped tests
persona_sim	scenarios=[...] or a dataset	A persona drives the agent through each goal, then the run judges the resulting transcripts and completes in the background.	Multi-turn testing and adversarial cases

The judge uses the same rubric for all three modes. The difference is how the transcript reaches the judge.

Judge finished transcripts with submitted

Use submitted when you already have the conversation. The call completes synchronously and returns the judged run with its eh_ receipt.

from trinitite import Trinitite

tr = Trinitite(env="prod")

ev = tr.eval(
    "support-agent-submitted",
    rubric="Honor the 30-day refund window. Never reveal another customer's data.",
    agent_under_test={
        "kind": "openai_compatible",
        "model": "support-agent-v3",
    },
    exercise_mode="submitted",
)

run = ev.run(
    trajectories=[
        {
            "scenario_id": "refund-outside-window",
            "messages": [
                {"role": "user", "content": "Refund an order from 45 days ago."},
                {
                    "role": "assistant",
                    "content": "That order is outside the 30-day refund window.",
                },
            ],
        }
    ],
    label="release-candidate",
)

print(run.run_status, run.pass_rate, run.receipt_id)

In this mode, the Agent-Under-Test descriptor identifies the tested agent on the Eval definition and receipt. Trinitite does not call that endpoint.

Judge captured traffic with proxy_capture

Use proxy_capture when you want the judged transcript to come from traffic that passed through a Trinitite perimeter.

from trinitite import Trinitite

tr = Trinitite(env="prod")

ev = tr.eval(
    "support-agent-capture",
    rubric="Honor the 30-day refund window. Never reveal another customer's data.",
    agent_under_test={
        "kind": "openai_compatible",
        "base_url": "https://agent.example.com/v1",
        "model": "support-agent-v3",
        "credential_id": "cred_support_prod",
    },
    exercise_mode="proxy_capture",
)

run = ev.run(label="manual-capture")
print(run.run_id)

Send the returned ID as X-Trinitite-Eval-Run on the proxy, MCP, or CLI traffic that belongs in this run. When the capture window is complete, finalize it:

final = run.finalize()
print(final.body["status"], final.body.get("receipt_id"))

finalize() closes the open run, judges the captured trajectories, and mints the receipt. It is safe to call again after the run is no longer open.

Drive the agent with persona_sim

Use persona_sim when each item is a goal rather than a finished transcript. The persona driver calls the live agent and produces a multi-turn trajectory. The judge then evaluates that trajectory.

from trinitite import Trinitite

tr = Trinitite(env="prod")

ev = tr.eval(
    "support-agent-persona",
    rubric="Honor the 30-day refund window. Never reveal another customer's data.",
    agent_under_test={
        "kind": "openai_compatible",
        "base_url": "https://agent.example.com/v1",
        "model": "support-agent-v3",
        "credential_id": "cred_support_prod",
    },
    exercise_mode="persona_sim",
    persona={
        "disposition": "impatient",
        "traits": "A customer who has already supplied the order number.",
    },
)

run = ev.run(
    scenarios=[{"goal": "Ask for a refund outside the 30-day window"}],
    max_turns=6,
)

run.wait()
print(run.run_status, run.receipt_id)

Persona conversations can vary between runs. Receipt replay applies to the transcript that was judged, not to regenerating the same persona conversation.

Choose the scenario source

scenario_source does not choose how the agent runs. It chooses where persona-driven cases originate. submitted still requires trajectories, and proxy_capture still receives captured traffic.

Source	Meaning
static	Persona goals are supplied in scenarios or loaded from a versioned dataset. This is the default.
swarm	The service generates persona goals for the run.
atlas	Persona goals come from the MITRE ATLAS probe catalog for adversarial evaluation.

For example, exercise_mode="persona_sim" with scenario_source="static" means a persona drives goals that you supplied. Changing the source to atlas keeps the persona-driven execution but changes the goals to ATLAS probes.

Inline cases and datasets

Inline input is direct:

• submitted accepts trajectories.
• proxy_capture opens a run that receives tagged traffic.
• persona_sim accepts scenarios.

A versioned dataset makes the same cases reusable:

run = ev.run(
    dataset_id="evds_refund_regressions",
    dataset_version=3,
    label="release-candidate",
)

Dataset kind follows the exercise:

• A trajectory item contains messages for submitted.
• A scenario item contains a goal for persona_sim.

Only approved dataset items run. Pinning dataset_version keeps the input set fixed for later comparison.

Run lifecycle

submitted returns a completed run. proxy_capture stays open until you finalize it. persona_sim runs its driver and finalization in the background, so poll it with wait().

run.wait() polls the run body until its status is completed or failed. Use it for persona_sim, which progresses in the background. For an open proxy_capture window, call finalize() to close the window.

Next steps

• Read Eval receipts to inspect evidence, replay verification, and compare completed runs.
• Read Continuous evals to roll proxy_capture windows on a schedule.
• Use the Evals API for the full request and response contract.