Skip to main content

The closed loop

Keep runtime decisions, test evidence, and release checks connected. Each step leaves an object you can inspect.

Status: Beta Companion docs: Evals onboarding · Testing and CI · Receipts

The loop has a clear handoff​

A useful loop is not a promise that every change ships itself. It is a set of steps your team can inspect and control:

  1. Bind a baseline, scope, and Guardian.
  2. Govern calls and keep their results.
  3. Turn selected traffic or fixed transcripts into eval cases.
  4. Compare a candidate against a known run.
  5. Use a gate before deployment.

Technical workshop

Inspect the closed loop

Nothing advances until you do.

Step 1 of 5

Pin the baseline, goal, and scope.

The same scope ID keeps the goal and evidence tied together. A result may carry violations, corrections, routing context, and a receipt. Optional fields stay optional, so the loop must not depend on every result having every field.

Scope each Guardian​

Different jobs can use different Guardian versions while one control plane keeps the assignments visible.

Warden fleet

One control plane, independently scoped Guardians

Warden coordinating three scoped Guardians

Choose a Guardian to inspect its assignment.

Bound scope
refunds/*
Active version
guardian-refund-v3

The Warden coordinates policy and promotion. Each Guardian keeps its own scope, version, and receipts.

Use a scope for the job:

refunds = tr.scope(
    "refunds",
    goal="Resolve valid refunds without exposing customer data.",
)

response = refunds.client("openai").chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Refund order 4821."}],
)
result = tr.result(response)

The result records scope_id when the service includes it. That gives later queries and evals a stable way to group the work.

Keep versions addressable​

A candidate and an active Guardian are different release objects. Keeping their versions separate makes rollback and comparison possible.

Version vault

Every version stays addressable

Active: guardian-refund-v3

guardian-refund-v3

Serving production calls

guardian://refund/v3

tr.distill creates v4 beside v3. Promotion and rollback change the active pointer; they do not overwrite the vault.

Pin the Guardian on a scope when a run must use a known version:

refunds = tr.scope(
    "refunds",
    goal="Resolve valid refunds without exposing customer data.",
    guardian="guardian-refund-v3",
)

Do not treat a candidate as active just because it exists. Compare it, review the result, then choose whether to promote it.

Compare completed eval runs​

Run the same approved scenarios against the baseline and candidate. Then compare the completed run IDs:

comparison = tr.eval.compare(
    a="evr_release_12",
    b="evr_release_13",
)

print(comparison.body["deltas"]["regressed_scenarios"])
print(comparison.body["statistical"]["regression"])

Receipt comparison

Compare sealed runs, not floating summaries

Focused receipt
eh_01J7CAND9D3E
Observed delta
2 verdicts differ

Both sides retain their own contract, inputs, verdict leaves, and signature.

Comparison pairs cases by scenario_id. Both runs must be complete. Receipt fields in the comparison can be empty when one of the runs has no receipt.

Put a gate in front of release​

A gate runs a bound suite against thresholds. It keeps its run history and returns typed gate receipts.

Promotion gate

Choose a gate result to inspect the path

Active
guardian-refund-v3
Candidate
guardian-refund-v4

Run the gate or inspect rollback. No deployment state changes automatically.

Python can create a gate by calling tr.gate(...):

gate = tr.gate(
    name="refund-release-gate",
    suite_id="suite_refunds",
    guardian_id="refund-guardian",
    thresholds={"accuracy": 0.95},
    schedule="nightly",
)

for receipt in gate.runs(limit=10):
    if receipt.passed is False:
        print(receipt.failures)

TypeScript uses tr.gate.create(...):

const gate = await tr.gate.create({
  name: "refund-release-gate",
  suiteId: "suite_refunds",
  guardianId: "refund-guardian",
  thresholds: { accuracy: 0.95 },
  schedule: "nightly",
});

Gate receipts use the gr_ family. A gate can be created, listed, read, updated, paused, resumed, deleted, and inspected through its run history. Permissions and account capabilities are enforced by the server.

Keep CI simple​

Your CI job should run the test script and fail on the gate result:

name: Governance gate
on: [push, pull_request]

jobs:
  governance:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
      - run: pip install trinitite==0.0.3
      - run: python ci/governance_gate.py
        env:
          TRINITITE_API_KEY: ${{ secrets.TRINITITE_TEST_API_KEY }}
          TRINITITE_ENV: test

Where to go next​

  • Training turns reviewed calls into a new Guardian or model candidate.
  • Release gates checks a candidate before promotion or deployment.
  • Guardian covers version pinning, promotion, and rollback.
  • Evals onboarding creates the first fixed test.
  • Eval receipts explains verification and run comparison.
  • Schedule adds a readable cadence to recurring checks.
Last updated on