Skip to main content

Models and routing

Use the model that fits the job. Trinitite can compare choices on your traffic, route each call within your limits, and fail over when the first choice is unavailable.

Status: Beta Companion docs: Training · FinOps · SDK observability Source of truth: Chat API · Proxy API · Inference service API.

Three surfaces, one model lifecycle​

  • tr.models tells you what your data says.
  • tr.router acts on that data during a wrapped call.
  • tr.inference turns a trained artifact into a governed model asset.

Ask for a recommendation​

recommendation = tr.models.recommend(
    scope="refund-flow",
    objective="balanced",
    constraints={
        "governance_risk_max": "medium",
        "latency_p95_ms": 2000,
    },
)

print(recommendation.winner)
print(recommendation.runner_up)
print(recommendation.reasoning)
print(recommendation.estimated_savings_usd)

The recommendation uses the scope's captured calls and evals. It is advice, not a route change.

Compare models on the same cases​

report = tr.models.compare(
    models=[
        "openai:gpt-4o",
        "anthropic:claude-3-5-sonnet",
        "trinitite:refund-slm-v3",
    ],
    dataset="refund-cases@v3",
    metrics=["accuracy", "cost", "latency", "governance_risk"],
)

print(report.winner)
print(report.by_metric)

Pin the dataset version so every model sees the same cases.

Route each call within clear limits​

Put the policy on a scope:

refunds = tr.scope(
    "refund-flow",
    goal="Resolve valid refunds without exposing customer data.",
    route="auto",
    failover=[
        "anthropic:claude-3-5-sonnet",
        "openai:gpt-4o",
        "trinitite:refund-slm-v3",
    ],
    cost_ceiling_usd_per_call=0.50,
    governance_risk_max="medium",
)

Then make the normal wrapped call:

response = refunds.client("openai").chat.completions.create(
    model="auto",
    messages=messages,
)
result = tr.result(response)

print(result.routed_model)
print(result.routing_reason)
print(result.failover_triggered)
print(result.cost_usd)

The result records what served the call and why. Your app can inspect the choice without rebuilding the scoring logic.

Preview a route without making the call​

preview = tr.router.dry_run(
    scope="refund-flow",
    input=messages,
)

print(preview.routed_model)
print(preview.estimated_cost_usd)
print(preview.failover_plan)

Use a typed policy when the route needs a named primary and explicit trip rules:

policy = tr.router.policy(
    primary="anthropic:claude-3-5-sonnet",
    fallback=["openai:gpt-4o", "trinitite:refund-slm-v3"],
    strategy="cost_then_accuracy",
    cost_ceiling_usd_per_call=0.50,
    governance_risk_max="medium",
    failover_on=["outage", "rate_limit", "over_budget", "governance_block"],
)

Graduate a trained model​

Training makes an artifact. Inference makes it an addressable asset:

job = tr.train.model(scope="refund-flow")
job.wait(timeout=1800)

asset = tr.inference.register(
    training_run_id=job.run_id,
    name="refund-slm-v3",
    guardian="guardian-refund-v3",
)
tr.inference.bind(asset, scope="refund-flow")

Check readiness before training or registration when you want a cost and data estimate:

plan = tr.inference.graduation(scope="refund-flow")
print(plan.ready)
print(plan.captured_traffic)
print(plan.estimated_cost_savings_usd)

Deploy with a passing gate receipt when the release requires one:

tr.inference.deploy(
    asset,
    gate_receipt_id="gr_01J7GATE4F9C",
)

A served model stays connected to its training run, scope, Guardian, and factsheet in the graph.

Where to go next​

  • FinOps groups spend by scope, model, and cost center.
  • Training creates a Guardian or model candidate.
  • Release gates blocks a model deployment when a fixed case regresses.
Last updated on