Skip to main content

Testing and CI

Test your application logic with forced verdicts, then use scheduled gates to measure a suite against release thresholds.

Status: Beta Companion docs: Test suites API · Evals · Authentication Source of truth: the tested force_verdict client path and the tr.gate SDK surface.

Test every verdict branch​

Forced verdicts are allowed only with test and dev clients. A production client raises ForcedVerdictInProdError before sending the request.

from trinitite import Trinitite

tr = Trinitite(env="test")

def governed(verdict: str):
    resp = tr.client("openai").chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": "test input"}],
        force_verdict=verdict,
    )
    return tr.result(resp)

def test_blocked_branch():
    result = governed("blocked")
    assert result.verdict == "blocked"

def test_corrected_branch():
    result = governed("corrected")
    assert result.verdict == "corrected"

def test_human_review_branch():
    result = governed("HiTL")
    assert result.verdict == "HiTL"

The supported SDK verdicts are passed, corrected, blocked, masked, HiTL, and failed.

Assert blocked calls as results or exceptions​

By default, a blocked call returns a response that can be normalized with tr.result(...).

result = governed("blocked")
assert result.verdict == "blocked"

Use raise_on_blocked=True when exceptions fit your application flow.

from trinitite import GovernanceBlockedError

client = tr.client("openai", raise_on_blocked=True)

try:
    client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": "test input"}],
        force_verdict="blocked",
    )
except GovernanceBlockedError as exc:
    assert exc.result_id is None or isinstance(exc.result_id, str)

Create a scheduled gate​

tr.gate(...) creates a persisted gate. A scheduler runs the bound suite at the requested cadence.

gate = tr.gate(
    name="refund-release-gate",
    suite_id="suite_refunds",
    guardian_id="refund-guardian",
    thresholds={"accuracy": 0.95},
    schedule="nightly",
)

print(gate.gate_id)

Inspect typed run receipts after ticks complete:

runs = gate.runs(limit=10)
for receipt in runs:
    print(receipt.gate_run_id, receipt.passed, receipt.score)
    if receipt.passed is False:
        print(receipt.failures)

Gate receipts use the gr_ family. A run can also expose drift, regression, PSI, ingested scenario count, and a linked test run_id.

Require a passing gate at deployment​

Deployment and A/B test calls accept a gate receipt id. A blocking 409 becomes DeploymentBlockedError.

from trinitite import DeploymentBlockedError

served = tr.inference.get("sm_refund_v4")

try:
    deployed = served.deploy(gate_receipt_id="gr_pass_123")
except DeploymentBlockedError as exc:
    print(exc.gate_receipt_id)
    for failure in exc.failures:
        print(failure)
    raise

The receipt check is opt-in. If gate_receipt_id is omitted, the SDK omits it from the deployment request.

GitHub Actions​

name: Governance tests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install trinitite pytest
      - run: pytest
        env:
          TRINITITE_API_KEY: ${{ secrets.TRINITITE_TEST_API_KEY }}
          TRINITITE_ENV: test

Keep production keys out of test jobs. The key prefix check and forced-verdict guard fail before a request can use the wrong environment.

Next steps​

Last updated on