Eval suites and CI gates

The AI Eval Suite runs the judge over a golden dataset and gates on the aggregate pass-rate — an AI regression gate. Trigger it on any prompt or model change.

Define a dataset

mode: evaluate            # or "probe" to call your agent per case
passThreshold: 0.8        # require >= 80% of cases to pass
criteria: [correctness, faithfulness, relevance, safety]
dataset:
  - { input: "Returns within 30 days with a receipt.", reference: "30 days with receipt",
      context: "Returns: 30 days, receipt required." }
  - { input: "We ship to 40 countries.", reference: "Ships to 40 countries",
      context: "Shipping: 40 countries." }

You can also import a CSV (header input|prompt, reference, context, query) instead of hand-writing cases.

Red-team preset

One click loads an adversarial dataset (jailbreak, prompt injection, secret-extraction, phishing, PII) scored with the safety evaluator at a strict threshold. Point it at your agent and schedule it to catch safety regressions.

Use it as a CI gate

# block the deploy if AI quality regresses
curl -fsS -X POST https://app.verops.io/api/workflow-executions/run \
  -H "Authorization: Bearer $VEROPS_TOKEN" \
  -d '{"workflowId": <suite-workbook-id>, "triggeredBy": "ci"}'
# the suite passes only if passRate >= passThreshold