AI quality monitoring overview
Evaluate-not-assert monitoring for chatbots, copilots and agents.
aillmevaluationoverview
Monitoring AI features
AI features — chatbots, copilots, RAG assistants, agents — can't be monitored with assert status == 200. A hallucinated answer, an unsafe response, a leaked secret, or the wrong tool call all return 200 OK. Their failures are semantic. VerOps shifts from assert to evaluate.
The loop
Probe the AI → capture answer, tools, citations → evaluate with an LLM-as-judge plus deterministic checks → assert on the scores → track, diagnose, act.
Where it runs
- In a Workbook (AI Probe → AI Evaluate) as a synthetic monitor or CI gate.
- As an Eval Suite over a golden dataset for regression gating.
- Continuously over production traffic as VerOps Sentinel.
See Evaluators and scoring, Eval suites and CI gates, and Model compare and KB grounding.