AI quality monitoring overview
Evaluate, don't assert: why AI features need semantic monitoring, the three surfaces that share one judge (Sentinel in production, Workbook AI nodes, eval suites), how Sentinel works end to end, and the three design rules behind it.
Why AI features need a different kind of monitoring
A chatbot that hallucinates, a copilot that leaks a customer's card number, an agent that calls the wrong tool or publishes the wrong campaign — every one of those returns 200 OK in under a second. Their failures are semantic: the request succeeded and the answer was wrong. Nothing in latency, error rate or a status code sees it.
VerOps moves from assert to evaluate: capture what the AI produced, judge it against a rubric with an independent model plus deterministic checks, and treat the scores as the signal you monitor, alert on and act from.
Three places the same engine runs
| Surface | What it is for | Where it lives |
|---|---|---|
| VerOps Sentinel | Continuous quality, safety and cost monitoring of AI agents in production: every run is evaluated, breaches open incidents with root-cause analysis, monitors watch for drift, and a closed catalogue of recovery actions can be recommended — or, within limits you set, applied. | Sentinel in the left navigation. This category. |
| Workbook AI nodes | Probe an AI endpoint and evaluate its answer inside a pipeline — as a synthetic monitor on a schedule, or as a gate in CI. | Workbooks → AI nodes; scoring rules in Evaluators and scoring. |
| Eval suites | Run the judge over a golden dataset and gate a deploy on the pass rate — the AI regression test. | Eval suites and CI gates. |
How Sentinel works, end to end
- Your agent emits events for each run — the prompt, the steps it took, the final output — to Sentinel's ingest endpoint or through MCP. Connect an agent.
- Sentinel evaluates the run once, about 90 seconds after its last event, on the dimensions its profile selects — five base dimensions every agent gets, plus domain ones for marketing, coding, support, RAG or data agents, or a profile you define. Runs, evaluation and incidents.
- A breach opens an incident with root-cause analysis and recommendations drawn from a closed catalogue of recovery actions. Monitors watch the fleet for the things a single run cannot show — drift, cost, silence, a judge that has quietly stopped working. Monitors, actions and recovery.
- You read it in the Control Plane, the Runs list, an Explorer over every run, stock dashboards, and a health page about Sentinel itself. Explorer, dashboards and health.
Three design rules worth knowing
- The judge is independent. Runs are scored by a model you configure for the purpose — never by the agent grading itself — and its prompt and policies are content-addressed, so an evaluation always says exactly which rubric and rules produced it.
- Absence is never rendered as health. A run the judge could not score is skipped, not zero; a dimension judged on partial evidence is flagged; an agent that has gone silent is a monitor condition in its own right. A dashboard that looks fine because nothing was measured is the failure the product is built to avoid.
- Nothing relaxes a safeguard automatically. The recovery catalogue is closed and conservative: the only actions that can ever run unattended make an agent more careful. Anything that reaches outside VerOps needs a person.
Where to start: register one agent, send it one run with a final response.final event, and watch it appear under Sentinel → Runs with a score and a rationale. Everything else in this category builds on that.