Sentinel: AI quality monitoring
An AI feature can return 200 OK and still give a wrong, unsafe or off-policy answer. Sentinel receives the runs of your own AI agents, scores them with an LLM judge, and raises monitors and incidents when quality, safety or cost drifts.
- Connect
- HTTP events or MCP
- Scoring
- LLM judge, per dimension
- Categories
- Nine, weighted
- Monitors
- 14 monitor kinds
What each run is scored on
Every evaluator belongs to a category. The category sets its default weight in the run score, so a safety failure is not averaged away by passing quality checks. A breach in safety, security or compliance always calls for a person.
| Category | Covers | Default weight |
|---|---|---|
| Safety | Harm, toxicity, bias, policy-violating content | 3 |
| Security | Data exposure, prompt injection, credential handling | 3 |
| Compliance | Regulatory or contractual obligations | 2.5 |
| Grounding | Whether the output is supported by evidence; hallucination and citations | 2 |
| Reliability | Whether it completed, retried correctly and handled errors | 1.5 |
| Quality | Whether it did the job well | 1 |
| User experience | Tone, clarity, helpfulness | 1 |
| Efficiency | Cost and latency | 0.5 |
| Custom | Your own criteria | 1, or your own |
What you get
Evaluator profiles
A general evaluator set (task success, instruction adherence, factual grounding, output quality, safety) plus profiles for support, coding, RAG, data and marketing agents.
Judge isolation
Each evaluator is a separate judge call, and the agent’s output is fenced off from the judge’s instructions so it cannot steer its own score.
Monitors
Fourteen kinds, including thresholds, trends, cost, latency, version regressions, safety breaches, prohibited actions and missing runs, with a preview of how many alerts a monitor would raise.
DocumentationExplorer and dashboards
Slice runs by agent, version and dimension, save views, and export to CSV.
DocumentationEval suites in CI
Run golden datasets and a red-team preset from a workbook, and gate a deploy on the result.
DocumentationModel compare
Run the same inputs through two models and have a judge compare the outputs.
Documentation
Connecting an agent
- 01
Register the agent
Create the agent in Sentinel and copy its key.
- 02
Send runs
Post each run as an event over HTTP, or connect over MCP.
POST https://sentinel.verops.io/api/v1.0/sentinel/events - 03
Choose a profile
Pick the evaluator profile that matches what the agent does, then add monitors.
Limits worth knowing
- Judging uses the model your organisation has configured. A separate judge model cannot be set yet.
- Scores are model judgements. Use them to spot drift and regressions, not as proof of correctness.
Read more in the docs
Try it on your own systems
Create a workspace, or tell us about your environment and we will walk you through it.