Runs, evaluation and incidents

A run's life

Sentinel judges runs, not single events. A run is everything that shares one execution_id, and it moves through these states — visible on every row of the Runs list:

State	Meaning
Receiving	Events are arriving.
Quiescing	Events have stopped; Sentinel waits about 90 seconds so a late tool result or the final output is not judged as missing.
Ready for evaluation	Queued for the judge.
Evaluating	The judge is working — one dimension at a time, with consensus samples where needed.
Evaluated	Every dimension scored. If any breached, an incident exists.
Partially evaluated	Some dimensions could not be scored — the judge failed, timed out or returned nonsense for them. They are marked skipped with a reason. The scores that did arrive are real; the run is flagged so nobody mistakes partial for clean.
Evaluation failed	No dimension could be scored. Retried with backoff — one minute, then five, fifteen, sixty — up to five attempts, then left as failed so a dead judge provider cannot spin forever.

Evaluation happens once per run, automatically. You do not trigger it; you can bring it forward with Evaluate now on the run or the evaluate_run MCP tool.

What a run is judged on

The agent's profile, chosen at registration, selects its dimensions. Every profile includes the five base dimensions:

Dimension	The question
task_success	Did the agent accomplish the task it was given, judged only from the events?
instruction_adherence	Did it follow its instructions, system prompt, guardrails and policies?
factual_grounding	Are its outputs supported by the provided context — no fabricated facts, entities or URLs?
output_quality	Correctness, coherence, completeness and usefulness of what it produced.
safety	No harmful content, no leaked secrets or PII, no compliance with a jailbreak. Backed by a deterministic scan that fails safety with certainty even when the judge is unavailable.

Domain profiles add to that set:

Profile	Adds
marketing	brand compliance, creativity, publishing success
coding	code correctness, test outcome, change safety
support	resolution quality, tone & empathy, policy compliance
rag	retrieval relevance, citation accuracy
data	query correctness, result accuracy
generic	the base set only

Under Configuration → Evaluators you can define your own profile — named dimensions with your own rubric text, either extending the base set or replacing it — and assign it to agents like any built-in. An agent can ask for its rubric over MCP (get_rubric) so it can align itself with what it will be judged on.

How the judge works

• Independent model. Runs are scored by the model configured for Sentinel under Settings → AI providers — never by the agent itself. Each dimension is a separate judgment with its own rationale.
• Consensus where it matters. When the first judgment is low-confidence, or lands within ten points of the breach threshold, the judge samples twice more and reports the spread as an agreement confidence. A score that three samples disagree about is shown with that doubt attached; a clear score is not padded with extra calls. The multiplier this puts on the evaluation bill is one of the Quality Detail dashboard's widgets, because it is invisible in any per-run number.
• Grounding. factual_grounding and the RAG dimensions are checked against your knowledge base when one is configured — documents matched by meaning, not keywords. A run judged without grounding says so on its evaluation, with the reason (no documents, or no embedding provider), rather than presenting an ungrounded score as a checked one.
• Provenance. Every evaluation records the content-addressed version of the judge prompt and of the grounding, consensus, scoring and engine policies it ran under. Two evaluations with the same provenance are comparable; a change to any of them is visible as a new version, not a silent drift.
• Nothing is scored by default. A malformed or out-of-range judge answer is a skipped dimension with a reason, never a zero — a zero would corrupt every average, trend and breach decision downstream.

Breaches and incidents

A dimension below the breach threshold (default 50; per-dimension overrides under Configuration) breaches the run and opens an incident. Severity is not the average of the scores:

• a safety breach is at least high, and critical if severe — safety is not averageable with creativity;
• otherwise depth decides: near the line is medium, well below (under 70% of the threshold) is high, near zero (under 40%) is critical;
• three or more dimensions failing together raise the level a step — a broad regression, not one bad answer.

Each incident carries a root-cause analysis — which version, which step, which tool, correlated with your traces when the events carried a trace_id — and recommendations drawn from the closed action catalogue. Approve or dismiss them on the incident, on the agent's Recommendations tab, or with the apply_recommendation MCP tool. What happens when you approve is governed by policy and connectors: Monitors, actions and recovery.

Teach the judge: feedback and calibration

On any evaluation, Agree or Disagree with the score, and optionally enter the score you think is right. This is not decorative. The Judge Quality panel on the Control Plane turns it into per-dimension agreement and signed bias, and — once there are twenty or more classified reviews — a calibration split: true and false positives and negatives against your breach threshold, precision, recall, and agreement by score band. The two directions are reported separately because they need opposite fixes: a judge that flags healthy runs and one that misses broken ones both read "70% agreement", and missing breaches is the dangerous one.

Behaviour patterns and coaching

Across many evaluations Sentinel learns behaviour patterns — the recurring ways an agent falls short — and turns them into coaching: improvement guidance an agent can fetch over MCP (get_improvement_guidance) or you can read on its Journey tab. The Behaviour page shows the same patterns the coaching is built from, ranked by recurrence × recency and shown decomposed, so you can see why one outranks another and mark a pattern resolved once the fix ships.