Monitors, actions and recovery

Two kinds of signal

A single run's evaluation tells you that one answer was bad. Monitors watch the fleet for what no single run can show — drift over a week, cost creeping up, an agent that has gone silent, a judge that has quietly stopped scoring one dimension. When something is wrong, actions are how Sentinel proposes, and within limits you set applies, a fix — always through a connector you configured and always from a closed catalogue.

Monitors

Sentinel → Monitors → New monitor. Pick a kind, scope it (an agent, a fleet, an environment, or everything), set the bar and the window, and preview it: Sentinel replays the condition over your recent history and reports how often it would have fired, projected to alerts per day and per week, with a verdict — "would fire about 38 times a day and would be ignored" is the kind of thing it says, because a monitor that pages that often is not a monitor by the second week.

Kind	Fires when	Reach for it because
Threshold	a dimension's score crosses a bar — safety < 90	the simplest gate on quality.
Trend	a metric moves materially against its own recent baseline — quality down 15% vs the previous 7 days	slow drift never crosses a fixed line.
Error rate	a tool, or the run itself, fails too often	tool failures show in the events before they show in scores.
Cost	spend per run moves — cost/run up 40%	a prompt change that doubled token use is a regression nobody sees on a quality chart.
Latency	run latency crosses a bar at a percentile — p95 > 15 s	users feel the tail, not the mean.
Percentile regression	p95 of a measure moves against its baseline	a mean absorbs a tail; p95 doubling while the average holds is exactly what a mean hides.
Version regression	a new agent version is materially worse than the one before	the deploy is the most common cause of a regression, and the easiest to roll back.
Absence	an agent has emitted no run in a window	the one condition an agent can never report about itself — silence is the signal.
Skip rate	too many runs were scored on only some dimensions	a judge quietly failing on one dimension leaves every other score looking fine.
Backlog age	the oldest ungraded run has waited too long	a count cannot tell a queue draining in seconds from one stuck for hours; age can.
Evaluation failure	runs that could not be graded at all	a dead judge provider — a model at end of life, a revoked key — shows here first and nowhere else until scores stop arriving.
Safety breach	critically breached runs in the window — default bar: any	a count, not a score threshold, so it cannot be tuned away by moving a number.
Confidence anomaly	the judge is unsure, or its consensus samples disagree	a low-confidence score is not a low score; it is one that should not be relied on, and every score-based monitor stays green while it happens.
Prohibited action	recovery actions the policy refused	something repeatedly proposing actions outside the catalogue or above the autonomy level is a signal in its own right.
Compound	several conditions together, AND/OR — quality down AND cost up after a deploy	a real regression has a shape; one condition alone is often noise.

Every monitor has an owner (default: whoever created it — an unowned monitor is the one nobody tunes), a cooldown so a flapping condition raises one alert rather than forty, and can be muted for a period with a reason. Changes are recorded in the monitor's audit trail — only what changed, so the one threshold quietly widened is not buried under every save. A firing monitor raises a platform alert through the normal routing, silences and on-call schedules. A monitor with nothing to measure reports no data, which is distinct from OK and is itself something to notice.

Policy: what Sentinel may do on its own

Sentinel → Configuration → Policy & recovery. Three controls, each scopable to an agent, a fleet or an environment:

• Breach threshold — the score below which a dimension breaches (default 50), with per-dimension overrides.
• Autonomy level — a ladder, not a switch. Observe: evaluate and report only. Recommend (the default): propose actions, execute none. Prepare: resolve an action's parameters and validate its target, but a person approves. Auto low-risk: execute actions whose own definition permits it at this level. Scoped autonomous: remediate within scope, still bounded by each action's ceiling. Higher levels include the lower ones; an unrecognised setting is read as Recommend, never as more.
• Halt all recovery actions — one switch that stops every outbound delivery for the organization, immediately, whatever the autonomy level. Turn it on first and ask questions after.

The toggle Raise a platform alert when an agent breaches sends each breach into the platform's alerting as well as opening a Sentinel incident.

The action catalogue

Recommendations can only name actions from a closed, engineer-defined list. The AI-SRE model that writes recommendations is shown these keys and may choose among them; anything it invents is rejected before it is ever stored. Each action declares its risk, whether a person must always approve it, and the highest autonomy level at which it may ever run unattended:

Group	Actions	Unattended?
Make the agent more careful, or tell someone	enable_fact_checking, require_citations, add_validation, increase_review, notify_owner	The only actions that can ever run without a person — and only at Auto low-risk or above.
Change the model	reduce_temperature, rollback_model, switch_provider	A person always approves; Sentinel may prepare them.
Tools, lifecycle and publishing	disable_tool, pause_agent, retry_publish	A person always approves; Sentinel may prepare them.

There is deliberately no key whose effect is to relax a safeguard — the catalogue is also what a model can ask for, and an action that cannot be named cannot be recommended.

Connectors: where an approved action goes

Sentinel never touches your systems directly. An approved action is delivered as a signed HTTP request to a connector you define under Sentinel → Actions:

1. Endpoint URL — your service that knows how to pause an agent or require citations in your stack. Private and link-local addresses are refused, so a connector cannot be pointed at the platform's own network.
2. Authentication — signature only, a bearer token, or a named header carrying a credential. The credential is shown once and cannot be read back; rotate it from the connector.
3. Actions this connector may carry — a subset of the catalogue. A connector for pause_agent cannot be sent rollback_model.
4. Scope — restrict it to an agent, a fleet or an environment.

Every delivery carries X-Sentinel-Signature (v1= an HMAC-SHA256 over the request), X-Sentinel-Timestamp, X-Sentinel-Org, X-Sentinel-Action, X-Sentinel-Request-Id and X-Sentinel-Idempotency-Key. Verify the signature, reject stale timestamps, and treat the idempotency key as the unit of "already done" — retries reuse it. Deliveries retry with backoff behind a circuit breaker; the log on the Actions page shows every attempt, and the Sentinel Health tab counts connectors that are failing and connectors that are unproven — configured but never successfully delivered to, which is the state most connectors are in the day they matter.

Approving a recommendation

From the incident, from the agent's Recommendations tab, or with the apply_recommendation MCP tool. Approval checks the policy (autonomy, halt switch), resolves a connector in scope that carries the action, signs and delivers. A recommendation with no eligible connector says so rather than pretending it was delivered; a delivery the policy refuses is recorded as rejected, which is what the Prohibited-action monitor counts.