Create an alert rule

Create an alert rule (step by step)

Go to Alerts → Rules → Create Rule. A rule turns a metric/condition into routed alerts.

1. Name and description — make it actionable (what's wrong and where).
2. Source / scope — choose the application, service, synthetic test or metric to watch.
3. Aggregation — pick the function: avg(), count(), countIf(), or a percentile (95th/99th), over an evaluation window.
4. Condition — the comparison and threshold (e.g. p95 > 800ms; success rate < 99%). Combine conditions with AND.
5. Severity — critical / warning / info; this drives routing and urgency.
6. For — how long it must hold before firing (avoids flapping).
7. Notifications — channels/people per severity (Slack, email, PagerDuty…).
8. Save. The rule starts evaluating immediately.

Three different clocks — don't mix them up. The window your query looks at (e.g. p95 over the last 5 minutes) lives in the condition itself. Evaluation Interval (default 5 minutes) is how often the rule runs. For Duration (default 0) is how long the breach must persist before an alert is created — 0 fires on the first breaching evaluation. Worst-case time to page is roughly evaluation interval + for duration.

Worked example: checkout p95 over 800 ms for 5 minutes

Goal: warn when the checkout service's p95 latency stays above 800 ms for 5 minutes, without paging on a single slow blip.

Field	Value
Name	checkout p95 latency > 800ms
Source / scope	service checkout
Aggregation	95th percentile of duration
Condition	> 800 ms
Severity	warning
Evaluation Interval (minutes)	1
For Duration (minutes)	5
Notifications	Slack #checkout-oncall

Expected behavior: the rule evaluates every minute; the first breaching minute puts it in pending; if p95 stays above 800 ms for 5 consecutive minutes it fires once to Slack, and a brief spike that recovers within the window never alerts.

Condition recipes

Goal	Aggregation + condition
Latency budget	p95(duration) > 800ms for 5m → warning
Error surge	countIf(status=error)/count() > 2% → critical
Availability SLO	success rate < 99.9% → critical
Silent source	count() == 0 over 5m (absence) → warning

Evaluation is distributed and de-duplicated, so one underlying problem yields one alert, not one per replica.