Alert rules
Condition types, severity, sustain windows, de-duplication and examples.
alertsrulesseverityconfig
Alert rules
A rule defines a condition, a severity, and how it routes.
Condition types
- Threshold — a metric crosses a value (e.g. error rate > 2%, p95 > 800ms).
- Change — a sudden shift from baseline.
- Absence — expected data stopped arriving (a dead check or silent service).
- Check failure — a synthetic test or Workbook failed N times in a row.
Anatomy of a rule
| Field | Purpose |
|---|---|
| Source / scope | What it watches (a service, test, metric). |
| Condition | The threshold/expression and evaluation window. |
| Severity | critical / warning / info — drives routing and urgency. |
| For | Sustain duration before firing (avoids flapping). |
| Notifications | Channels/people to notify (by severity). |
Note: For defaults to 0 — the rule fires on the first breaching evaluation. Set it (e.g. 5 minutes) for noisy signals, and remember the rule only checks at each evaluation interval (default every 5 minutes), so detection latency is interval + For.
Examples
- Checkout uptime: synthetic test fails 2 times in a row → critical → page on-call.
- Latency budget:
checkout-apip95 > 800ms for 5m → warning → Slack. - Silent agent: no heartbeat from a private agent for 5m → warning.
Evaluation is distributed and de-duplicated, so one underlying problem produces one alert, not four (once per replica).