Failure classes

Window boundary miss

An incident burned error budget but never paged, because it fitted entirely inside the detection window the alert needs to sustain a burn over.

Detection

The incident burned budget, the alert did not fire, and the incident lasted less than the long window plus the rule’s `for:` duration.

Remediation

Add a fast-burn arm on a short window so brief, severe burns page, and shorten `for:` on that arm.

Case study: The 5-minute window that hid an 880-error burst.

Recording rule evaluation lag

The alert reads a recorded metric, and the recording group’s evaluation lag delayed the page by at least the lag itself.

Detection

The alert depends on a recording rule, that rule has a non-zero evaluation lag, and the observed page delay is at least as long as the lag.

Remediation

Shorten the recording group’s evaluation interval, or alert on the raw series, so the derived value is fresh when the rule evaluates.

Burn-rate math off by the error budget

The configured burn-rate threshold diverges from the value the SLO and the window require, so the alert fires too early or too late.

Detection

The configured threshold differs from the correct one by more than rounding tolerance, given the SLO target and the window topology.

Remediation

Recompute the multiplier from the SLO target and the window, and set the threshold to the value the error budget implies.

Silent inhibit loop

An Alertmanager inhibition suppressed an alert that should have paged — in the worst case through a source/target cycle where a firing alert silences itself.

Detection

The alert was inhibited and either the inhibition forms a source/target cycle or the inhibited alert was one the incident says should have paged.

Remediation

Narrow the inhibition’s target matchers so the real signal is never its own suppression target, and break any source/target overlap.

Cardinality explosion hides the signal

A label blew the series count past an order of magnitude and the alert aggregates across it, averaging a hot instance below the threshold.

Detection

The series count exceeds its baseline by at least ten times and the alert’s aggregation drops the label that carries the signal.

Remediation

Group the expression by the signal-bearing label so a spike in one series stays visible instead of being averaged out.

rate() versus increase() quirk

The expression uses the wrong range-vector function, and a range too short for the scrape interval compounds the under-counting.

Detection

The applied range function differs from the one the expression needs, most damagingly when the range holds fewer than four scrape samples.

Remediation

Use the range function the quantity calls for and widen the range to hold at least four scrape intervals.