The 5-minute window that hid an 880-error burst

The rule

checkout-slo is a two-level recording cascade feeding one fast-burn alert: a 5xx ratio, evaluated every 5 minutes, pages the moment it crosses 5%.

groups:
  - name: checkout-slo
    interval: 5m
    rules:
      - record: job:http_requests:rate5m
        expr: rate(http_requests_total{job="checkout"}[5m])
        labels:
          job: checkout
      - record: job:http_errors:ratio_rate5m
        expr: rate(http_requests_total{job="checkout",status="500"}[5m]) / rate(http_requests_total{job="checkout"}[5m])
        labels:
          job: checkout
      - alert: CheckoutErrorBudgetBurnFast
        expr: job:http_errors:ratio_rate5m{job="checkout"} > 0.05
        for: 0m
        labels:
          severity: critical
          slo: checkout-availability
        annotations:
          summary: Checkout error budget burning faster than 14.4x

"14.4x" is not decoration — it is the fast-burn multiplier from Google's multiwindow multi-burn-rate method (SRE Workbook, ch. 5): a burn rate of 14.4 sustained for one hour consumes 2% of a 30-day, 99.9% error budget (14.4 × 1h ÷ 720h = 2%). The companion rates in that table — 6× over 6 hours for 5%, 1× over 3 days for 10% — exist because a single window, on its own, has a blind spot. This rule has only the fast one.

The incident

A burst of failed checkouts lands at t = 43200 (12:00:00 into the replay window) — 5xx count jumps from a steady ~4-per-sample baseline to +880 in one step, against a roughly flat request volume:

{ "metric": "http_requests_total", "labels": { "job": "checkout", "status": "500" } }
// … 40, 44, 48, 52, 56, 60, 940, 944, 948 …
//                        ^ t = 43200

Taken as one interval, that is 880 errors against 13,380 total requests — a 6.6% error ratio, well past the rule's 5% threshold. This is the fixture BurnRateSurgeon's own regression suite ships as fixtures/scenarios/window_boundary_miss/ — one of six synthetic scenarios, one per canonical root-cause class the classifier covers (classes.ts).

Replaying it

The repo is Apache-2.0 (CLI + replay engine); building it from source and pointing brs at the fixture reproduces the incident exactly:

git clone https://github.com/BurnRateSurgeon/burnratesurgeon
cd burnratesurgeon
pnpm install
pnpm --filter @burnratesurgeon/cli build
node packages/cli/dist/index.js replay \
  --tsdb fixtures/scenarios/window_boundary_miss \
  --rules fixtures/scenarios/window_boundary_miss/rules.yml \
  --alertmanager fixtures/scenarios/window_boundary_miss/alertmanager.yml

# brs replay report

- Engine: 1.0.0
- Window: 1970-01-01T00:00:00.000Z → 1970-01-02T00:00:00.000Z
- Alerts simulated: 0
- Root-cause findings: 0
- Critical findings: 0

CheckoutErrorBudgetBurnFast never fires. Deterministic replay of the exact history that should have paged an on-call engineer says, byte for byte, that it did not — this report is the golden file packages/cli/src/replay/golden.test.ts pins in CI, so the miss stays reproducible rather than becoming an anecdote.

Why the rule stayed quiet

rate(...[5m]) is anchored to the evaluation timestamp and looks backward exactly 5 minutes; it does not know where inside that window a burst happened. A spike that straddles an evaluation boundary gets split across the two evaluations on either side of it, and each one sees only its own fragment — diluted by the rest of that window's ordinary traffic — of a burst that would have been unmistakable if any single window had captured it whole. That is window_boundary_miss, one of the six root-cause classes the classifier is built to name; it has no fix in the alert's own math, because the alert's own math is the cause.

The fix

A single window cannot corroborate itself. The multiwindow multi-burn-rate method exists for exactly this blind spot: require the fast window and a slower, independently-evaluated one to agree before paging.

       - record: job:http_errors:ratio_rate5m
         expr: rate(http_requests_total{job="checkout",status="500"}[5m]) / rate(http_requests_total{job="checkout"}[5m])
         labels:
           job: checkout
+      - record: job:http_errors:ratio_rate1h
+        expr: rate(http_requests_total{job="checkout",status="500"}[1h]) / rate(http_requests_total{job="checkout"}[1h])
+        labels:
+          job: checkout
       - alert: CheckoutErrorBudgetBurnFast
-        expr: job:http_errors:ratio_rate5m{job="checkout"} > 0.05
+        expr: |
+          job:http_errors:ratio_rate5m{job="checkout"} > 0.05
+          and
+          job:http_errors:ratio_rate1h{job="checkout"} > 0.05
         for: 0m

The 1-hour ratio is evaluated on the same 5-minute cadence, so it sees the burst from many more overlapping angles than the 5-minute window ever does on its own — a fragment too small to trip the fast window on either side of the boundary is not too small to move an hour-long average. It will not always be enough on its own; the general answer, when a window's own math is the failure mode, is to replay the incident and check, the way this fixture is checked on every commit.