The 10x threshold that made a real search burn invisible

The rule

SearchErrorBudgetBurnFast is a single-window fast-burn alert over a 99.9% availability objective, and its own author left the arithmetic in a comment above it:

# The SLO target is 99.9% (error budget 0.001). The fast-burn threshold should be
# 14.4 * 0.001 = 0.0144, but the rule hardcodes 0.144 — a 10x error-budget miscalc
# (burn_rate_math_off_by_error_budget). The alert never fires even while the budget is
# consumed at ~1.7x, so the real burn is invisible.
groups:
  - name: search-availability-slo
    interval: 1m
    rules:
      - record: job:slo_error:ratio_rate1h
        expr: rate(slo_requests_total{job="search",result="error"}[1h]) / rate(slo_requests_total{job="search",result="total"}[1h])
        labels:
          job: search
      - alert: SearchErrorBudgetBurnFast
        expr: job:slo_error:ratio_rate1h{job="search"} > 0.144
        for: 2m
        labels:
          severity: critical
          slo: search-availability
        annotations:
          summary: Search error budget fast burn (threshold miscalibrated)
          budget: '0.001'

The multi-window multi-burn-rate method's fast-burn factor is 14.4× (Google's SRE Workbook, ch. 5): multiply it by the error budget — 1 - 0.999 = 0.001 here — and a fast burn should trip at 14.4 × 0.001 = 0.0144, a 1.44% error ratio. The rule compares against 0.144 instead: fourteen-point-four percent, not fourteen-point-four times the budget. The decimal point moved one place and the threshold moved ten.

What the miscalc costs

Fed the two thresholds as structural evidence — the same pair packages/cli/src/replay/precision.test.ts pins as this fixture's ground truth — the classifier does the division and calls it:

neutralInput('SearchErrorBudgetBurnFast', {
  burnRateMath: { configuredThreshold: 0.144, correctThreshold: 0.0144 },
});

rootCause:  burn_rate_math_off_by_error_budget
confidence: 0.95
explanation: The burn-rate threshold is 0.144 but the SLO and window require 0.0144
             (off by 900%). The wrong multiplier mis-sizes the error budget, so the alert
             fires too early or too late (burn_rate_math_off_by_error_budget).
prescribedFix.diff:
  - expr: burn_rate > 0.144
  + expr: burn_rate > 0.0144

packages/classifier/src/detectors.ts only raises this class once the relative error between configured and correct clears a 5% tolerance — this one clears it by 900%, which is also why the confidence lands at 0.95, the top of the range the detector assigns: 0.45 plus half of the relative error, capped at 1. packages/classifier/src/classify.test.ts and the precision suite both hold this exact pair of thresholds to that exact root cause.

Replaying it

The fixture is fixtures/scenarios/burn_rate_math_off_by_error_budget/:

git clone https://github.com/BurnRateSurgeon/burnratesurgeon
cd burnratesurgeon
pnpm install
pnpm --filter @burnratesurgeon/cli build
node packages/cli/dist/index.js replay \
  --tsdb fixtures/scenarios/burn_rate_math_off_by_error_budget \
  --rules fixtures/scenarios/burn_rate_math_off_by_error_budget/rules.yml \
  --alertmanager fixtures/scenarios/burn_rate_math_off_by_error_budget/alertmanager.yml

# brs replay report

- Engine: 1.0.0
- Window: 1970-01-01T00:00:00.000Z → 1970-01-02T00:00:00.000Z
- Alerts simulated: 0
- Root-cause findings: 0
- Critical findings: 0

Zero alerts simulated is the correct replay of a genuinely broken rule: a threshold ten times too high does not fire on the incident it was written for, so a deterministic replay of that incident against that rule reports nothing — which is the failure, not an artefact of it. Turning "reports nothing" into a labelled finding on the report itself is the per-class evidence extractor tracked as follow-up work in fixtures/scenarios/README.md; until it lands, the classification above is what demonstrates the miss, and it is pinned in CI rather than asserted in prose.

Why a threshold this far off still shipped

0.144 and 0.0144 differ by one digit's position, and the annotation on the rule — budget: '0.001' — records the number the author had in front of them while writing the comparison on the line below it. A decimal-point slip like this one does not fail a linter or a promtool check rules: both confirm the expression parses and the labels are well-formed, and a PromQL comparison against 0.144 is exactly as valid as one against 0.0144. Nothing about the rule's syntax names the arithmetic wrong, and nothing runs the history through it to notice that the number never gets close.

The fix

The fix is the diff the classifier already prescribes, applied to the real expression:

       - alert: SearchErrorBudgetBurnFast
-        expr: job:slo_error:ratio_rate1h{job="search"} > 0.144
+        expr: job:slo_error:ratio_rate1h{job="search"} > 0.0144
         for: 2m

The general form of the check, for any fast-burn rule: recompute burnRateFactor × errorBudget from the SLO the rule names in its own annotations, and compare it against the literal in the expr. Where they diverge by more than rounding, the rule is not protecting the budget its annotations claim it is — and the only way to know how far off is to replay it against a burn it should have caught, the way this fixture is replayed on every commit.