SLO Burn-Rate Alert Forensics
BurnRateSurgeon replays your historical time-series against proposed burn-rate rules and tells you exactly where they would have missed, over-fired, or stayed silent. Shadow-simulate every change before it ships.
groups: - name: checkout-sloRemoved: interval: 5mAdded: interval: 1m rules: - alert: CheckoutErrorBudgetBurnFastRemoved: expr: job:http_errors:ratio_rate5m{job="checkout"} > 0.05Added: expr: job:http_errors:ratio_rate1h{job="checkout"} > 14.4 * 0.001Added: and job:http_errors:ratio_rate5m{job="checkout"} > 14.4 * 0.001 for: 0mBefore — MissedThe burst lands between two 5m evaluations. No window spans it, the ratio reads diluted, nobody is paged.
After — CaughtMinute-level evaluation overlaps the windows and the 1h burn rate holds the whole burst. The page fires.
Counted over the public failure-signature corpus — root-cause clusters, not customer infrastructure.
The rules are generated, validated and reviewed. Nothing in that chain ever runs them against an hour of your own history.
The book
Multi-window multi-burn-rate. A 14.4x fast arm, a 6x slow arm, the windows paired as written. The maths is right. It assumes your windows overlap and your evaluation keeps up.
The tooling
The generator turns an SLO into rules and recording groups, and validates the YAML it just wrote. It never replays one hour of your own history against it.
The 3 AM page
Five hours of budget burned before the conjunction was satisfied. The rule was valid the whole time. Nothing in the pipeline told you it would wait.
You designed alerts by the book — Multi-Window Multi-Burn-Rate from the Google SRE Workbook — and they still break. These are the two ways they break.
Your service burns through its error budget at 3 AM, but the alert never fires. A window-boundary gap, a recording-rule eval lag, or an Alertmanager inhibit rule created a blind spot your team only discovers in the morning post-mortem.
A cardinality explosion or a rate-vs-increase quirk floods your on-call rotation with hundreds of alerts for a non-event. Every page is noise, and the real signal drowns in the storm.
Feed BurnRateSurgeon your TSDB snapshot, rules, and Alertmanager config. It replays the last 7 days of real metrics against your proposed rules and classifies every failure by root cause.
Point the CLI at your Prometheus TSDB snapshot, rules.yml, and alertmanager.yml. Works offline — no data leaves your network.
The replay engine evaluates every recording rule and alert rule against the real time-series, respecting evaluation intervals, for-clauses, and inhibit chains.
Get a classified report: which rules would have missed, which would have stormed, and a prescribed fix for each — with confidence scores backed by the failure-signature corpus.
Forensic
The replay grades your rules against the incidents that really happened in the window. Each miss keeps its evidence — which incident, and the reason the alert stayed silent.
2 of 5 incidents would have gone unpaged.
rpl_7f3c1a94 · engine 1.4.2
Window boundary miss — never fired
91% confidence · 2 matched · 0 false positives
Recording rule evaluation lag
74% confidence · 1 matched · 1 false positive
Checkout 503s, 90 seconds
inc-2026-08-24-checkout-503s
Missed — never fired
Checkout 500s after deploy
inc-2026-08-25-checkout-500s
Missed — never fired
Checkout upstream timeouts
inc-2026-08-26-checkout-timeouts
Paged
Payments p99 above objective
inc-2026-08-27-payments-latency
Paged
Checkout partial outage, one region
inc-2026-08-29-checkout-partial
Paged
2 of these 5 incidents burned budget without paging anyone. Every one keeps the rule it was graded against.
Prescriptive
Every classified miss resolves to a signature in the public corpus: the root cause in one paragraph, and the diff that fixes it. Readable without an account, indexed for search.
26 public entries · no account needed
Window boundary miss
sig_synth_checkout_burst_between_evaluations
A 90-second wave of 503s from the checkout service burned 4% of the monthly error budget, but the page never fired: the burst sat between two evaluations of the 1h long window and was averaged away before the rule next ran.
Applies to: Prometheus, Grafana
Silent inhibit loop
sig_synth_cluster_wide_inhibit_swallows_service_burn
A single node stayed NotReady for three hours after a kernel upgrade. Because the inhibition rule matched only on `cluster`, that one node alert suppressed every burn-rate page in the cluster, including a real recommendations outage on healthy nodes.
Applies to: Prometheus
Cardinality explosion hides the signal
sig_synth_tenant_label_explodes_ratio_series
The burn-rate rule kept the `tenant` label, so onboarding a batch of trial accounts multiplied the recorded series by forty. The rule group hit the per-user series limit, the recording rule was refused, and the alert evaluated against a series that had stopped updating.
Applies to: Mimir, Grafana Cloud
Live
Shadow-mode keeps replaying after the rule ships, so the dashboard is a measurement and not a launch claim. One figure: critical alerts your rules predicted correctly, week by week.
Critical alerts correctly predicted
17
Week of 2026-08-24 · 65 alerts over 8 weeks
24
Replays run
19 graded against alert history
41
Predicted correctly
Alerts the rules raised that really fired
2
Rules about to fail
Alerts that fired and the rules would have missed
6
Storms predicted
Alerts the rules would have raised and reality did not
5%
Alert burn
Warning
2 of 43 paging-worthy incidents went unpaged in the last 30 days.
Last 30 days
Sloth, Nobl9, Grafana and Datadog are good at what they were built for, and most teams that need us are already running one of them. The line between us is narrow and worth stating plainly: they turn an SLO into rules and keep score against it, and we run those rules back over your own history to find where they break.
| Capability | BurnRateSurgeonAlert forensics | SlothSLO generatorFull comparison → | Nobl9SLO platformFull comparison → | Grafana SLOSLO in Grafana CloudFull comparison → | Datadog SLOSLO in DatadogFull comparison → |
|---|---|---|---|---|---|
| Turns an SLO spec into alert rulesMulti-window multi-burn-rate rules and their recording groups, generated for you. | PartialPrescribes a fix as a rule diff. It does not author your SLO spec — bring Sloth or Pyrra for that. | YesIts core job, and the reference implementation for Prometheus and OpenSLO. | YesSLOs as code through sloctl and OpenSLO, across many data sources at once. | YesCreates the SLO with its recording rules and burn-rate alerts in one step. | YesSLOs built from monitors or metrics, with burn-rate alert conditions attached. |
| Replays your own history against the proposed rulesEvaluates the rules you are about to ship over real past time-series, respecting scrape lag, evaluation intervals, for-clauses and inhibit chains. | YesThe product. A deterministic replay over a TSDB snapshot, down to the scheduler. | NoValidates the spec and the YAML it wrote; it never evaluates either over data. | PartialReplay backfills an error budget from historical data. Your alert routing is not re-run over it. | PartialCharts the SLI once the SLO exists. The alert path itself is not re-evaluated over the past. | PartialA monitor preview graphs a threshold over past data. Composite and downtime logic is not simulated. |
| Says why an alert missed or stormedAttributes a root cause to each failure — window-boundary gap, evaluation lag, inhibit loop, cardinality explosion — and shows the evidence behind it. | YesEvery finding is classified and carries the incidents it was attributed from. | NoOut of scope by design: it is a generator, not an evaluator. | NoReports how much budget burned. Which rule let the incident past stays a manual question. | NoDashboards show what burned, not which rule was silent while it burned. | NoAlerts on burn rate. Diagnosing a miss afterwards remains an investigation. |
| Fails a pull request on an alerting regressionBlocks the merge when a rule change turns an incident the old rules caught into one the new rules would miss. | YesThe PR gate replays the branch and comments the incidents the change would have lost. | Partialsloth validate runs in CI and catches an invalid spec — a schema check, not a behavioural one. | Partialsloctl and the Terraform provider apply from CI, and the check they run is schema-level. | PartialSLOs are provisionable as code. There is no replay result to gate the merge on. | PartialMonitors and SLOs are Terraform-managed, and validation stays at the configuration layer. |
| Cross-vendor corpus of alerting failure signaturesA catalogue of failure modes observed in real production alerting, indexed across vendors and readable by your own agents over MCP. | YesPrometheus, Grafana, Mimir, VictoriaMetrics and Datadog signatures, in one index. | NoNone — and none is expected of a single-binary generator. | NoReads many data sources, but keeps no catalogue of how alerting fails on them. | NoNo shared record of failure modes across the estates it runs on. | NoDetection insights are per-account; nothing is catalogued across vendors. |
| Runs offline, on data that stays in your networkWhether the analysis can happen without shipping your time-series to a vendor. | YesThe CLI reads a local snapshot. Nothing leaves the cluster unless you upload it. | YesA single binary over local YAML — nothing to send anywhere. | NoAgents run in your network, and evaluation happens in the hosted platform. | PartialGrafana Cloud hosts it; self-managed needs Enterprise. The OSS build has no SLO feature. | NoFully hosted — SLO evaluation happens on Datadog’s side, on data you send them. |
| Open-source coreHow much of the thing you are evaluating you can read, fork and run for free. | YesCLI and replay engine are Apache-2.0. The corpus and shadow-mode are the paid part. | YesApache-2.0, end to end. Nothing is held back behind a paid tier. | Partialsloctl and the OpenSLO spec it helped start are open; the platform is not. | PartialGrafana itself is AGPL-3.0. The SLO feature is not part of that distribution. | NoProprietary and hosted, which is the trade the rest of the platform makes too. |
Checked against each product’s public documentation in September 2026 — the column headings link to it. If something here is out of date or unfair, open an issue and we will correct it.
The CLI and the replay engine are Apache-2.0 and always will be. The SaaS adds shadow-mode, the PR gate and the private corpus.
Free
Free
Apache-2.0, forever
Team
$99
per cluster, per month
Business
$299
per cluster, per month
Enterprise
Let's talk
annual contract
Forensic write-ups of burn-rate alert failures, and what the fix looked like.
A three-level recording cascade evaluates behind the alert reading it — replayed against a payments latency burn, the page the classifier attributes to the lag arrives 130 seconds after the ratio it depends on already had.
A fast-burn alert hardcodes a threshold ten times higher than its own 99.9% SLO math requires — the rule that should have paged never gets within reach of the number it was actually written to catch.
A single-window burn-rate alert reads the ratio over its own lookback and nothing else — replayed against a checkout error burst, the rule that should have paged never crosses threshold.
Validate your burn-rate rules against real history. The open-source CLI is free forever — the SaaS adds shadow-mode, PR gates, and a cross-vendor failure-signature corpus.
Not ready yet? Read one failure a week instead.