Your rotation already knows which alerts are noise and which ones arrive late. Nothing outside the rotation has a number for it. BurnRateSurgeon replays your own history and produces one.
A handful of alertnames produce most of the pages, and everyone on the rotation can name them from memory. Nothing writes it down, so it never becomes a piece of work with an owner and an estimate.
A late page still reads as a page that fired
An alert that arrives five hours after the burn started is green on every dashboard you own. It is also the one that spent the budget, and the only artefact that says so is a post-mortem.
Rules are edited under pressure or not at all
Between incidents there is no forcing function and no review, so the ruleset drifts with the services under it. The next incident is what measures the drift.
Three numbers you can take into a review
An alert quality score, per run
The score has a name in the product already: coverage — the share of the incidents in the replayed window your rules would actually have paged for, with the previous run beside it and the delta between the two.
A run is scored only when it is graded against alert history you supplied. A window with nothing to grade against reports no score rather than a flattering one.
The report timestamps every incident in the window and says, for each one, whether a rule paged and which rule suppressed it. The 03:00 pages separate from the 15:00 ones, and the avoidable ones separate from the rest.
The timestamps are the incidents your own history recorded, not a sample. Nothing is inferred about who was awake — the report gives the hour and the verdict, and the rota says the rest.
Critical alerts your rules predicted correctly, counted week by week. It is one line, it moves when the rules improve, and it is the same number whether it is read by your team or by your CTO.
Only graded runs are counted, so a week nobody measured reads zero instead of drifting upward on its own. The holes in the line are the point of it.
Every incident, and the rule that should have caught it
The replay grades your rules against the incidents that really happened in the window. Each miss keeps its evidence — which incident, and the reason the alert stayed silent.
An example replay report: five incidents over seven days, two of them missed by CheckoutErrorBudgetBurnFast, each miss listed with the incident it was attributed from.
2 of these 5 incidents burned budget without paging anyone. Every one keeps the rule it was graded against.
Prescriptive
A catalogue of failures, each with the rule change that ends it
Every classified miss resolves to a signature in the public corpus: the root cause in one paragraph, and the diff that fixes it. Readable without an account, indexed for search.
The public failure-signature corpus: three catalogued entries, each with its root-cause class, its identifier and the vendors it applies to.
burnratesurgeon.com/corpus
26 public entries · no account needed
Window boundary miss
sig_synth_checkout_burst_between_evaluations
A 90-second wave of 503s from the checkout service burned 4% of the monthly error budget, but the page never fired: the burst sat between two evaluations of the 1h long window and was averaged away before the rule next ran.
A single node stayed NotReady for three hours after a kernel upgrade. Because the inhibition rule matched only on `cluster`, that one node alert suppressed every burn-rate page in the cluster, including a real recommendations outage on healthy nodes.
Applies to: Prometheus
Cardinality explosion hides the signal
sig_synth_tenant_label_explodes_ratio_series
The burn-rate rule kept the `tenant` label, so onboarding a batch of trial accounts multiplied the recorded series by forty. The rule group hit the per-user series limit, the recording rule was refused, and the alert evaluated against a series that had stopped updating.
Applies to: Mimir, Grafana Cloud
Live
The number that says the fix held
Shadow-mode keeps replaying after the rule ships, so the dashboard is a measurement and not a launch claim. One figure: critical alerts your rules predicted correctly, week by week.
An example north-star dashboard: critical alerts correctly predicted per week over eight weeks, above the counters for the last thirty days.
burnratesurgeon.com/dashboard/checkout-slo
Critical alerts correctly predicted
17
Week of 2026-08-24 · 65 alerts over 8 weeks
24
Replays run
19 graded against alert history
41
Predicted correctly
Alerts the rules raised that really fired
2
Rules about to fail
Alerts that fired and the rules would have missed
6
Storms predicted
Alerts the rules would have raised and reality did not
CheckoutErrorB…
PaymentsLatenc…
5%
Alert burn
△Warning
2 of 43 paging-worthy incidents went unpaged in the last 30 days.
Last 30 days
Is this not just another dashboard nobody opens?
It is one number with the evidence underneath it, and the number is refusable: a week with no graded run reports nothing rather than a smoothed estimate. That is what makes it worth putting in front of somebody who was not on the rotation.
Validate your burn-rate rules against real history. The open-source CLI is free forever — the SaaS adds shadow-mode, PR gates, and a cross-vendor failure-signature corpus.