A policy nobody measures is a document. Shadow-mode keeps replaying your rules after they ship, so the reliability programme has a trend line instead of a set of intentions.
Where a reliability programme quietly stops working
The policy is written once and never graded
The error-budget policy says what happens when the budget burns. Nothing in it checks that the alerts guarding the budget would still fire, and nothing raises its hand when they stop.
Regressions arrive between incidents
A label added upstream or a series limit hit in a recording rule degrades coverage weeks before it costs anything. There is no incident to notice it in, and no review scheduled to look.
Every post-mortem learns the same lesson twice
The root cause is written up, the fix ships, and the knowledge stays in a document nobody rereads. A different team hits the same class on a different service two quarters later.
Three things a programme can be held to
Coverage, measured continuously
Shadow-mode replays each monitor against every new window of history and compares coverage with that monitor's previous settled run. A drop past the threshold is raised as a regression, without an incident to raise it.
The comparison is against the monitor's own last run, so what it reports is a change in your rules rather than a difference between two teams' services.
One weekly figure — critical alerts your rules predicted correctly — summed only from runs graded against real alert history, week by week, for as long as the monitors run.
A week that graded nothing reads zero. The series is allowed to have holes, because smoothing them would turn a measurement back into a claim.
Every classified miss resolves to a signature in the corpus: the root cause in a paragraph, the vendors it applies to, and the rule change that ends it.
The corpus is public and readable without an account, so an engineer who has never seen the failure can find it before they cause it.
Every incident, and the rule that should have caught it
The replay grades your rules against the incidents that really happened in the window. Each miss keeps its evidence — which incident, and the reason the alert stayed silent.
An example replay report: five incidents over seven days, two of them missed by CheckoutErrorBudgetBurnFast, each miss listed with the incident it was attributed from.
2 of these 5 incidents burned budget without paging anyone. Every one keeps the rule it was graded against.
Prescriptive
A catalogue of failures, each with the rule change that ends it
Every classified miss resolves to a signature in the public corpus: the root cause in one paragraph, and the diff that fixes it. Readable without an account, indexed for search.
The public failure-signature corpus: three catalogued entries, each with its root-cause class, its identifier and the vendors it applies to.
burnratesurgeon.com/corpus
26 public entries · no account needed
Window boundary miss
sig_synth_checkout_burst_between_evaluations
A 90-second wave of 503s from the checkout service burned 4% of the monthly error budget, but the page never fired: the burst sat between two evaluations of the 1h long window and was averaged away before the rule next ran.
A single node stayed NotReady for three hours after a kernel upgrade. Because the inhibition rule matched only on `cluster`, that one node alert suppressed every burn-rate page in the cluster, including a real recommendations outage on healthy nodes.
Applies to: Prometheus
Cardinality explosion hides the signal
sig_synth_tenant_label_explodes_ratio_series
The burn-rate rule kept the `tenant` label, so onboarding a batch of trial accounts multiplied the recorded series by forty. The rule group hit the per-user series limit, the recording rule was refused, and the alert evaluated against a series that had stopped updating.
Applies to: Mimir, Grafana Cloud
Live
The number that says the fix held
Shadow-mode keeps replaying after the rule ships, so the dashboard is a measurement and not a launch claim. One figure: critical alerts your rules predicted correctly, week by week.
An example north-star dashboard: critical alerts correctly predicted per week over eight weeks, above the counters for the last thirty days.
burnratesurgeon.com/dashboard/checkout-slo
Critical alerts correctly predicted
17
Week of 2026-08-24 · 65 alerts over 8 weeks
24
Replays run
19 graded against alert history
41
Predicted correctly
Alerts the rules raised that really fired
2
Rules about to fail
Alerts that fired and the rules would have missed
6
Storms predicted
Alerts the rules would have raised and reality did not
CheckoutErrorB…
PaymentsLatenc…
5%
Alert burn
△Warning
2 of 43 paging-worthy incidents went unpaged in the last 30 days.
Last 30 days
We already declare our SLOs in Nobl9 or Grafana.
Those declare the objective and render the burn against it. This grades the alerts that are supposed to defend it. They answer different questions, and only one of them tells you the page will arrive in time.
Validate your burn-rate rules against real history. The open-source CLI is free forever — the SaaS adds shadow-mode, PR gates, and a cross-vendor failure-signature corpus.