Ethan's Notes

Alert fatigue and how I cut our Grafana alerts by half

We had 214 alert rules. 37 of them had never fired. 61 fired only during deploys. After a month of tagging every page with acted / ignored / already knew, we deleted 98 rules and moved 40 to a dashboard-only panel.

The remaining rules follow one shape: a symptom the user would notice, a threshold derived from an SLO, and a runbook link. Nothing else gets to page a human at 3 a.m.