Short answer: a fix for noisy alerts. The team discovered that the recurring paging and failures weren't caused by poor behavior, but by poor signal design —...
How Airbnb Rebuilt Its Alerting System After Realizing It Wasn't a Culture Problem
Short answer: a fix for noisy alerts. The team discovered that the recurring paging and failures weren't caused by poor behavior, but by poor signal design — and then rebuilt the alerting approach from the ground up. Frankly, the result leaned on better metrics, improved runbooks, and pragmatic tooling choices rather than simply preaching about responsibility. — Ernie Castro
Key Takeaways
- Problem identification matters: alerts are tooling and math, not only character.
- Measure first: collect signal, calculate noise, and prioritize thresholds.
- Fix process and tooling together: better routing, on-call ergonomics, and smarter rules win.
What is Airbnb's alert rebuild?
Short phrase: alert fatigue. The company was paged for dozens of low-value conditions every week, causing on-call engineers to ignore or disable pages, and that produced larger outages since high-severity signals were lost in the noise. Why did this happen? Because alerts were created by teams without centralized guardrails, thresholds were copied from incidents that weren't fully understood, and there was no consistent classification for severity.
Short list: audit alerts, prioritize signals, reduce noise. The first step is quantitative: measure how many pages are actionable, how long people take to respond, and which alerts lead to real remediation. Do the math, and then treat alerting as code — version it, review it, and require owners. Simple, but often skipped.
What is alerting at scale?
Short definition: turning metrics into action. Alerting means converting monitored data into a reliable signal that triggers human or automated responses without wasting attention. Ask a clear question: is this condition worth waking someone up? If the answer is no, then the line should be a metric, not a page.
Short reminder: use established frameworks. Google’s SRE principles discuss error budgets and toil reduction, and their approach helps decide whether a symptom merits paging or automated remediation; see the Google SRE book for background. Many companies couple Prometheus metrics with Alertmanager and route pages through PagerDuty or OpsGenie so the signal is filtered before it hits a human, and that prevents a lot of needless wake-ups.
Core Benefits/Importance
Short list: faster incident response. Better alerts lead to lower mean time to repair, fewer false positives, and calmer on-call rotations. The truth is, good alerting preserves human dignity at work: it respects attention, and it treats on-call staff as professionals who deserve meaningful signals rather than punishment by noise.
Short example: when I ran on-call for a high-traffic service, I saw teams duplicate alerts in different tools, and that meant multiple pages for the same root cause, and the result was confusion across teams. The fix was mundane: consolidate metrics into a single source of truth, remove duplicated firing rules, and set a single escalation path. The result was immediate: fewer redundant pages and faster focused responses.
Short point: ethical engineering matters. There's a stewardship angle here — teams are stewards of the system and of colleagues' time, and designing alerts responsibly is a simple expression of that duty to serve the common good. Make alerts specific, actionable, and owned.
Step-by-Step Implementation/Guide
Short plan: audit, categorize, tune. First, inventory all alerts and tag them by owner and severity. Second, measure signal-to-noise ratio per alert and remove or convert the low-signal ones into dashboards or logs. Third, implement better routing and escalation rules so the right team gets the right page. Fourth, automate remediations where possible so the system heals itself without waking anyone. Fifth, make runbooks easy and test them with drills.
Short checklist: what I did in practice. When I led an effort to rebuild alerts, we started with a simple spreadsheet exported from our monitoring system and then enriched rows with owner, last-firing date, time-to-ack, and whether the alert led to a code change or config rollback. We then prioritized alerts by frequency and cost — the ones that fired often and had low value were either demoted to dashboards or turned into log-only signals. Here's the kicker: we required Pull Requests for alert changes and insisted on a short rationale, and that stopped people from adding noisy rules on a whim.
Short note: automation and tooling. Use alert-routing rules in your on-call tool, connect runbooks in your paging payload, and prefer automated runbook hyperlinks rather than bulky text. Drill the process with fire drills so engineers practice responding to the right signals, and log every drill for continuous improvement. For further reading on practical alerting practices, see PagerDuty’s guide on alerting best practices.
Comparison Table
Short intro: a side-by-side look.
| Feature | Rebuilt Airbnb Alerts | Typical Noisy Alerting System |
|---|
| Ownership model | Clear owner per alert | Alerts orphaned or unclear |
| Severity labeling | Consistent S1–S4 taxonomy | Inconsistent or informal labels |
| Signal-to-noise | High | Low, lots of false positives |
| Automation | Automated remediation where possible | Mostly manual pages |
| Review process | Code-reviewed alert changes | Ad hoc changes by individuals |
Common Mistakes to Avoid
Short warning: don't blame the team. Many organizations point at culture when alerts are noisy, but that's often a dodge that hides missing measurements, missing ownership, and poor tooling. I've seen teams punish junior engineers for responding to noise, and that only amplifies fear rather than teaching reliability.
Short correction: measure first. If you assume culture without measuring, you are guessing and will waste time on training that doesn't matter. Measure how many pages are actionable, how long they take to resolve, and how many pages lead to a code fix; then prioritize the ones that produce value. The Atlassian approach to blameless postmortems helps here: focus on causes and fixes, not people — see Atlassian’s postmortem guidance.
Short reality: avoid alert proliferation. Don't create an alert for every SLA blink; aggregate and use rate-based rules, and prefer severity tiers so responders know what to expect. Also, don’t skip runbooks — a page without a clear owner and remediation path is a test of patience, not a tool for reliability.
Frequently Asked Questions
Short Q1: How do you start cleaning alerts? A: Audit, measure, and reduce. Create an inventory and compute a simple metric: action rate equals the number of pages that led to a remediation divided by total pages. Then remove or demote the ones with very low action rates.
Short Q2: Should all alerts page people? A: No. Use dashboards or logs for informational signals and reserve pages for actionable incidents that require human judgment or intervention. Automated remediation handles many transient issues, and that reduces fatigue.
Short Q3: How do you prevent regressions? A: Require code review for alert changes, supply runbooks, and assign owners. Also integrate alerting changes into the CI pipeline so tests can detect obvious misconfigurations before they roll out.
Short Q4: What about the cultural side? A: Culture matters, but it rarely solves basic measurement gaps. Build rituals that encourage stewardship, respect for colleagues, and moral responsibility for shared systems — small virtues that reflect dignity of work and produce better long-term outcomes.
Final Thought
Short final: alerts reflect stewardship. When I rebuilt alerting systems at several companies, the best outcomes came when we treated attention as sacred, invested in tooling, and taught teams to classify and measure their signals. The changes were boring: we cleaned rules, added owners, and enforced reviews — and yet the business impact was huge because we reduced fatigue and improved uptime.
Short subtlety: ethical tech matters. Designing with care is a small act of stewardship that respects coworkers and customers, and that aligns with a simple moral principle: use your talents to serve the common good, not to create extra noise. If you take one thing from this story, let it be this: measure before you blame, and make the alerts you send worth someone's sleep.
— Ermenildo V Castro