When an on-call channel is noisy, the first request is usually to tune thresholds. Sometimes that removes an obvious mistake. It rarely fixes alert fatigue for long, because fatigue is not a number in PromQL. It is the result of unclear ownership, mixed priorities and pages that do not correspond to a decision.
I rebuild alerting from the response I expect. The rule comes last. First I decide what promise is at risk, who can act and how quickly a human decision is required.
I classify every notification by the action it deserves
A page interrupts a person and therefore needs a high bar. It should represent meaningful user impact or an imminent threat to it, require timely human action and route to an owner who can change the outcome. Everything else belongs in a ticket, a dashboard, an automated remediation path or nowhere.
This classification often removes more noise than threshold work because it exposes alerts that were created only because a metric existed. Prometheus guidance makes the same practical point: keep alerts simple, alert on symptoms and avoid pages when there is nothing actionable.
- Page: urgent, user-relevant and actionable now
- Ticket: needs ownership, but not an interruption
- Dashboard: useful context during investigation
- Delete: no decision depends on it
Ownership is part of the alert definition
A route to a shared channel is not ownership. I require an accountable service team, a severity, a first action and a review date. If no team can own the response, the organization has found a service-boundary problem, not an alert-routing problem.
I also separate platform symptoms from product symptoms. A Kubernetes node event may be useful context, but the page should normally describe the service promise at risk. Otherwise the on-call person spends the first minutes translating infrastructure conditions into possible customer impact.
SLOs give paging a decision framework
For critical services, I connect paging to service level objectives and error-budget consumption. A burn-rate alert answers a better question than a static resource threshold: are we consuming the reliability budget fast enough that a human must act now?
I do not apply one SLO template to every service. The indicator, target and response window must match the product promise. Multi-window, multi-burn-rate alerting is useful because it can detect fast damage while confirming that it persists, and also catch slower budget loss without making the urgent path hypersensitive.
An SLO is valuable when it changes prioritization. A percentage on a slide is not an operating model.
Then I tune and test the rules
Once the policy is clear, threshold and duration tuning become straightforward. I replay recent incidents where possible, inspect how the proposed rules would have behaved and run controlled failure tests. The review includes missed pages as well as noisy ones; silence is not a success if real user impact becomes invisible.
Every page receives an owner and an expiry or review date. Services change, traffic changes and alerts decay. A small recurring review is cheaper than another large cleanup after the channel becomes unusable.
The result is trust, not zero notifications
I am not trying to produce an empty on-call channel. I am trying to make an interruption credible. When engineers trust that a page represents a real decision, response becomes faster and the organization can discuss reliability using the same language.
That trust comes from a visible contract between product risk, service ownership and operational action. PromQL implements the contract. It cannot replace it.
