I often enter an incident review where every individual tool appears to work. Prometheus is scraping, Grafana has dashboards, logs are searchable and alerts reach the on-call channel. Yet the first twenty minutes are still spent asking the same questions: Is this real? Who owns it? Which users are affected? What changed?
That is not primarily a tooling failure. It is a failure in the path between a production symptom and an engineering decision. My observability audits focus on that path, because another dashboard rarely fixes it.
I start with the incident, not the stack
A tool inventory tells me what a company bought or deployed. An incident timeline tells me what the system actually enables under pressure. I take one or two recent incidents and reconstruct the first signal, the first useful hypothesis, the handovers and the point at which responders understood user impact.
This immediately separates telemetry that merely exists from telemetry that helps. A dashboard opened after the root cause was already known is documentation, not detection. An alert that reaches five teams before an owner is found is distribution, not response.
- What was the earliest trustworthy symptom?
- Could the responder move from service impact to a likely dependency?
- Did labels, service names and ownership match across metrics, logs and traces?
- Did the alert include a decision, or only report a condition?
The missing layer is usually context
Most production systems do not suffer from a shortage of signals. They suffer from inconsistent context. The same workload may have one name in Kubernetes, another in a dashboard and a third in the support ticket. Environment, region, version and customer tier may disappear somewhere between collection and visualization.
I standardize the smallest useful identity for a service and carry it through the telemetry pipeline. OpenTelemetry semantic conventions are useful here because common attribute names make signals easier to correlate, but the important part is organizational agreement: the team must know what a service is called and who owns it.
A page should begin an investigation
Prometheus recommends alerting on symptoms and keeping pages actionable. I use that as a practical test. When a page fires, it should tell the responder what user-visible promise is at risk, which service owns the first action and where the next evidence lives.
This does not mean placing a runbook novel inside every notification. It means removing ambiguity. A useful page has a clear severity, an owner, a link to a focused view, a short first action and enough labels to compare it with a deployment or dependency event.
If an alert cannot change a human decision, I do not treat it as a page.
What I change after the review
The output is a prioritized correction plan, not a generic maturity score. I map each finding to the delay it caused: detection, validation, routing, diagnosis or recovery. Then I fix the few paths that affect the most incidents before discussing a platform rewrite.
- A service and ownership map that is consistent across the stack
- A small set of user-impact views for the critical services
- Alert rules tied to an action, severity and runbook
- Deployment and configuration changes visible beside operational signals
- A short verification exercise using a real or safely simulated failure
The outcome I look for
I consider the work successful when the first responder can move from symptom to owner and a defensible hypothesis without knowing the system by memory. That is the difference between monitoring components and operating a service.
More telemetry may eventually be useful. First, I make the existing telemetry answer the questions people ask during an incident. That is usually the fastest route to a calmer on-call rotation and a more credible observability investment.
