The observability industry promises complete visibility into your systems. The reality for most teams is different: dashboards nobody looks at, alerts that fire so often they’re ignored, logs that are either too sparse to be useful or so verbose that finding anything is impossible, and bills that grow faster than understanding.
The problem isn’t lack of data. It’s lack of focus on what that data is supposed to accomplish.
What Observability Is Actually For
Observability exists to answer questions about system behavior. Not to collect data for its own sake, not to meet compliance checkboxes, not because everyone else does it—to answer questions.
Useful questions look like: Is this system working correctly? If something is wrong, where is it wrong? What changed before it broke? Is this request slow because of the network, the database, or the application? Which customers are affected by this issue?
Bad observability gives you data without helping answer questions. Good observability surfaces answers quickly, sometimes before you’ve finished forming the question.
This distinction matters because data collection has costs—storage, processing, cognitive load, and actual money. Every metric, log line, and trace that doesn’t help answer real questions is waste. The goal isn’t maximum data; it’s sufficient insight.
Start With Questions, Not Tooling
Teams often approach observability backwards. They choose a platform, instrument everything it supports, and then wonder why understanding their systems is still hard.
Start instead with the questions you need to answer. What does “healthy” look like for your system? What failures do you most need to detect quickly? When something goes wrong, what information do you need to diagnose it? When you’ve had outages in the past, what information was missing?
These questions point directly to what you should observe. A checkout system needs to track transaction success rates, payment provider latency, and error rates by type. A data pipeline needs to track throughput, processing latency, and data quality metrics. A user-facing API needs response times, error rates, and availability.
The specific signals depend entirely on what your system does and how it fails. Generic dashboards showing CPU and memory are rarely the signals that matter—they’re proxies that sometimes correlate with actual problems. Instrument what directly indicates health for your specific system.
The Three Pillars Actually Integrated
Observability tools are typically described in terms of metrics, logs, and traces. Most teams collect all three but use them in isolation—checking metrics first, then searching logs separately, then maybe looking at traces if they think of it.
Integrated observability connects these signals so you can navigate between them. An alert fires on high error rate (metric). Click through to see the error breakdown and affected endpoints. Drill into example errors to see the full request traces. Click through to the relevant logs for a specific problematic request. Each signal provides context that makes the others more useful.
Metrics tell you something is wrong and give you the scope. Error rate increased. Latency spiked. Throughput dropped. These are the signals that drive alerts because they indicate impact.
Logs give you the detail of what happened. The specific error message, the input that caused it, the state of the system at that moment. They’re the forensic evidence you need for diagnosis.
Traces show you the path of a request through distributed systems. Where time was spent, which services were involved, where failures occurred in the chain. They’re essential for debugging cross-service problems.
The power is in the connections. Knowing that error rate increased isn’t actionable without the logs that explain the errors. Having logs for individual failures doesn’t show patterns without the metrics that aggregate across requests. Traces without context don’t tell you which paths through the system are problematic.
Alerting That Works
Most alerting is counterproductive. Alerts fire constantly, get ignored, and create a culture where the oncall person’s job is to dismiss noise rather than respond to problems.
Effective alerting follows principles that seem obvious but are rarely implemented:
Alert on symptoms, not causes. Users don’t care if CPU is high—they care if the site is slow or returning errors. Alert on response time and error rate, not on resource utilization. The symptom-based alert tells you impact directly; the resource alert makes you guess.
Every alert should be actionable. If the alert fires and the correct response is “wait and see if it resolves itself,” remove the alert. If the correct response is always the same automated remediation, automate the response instead of alerting a human. Humans should only be paged when human judgment is needed.
Reduce alert volume ruthlessly. The number of alerts should be small enough that every single one gets genuine attention. If your team ignores more than 10% of alerts, you have too many. Cut the noise until every alert matters.
Distinguish urgency. Some problems require immediate response—the production site is down, customers can’t complete purchases. Some problems require attention today—a background job is failing, a queue is growing slowly. Some problems can wait until business hours—a non-critical metric is outside normal bounds. Alert channels and escalation should match urgency.
Include context in alerts. The alert should tell you what’s wrong, how bad it is, and where to start investigating. “Error rate high” is useless without knowing which service, which endpoints, and how far above normal. Link directly to relevant dashboards and runbooks.
Logging That Scales
Log volume grows faster than teams realize. At some point, the cost of storing everything exceeds the value of having it, and queries become too slow to be useful for incident response.
Structured logging makes logs actually searchable. Instead of log lines like 2025-01-15 Error processing user request: timeout, use structured fields: timestamp, level, service, operation, user_id, error_type, duration. Searching for all timeout errors in the checkout service for a specific user becomes a simple query instead of regex archaeology.
Log levels exist for a reason. ERROR means something is broken and someone should know. WARN means something unexpected happened but the system handled it. INFO means normal operation that’s worth recording. DEBUG means detailed information useful during development but not in production. Most production systems should emit almost no DEBUG logs and relatively few INFO logs.
Sampling for high-volume logs preserves visibility without bankrupting you. If your service handles millions of requests per day and you need to debug occasional problems, logging 1% of successful requests is probably sufficient—the patterns will be visible. Log 100% of errors and slow requests, where you need complete information.
Retention should match value. Detailed logs from last week are useful for debugging recent issues. Detailed logs from last year are rarely needed—aggregate metrics and summary data serve historical analysis better. Tier your storage: recent logs in fast storage for active debugging, older logs in cheaper storage for compliance or rare investigations, and delete what you’ll never need.
Dashboards That Get Used
Most dashboards are created once and never viewed again. Someone builds a dashboard for a new service, adds everything they can think of, and then nobody develops the habit of looking at it.
Dashboards that get used are specific to needs. A dashboard for morning health checks shows different information than a dashboard for active incident debugging. A dashboard for the data team shows different information than a dashboard for the platform team. One giant dashboard for everything serves nobody well.
The best way to design useful dashboards is to build them during incidents. When you’re debugging a problem and wish you could see some data more easily, add it to the dashboard right then. Over time, the dashboard evolves to show exactly what you need during problems—because it was built from that need.
Review dashboards periodically. If nobody has looked at a dashboard in three months, archive it. If there are panels nobody understands, remove them. A smaller number of actively used dashboards is better than a large number of abandoned ones.
Making It Sustainable
Observability isn’t a project that ends. Systems change, failure modes evolve, and what you needed to see last year might not be what you need to see this year.
Build observability into your development process. When you add new features, consider what signals indicate that feature is working. When you have incidents, ask what observability would have helped you detect or diagnose the problem faster. Treat observability debt like other technical debt—track it, prioritize it, and pay it down.
Control costs proactively. Understand what you’re paying for—most observability platforms charge by data volume. Instrument intentionally rather than by default. Review costs regularly and cut signals that don’t provide value. The cheapest data is data you don’t collect.
The goal is simple even when the implementation is complex: know when your system is healthy, know quickly when it isn’t, and be able to diagnose problems fast. Everything else is details.