u/omarghader

Your monitoring bill went up 40% this year. Your MTTR didn't move an inch. That's called "The Observability Paradox".

More tools. More dashboards. More alerts. Still 40 minutes at 3 AM trying to find which of 40 services actually broke.

Here's what's really happening: nobody owns the cardinality/log cost until the invoice hits, alerts fire on symptoms instead of user impact, and there's no shared definition of "healthy" — so every incident starts with a debate, not a fix.

New vendor won't fix that. Governance will.

Full breakdown — including the 3 root causes and the framework that got us there 👇

https://omarghader.github.io/the-observability-paradox/

Full disclosure: I wrote this, so take the self-promotion tag as intended. Mods, feel free to remove if it doesn't fit.

I do observability/monitoring work for a living, and the question I get asked the most isn't "which tool should I use," it's "why do we still get paged for things we didn't see coming even though we have Grafana and a bunch of dashboards." Almost every time, the answer isn't a missing tool, it's a missing method.

So I wrote down the actual process I follow every time I build infra monitoring from scratch, mostly so I stop re-explaining it in Slack threads:

1. Define the business SLI/SLO *before* touching any tool. Not "CPU usage," something like "99.95% of payment authorizations should succeed" if you're a payments company.
2. Figure out what infra/apps you actually need visibility into to measure that SLI, *then* pick your stack. (I use VictoriaMetrics + vmagent for metrics, Loki for logs, OpenTelemetry + Jaeger for traces, but the method matters more than the specific tools.)
3. Instrument with OpenTelemetry auto-instrumentation wherever possible instead of hand-rolling everything.
4. Enrich all telemetry with the same standard labels (service.name, service.namespace, deployment.environment) so metrics/logs/traces actually correlate instead of being three separate silos.
5. Build one RED dashboard (Rate, Errors, Duration) per service before anything fancier.
6. Alert on symptoms, not noise, using multi-window multi-burn-rate alerts (straight from the Google SRE book) so you can actually tell a 3am-page incident from a "ticket, deal with it Monday" one.
7. Only then layer on business dashboards and deeper investigation tooling.

The full write-up has the actual configs, PromQL, and a worked burn-rate alerting example for a payments SLO: https://omarghader.github.io/monitoring-infrastructure-guide-2026/

Curious how this lines up with how other people here approach it, particularly if anyone's doing SLO-first in practice vs. it staying an aspiration on a wiki page somewhere.