I'll go first:
"We have 500 alerts, but nobody knows which ones actually matter."
Your turn.
Complete this sentence:
>
Some possibilities:
Everyone ignores PagerDuty
The dashboard has 73 panels
Alerts fire only after customers complain
Nobody knows what half the metrics mean
"Just restart the pod" fixes everything
Logs contain everything except the thing you're looking for
What's yours?
Every time i add another log statement or metric, i feel like i'm playing fortune teller. "let me predict what might break tomorrow and instrument that." we are constantly chasing our tails, adding instrumentation after incidents because we didn't guess right the first time. There's always some code path we didn't think to instrument having a system that can map your codebase and show you what's happening without manual effort changes the game significantly. How much time is your team spending on instrumentation? and do you feel like you are actually capturing what matters or just covering the obvious stuff
Hi,
I built a small TUI tool for those who work day in and day our with logs.
Demo and docs: https://nightbuildlabs.github.io/lumen/
Repo: https://github.com/nightbuildlabs/lumen (MIT)
Recently started working extensively with Kubernetes. Debugging logs was hard with grep. Started exploring TUI log processors, what was missing was the existing tools did not handle the structure well. The structure is right there in the logs — field names, severities, timestamps, but all of them required me to understand the structure or regex or some sort of pre setup.
So I built lumen. cat anything.log | lumen:
/, type three characters, and it autocompletes field names discovered from the data you just piped in, ranked by occurrence count — no schema files, everstatus_code=500 flips to a query mode backed by an embedded DuckDB; a 1M-row result scrolls without materializing more than a viewportWhy build it when there's lnav?
- I found the learning curve of lnav to be comparatively steep; I need to know the exact pattern or attribute of the log item I am looking for.
- While juggling multiple projects this becomes challenging to an extent - hence built out auto indexed logs; now I don't need to remember the regex and can query like a search box.
Currently support JSON and logfmt; adding support for nginx, syslog, django and cloudwatch soon.
Give it a spin! Would love some feedback.
We just released a benchmark of OpenObserve vs Prometheus and Grafana Mimir.
OpenObserve consistently outperforms Prometheus and Mimir by 5x-15x on various queries. This test was done on very high cardinality dataset that pushes the systems to their limits.
While high performance gives you much better experience, it also allows you to use less hardware and save money on your observability costs.
https://openobserve.ai/blog/openobserve-vs-prometheus-mimir-metrics-benchmark/
I'm trying to model an agent that gets triggered whenever a production incident occurs. I'm looking for inputs on a few things based on your past experiences.
Whenever an incident occurs, what are some of the most common reasons? For example, it could be a false alarm, it could be an issue due to a new upgrade, network issues or a crash due to some bug in one of the services etc. Out of 100 incidents, what are the chances that it could be a false alarm, upgrade issue, bug in code etc.
I would like to know what are all the common issues or reasons that I can add as part of a v1 for my agent? How do I think through this as I've never worked on such a thing till now?
If you haven’t heard of chmonitor.dev yet: it's a UI to monitor and overview your ClickHouse cluster at any size. Queries, merges, parts, replication, health. An AI advisor on top (of course). Open source.
Github: https://github.com/chmonitor/chmonitor a star helps a lot 🙏
Homepage: https://chmonitor.dev
Got tired of juggling separate tools for uptime, traces, errors and logs, so I built one that does all of it without being a pain to set up. SDKs for Go, TS, Python, Rust, plus a host agent that explains why an incident happened instead of just dumping graphs.
Still rough around the edges. Looking for a few people to try it and tell me what's broken. Link in the comments.
Hey everyone! I’m building an open-source production readiness checklist with extensive testing and quality coverage.
I would appreciate feedback on test strategy, integration testing, acceptance testing, regression, test environments, test data, exploratory testing and nonfunctional testing.
If anything is missing, unclear or unrealistic, please let me know. Corrections and pull requests are very welcome.
Thanks!
https://github.com/MarinJursic/production-readiness-checklist
Hello Everyone,
I am in an extreme dilemma right now in terms of choosing my career path and would like to get inputs/advice.
I have 13 years of experience in network security and have worked on R&S, Firewalls, LBs, WAF, Wireless, ISE etc. Recently i attended a job interview and got selected for the position "Senior network capacity and data engineer". As i discussed with the hiring team, the role is about predictive analysis, capacity forecasting, coordinating with internal teams to plan for link and device upgrades before an impact is seen. i will not be having any hands on work on the network devices but its more like working across stakeholders in an organization and coming up with data points on where capacity increase is required proactively. i understand this role comes under "Observability" tower. With current advancements in AI and cybersecurity roles surging in demand, i am not sure whether to take this role or not.
Please if someone can shed some light on network capacity planning and how a typical day, month or a year look like in terms of growth and how valued this role is in current job market landscape.
I have also posted in networking sub and asking for inputs. Thanks a lot
The way we operate is we have teams in different regions having multiple AWS accounts. Similar way is on GCP. But the thing is that we have lot of nodes & clusters which are not heavy in terms of generating APM data. When I say heavy I mean like 1TB/day - 3 TB per day per host. I believe we are generating data around 30GB-50GB per host per day data which I believe is average or maybe lower side of spectrum. So, when I look around what would it cost me for a BYOC observability tool, its insanely high for per host pricing tool as compared to ingestion based pricing tools. So, I am really confused why companies put per host pricing ? Also is my assumption right that 30GB-50GB per host per day data generation is average in industry ?
When testing our eBPF agent, I don’t always get the same experience as our users, especially in performance critical sections. I realize that the benchmark test suite isn’t always enough, because user’s environments can be completely different from our benchmarks.
My goal was to gather eBPF metrics based on the user’s usage and quickly answer questions about why things are slow (improve MTTR). To do this, I wanted:
Here is a post https://naveensrinivasan.com/posts/2026-08-02-measuring-an-ebpf-cache-without-leaving-the-kernel/
I want to hear if others have better ways to measure this.
This is not another AI generated post.
Hey r/zabbix,
I got tired of staring at a Zabbix dashboard on a second monitor waiting for something to go red, so I built a small Chrome/Edge extension that does it for me: it polls the Zabbix API in the background and fires a sound alarm + desktop notification the moment a new problem appears, even if the browser tab is in the background or the dashboard isn't open at all.
It's free, MIT licensed, and there's nothing hardcoded, no telemetry, no analytics. It only talks to the Zabbix URL(s) you configure.
Some things it does:
- Multi-instance: watch up to 8 independent Zabbix servers at once, each problem badged with its instance.
- 3 auth modes per instance: browser session (zero credentials, just keep a Zabbix tab logged in), API token, or username/password (extension logs in and renews the session itself).
- Re-alarm while a problem stays unacknowledged, until you ack it or mute.
- Working hours aware: reads the Working time from your Zabbix server and stays quiet outside it (list/badge still update).
- Maintenance aware: problems in a maintenance window get tagged MNT and stay silent (or can be hidden).
- Meeting mode: auto-silences sound/notifications while you're in a Google Meet call.
- Snooze a single problem, ack from the popup, per-severity sound/volume, host group filters, min severity, max age, export/import config as JSON, etc.
- Works with Zabbix 6.0 through 7.4 (frontend session + API calls). Haven't validated 8.0 stable yet.
- PT/EN/ES UI, auto-picked from your browser locale.
Chrome Web Store: https://chromewebstore.google.com/detail/zabbix-noc-alerter/nlbihmhpbdfhnglclecbaebnfpjbngep
Source (MIT): https://github.com/opastorello/zabbix-noc-alerter
I'm the author, so full disclosure on that. It's a side project I use daily for our own NOC, figured others watching Zabbix all day might find it useful too.
Happy to take feature requests/bug reports on GitHub, and open to feedback here.
All observability telemetry seems to be moving to clickhouse. I've been seeing this trend for the last 2 years or so and it's really accelerated in the last 6 months. I had heard its because its scalable and cheap but I didn't really understand how and why. So I spent the last few weeks researching to get to the bottom of this. And the reasons I found were a lot more convincing than I had first thought. I wrote a blog on it if anyone is interested: https://telflo.com/blog/why-observability-keeps-ending-up-on-clickhouse
Let me know if I missed something. Would be cool to hear other people's take on it
Your monitoring bill went up 40% this year. Your MTTR didn't move an inch. That's called "The Observability Paradox".
More tools. More dashboards. More alerts. Still 40 minutes at 3 AM trying to find which of 40 services actually broke.
Here's what's really happening: nobody owns the cardinality/log cost until the invoice hits, alerts fire on symptoms instead of user impact, and there's no shared definition of "healthy" — so every incident starts with a debate, not a fix.
New vendor won't fix that. Governance will.
Full breakdown — including the 3 root causes and the framework that got us there 👇
Hi everyone,
I'm doing some early market research around production troubleshooting and log analysis and would really appreciate feedback from people who actually deal with production systems.
I'm exploring a tool where engineers could interact with their logs using natural language, investigate production incidents, correlate events across different systems, perform RCA, and generate custom analysis/visualizations from historical log data.
Before building this further, I want to understand the actual problems engineers face today — how they investigate incidents, where existing tools fall short, how much time RCA takes, and whether this is a problem worth solving.
I've made a short 2–3 minute anonymous survey:
If you work with SRE, DevOps, infrastructure, backend, Kubernetes, networking, observability, or production operations, your experience would be especially valuable.
I'm looking for honest feedback, including negative feedback. I'm trying to validate the problem, not just validate my idea.
Thanks!
Been evaluating tools for a while now because in my previous company every time something broke downstream, tracing it back through the warehouse, dbt, BI layer took way longer than it should have.
Monitoring/cost observability/observability/cost management or however you want to call it for Snowflake means different things depending on who's selling it, some tools are really cost dashboards with lineage bolted on, some are full data catalogs that happen to touch Snowflake, and a few actually do both well. Here's where I landed:
Snowflake native - Free! (which is always awesome) built in, and fine for a quick "what feeds this table" check inside a single Snowflake account. Doesn't extend past the warehouse though, so if your incident actually started in Fivetran or your BI layer, you're out of luck. Good starting point, not a real monitoring solution on its own. works well with coco
Monte Carlo- Probably the best known data observability platform, and the lineage is genuinely strong, column level, cross warehouse, good for root causing data quality incidents. Downside is it's built for data quality first not for cost optimization so if thats what youre looking for then great but if FinOps is your main pain point you're paying enterprise pricing for a lot of stuff you won't use.
Atlan- More of a data catalog with observability features than the reverse. Lineage is excellent if you also want governance, ownership tagging, and a searchable catalog for the whole org. Heavier lift to roll out and honestly overkill if lineage is mainly a debugging/cost tool for a data platform team rather than a company wide catalog initiative.
Seemoredata- Full stack wide lineage (warehouse, pipeline, BI), so when a cost spike or broken dashboard shows up you can trace the actual dependency chain instead of jumping between four tools. Not a catalog platform like Atlan, but for "why did this cost or break, and what's the blast radius," it's the strongest of the cost focused tools, closer to Monte Carlo level lineage than Select or Keebo.
Bigeye- Solid observability tool, lineage is decent (comparable to Monte Carlo level). Same category problem as Monte Carlo though it's optimized for data quality monitoring, cost isn't really the focus.
Metaplane (now datadog)- Lighter weight observability tool, easier to set up than Monte Carlo or Bigeye. Lineage is there but shallower good for smaller teams that want quick anomaly detection without a huge lift.
Select (now part of DoIT)- Really good at what it does. query level Snowflake cost visibility, warehouse sizing, workload analysis. Lineage-wise it's more query/workload level than true stack-wide lineage — you get great cost data but not much cross-system context (pipeline → BI).
Keebo- Automated warehouse tuning is the strong suit here. Lineage is minimal it's really not a lineage tool, so I'd only put it on this list with a big asterisk.
Curious what others are running, especially if you're doing lineage across dbt Snowflake BI tools without needing a full catalog rollout. Also curious if anyone's actually used Atlan or Monte Carlo primarily for cost, feels like using a sledgehammer but maybe I'm wrong.
I'm Kerolos, founder of OutageDeck.
The question I wanted to answer quickly was: is my app broken, or is AWS, Cloudflare, GitHub, OpenAI, Stripe, or another dependency reporting an incident?
I built OutageDeck to read and normalize the official status sources for 172 cloud and SaaS providers. It shows provider and service-level state, current incidents, source freshness, incident history, and 90-day uptime in one place.
What works today:
- check a whole dependency stack without creating an account
- free email alerts for up to five providers
- Slack, Teams, Discord, and webhook destinations on paid plans
- a keyless JSON API, RSS feeds, badges, CLI tools, and an MCP endpoint
- custom Statuspage or Instatus feeds on paid plans
What it deliberately does not do is pretend an official status feed is independent truth. Providers can acknowledge incidents late, so I think a serious production setup should use two signals: the vendor feed plus synthetic probes from your own regions and accounts.
You can try the stack checker here:
I would value blunt feedback from people who handle incidents. What would this need to do before you would keep it open or route its alerts into your on-call workflow?
Hello Community,
Standard budget caps tell you that an agent failed, but they don't tell you why (did it loop on a broken tool? did its context spiral?).
To solve this, we open-sourced state-harness (https://github.com/vishal-dehurdle/state-harness); a lightweight Python/Rust runtime guard that tracks a normalised token growth ratio (inspired by discrete Lyapunov stability) and exports failure signatures straight to OpenTelemetry.