u/Spiritual-Kitchen-79

Been evaluating tools for a while now because in my previous company every time something broke downstream, tracing it back through the warehouse, dbt, BI layer took way longer than it should have.

Monitoring/cost observability/observability/cost management or however you want to call it for Snowflake means different things depending on who's selling it, some tools are really cost dashboards with lineage bolted on, some are full data catalogs that happen to touch Snowflake, and a few actually do both well. Here's where I landed:

Snowflake native - Free! (which is always awesome) built in, and fine for a quick "what feeds this table" check inside a single Snowflake account. Doesn't extend past the warehouse though, so if your incident actually started in Fivetran or your BI layer, you're out of luck. Good starting point, not a real monitoring solution on its own. works well with coco

Monte Carlo- Probably the best known data observability platform, and the lineage is genuinely strong, column level, cross warehouse, good for root causing data quality incidents. Downside is it's built for data quality first not for cost optimization so if thats what youre looking for then great but if FinOps is your main pain point you're paying enterprise pricing for a lot of stuff you won't use.

Atlan- More of a data catalog with observability features than the reverse. Lineage is excellent if you also want governance, ownership tagging, and a searchable catalog for the whole org. Heavier lift to roll out and honestly overkill if lineage is mainly a debugging/cost tool for a data platform team rather than a company wide catalog initiative.

Seemoredata- Full stack wide lineage (warehouse, pipeline, BI), so when a cost spike or broken dashboard shows up you can trace the actual dependency chain instead of jumping between four tools. Not a catalog platform like Atlan, but for "why did this cost or break, and what's the blast radius," it's the strongest of the cost focused tools, closer to Monte Carlo level lineage than Select or Keebo.

Bigeye- Solid observability tool, lineage is decent (comparable to Monte Carlo level). Same category problem as Monte Carlo though it's optimized for data quality monitoring, cost isn't really the focus.

Metaplane (now datadog)- Lighter weight observability tool, easier to set up than Monte Carlo or Bigeye. Lineage is there but shallower good for smaller teams that want quick anomaly detection without a huge lift.

Select (now part of DoIT)- Really good at what it does. query level Snowflake cost visibility, warehouse sizing, workload analysis. Lineage-wise it's more query/workload level than true stack-wide lineage — you get great cost data but not much cross-system context (pipeline → BI).

Keebo- Automated warehouse tuning is the strong suit here. Lineage is minimal it's really not a lineage tool, so I'd only put it on this list with a big asterisk.

Curious what others are running, especially if you're doing lineage across dbt Snowflake BI tools without needing a full catalog rollout. Also curious if anyone's actually used Atlan or Monte Carlo primarily for cost, feels like using a sledgehammer but maybe I'm wrong.

Hey community...I dont usually post things that we do at Seemoredata in this community but this is so beautiful I had to share it:

Its scary to play around with warehouse vertical sizing on Snowflake due to performance degradation...so almost all companies that we've seen over provision....but there is something super interesting...we've also seen many companies underprovision! i.e. they can pay less and have faster runtime of queries if they add more compute power.

The graphs below shows a real workload usecase of one of our customers.

It has periods of significant remote spill, so leaving it undersized can dramatically increase execution time.

Our latest SmartPulse logic can now do both-

Upsize when the performance improvement justifies the additional cost. Then downsize again when it doesn’t.

In the example below, the same warehouse moves between L, XL and 2XL throughout the week based on how the workload actually behaves.

We also rebuilt a significant part of SmartPulse over the past few weeks:

improved hourly predictability
better Gen2 decisions
new versioning infrastructure

...and a lot of work on the underlying optimization logic.

We're always working to optimize our algorithms and I am super excited to see what Seemoredata's Smart Pulse will be able to do with AI infrastructure optimization on GPUs once we get there...theoretically it will be able to reduce prompt runtime and GPU usage by around 33%.

a special shoutout to Guy Biecher the brains behind this beautiful optimization

so much work to do!

Run every workload on the right compute at the right time.

feel free to connect on linkedin -> https://www.linkedin.com/in/yanivleven/
or follow us -> https://www.linkedin.com/company/seemore-data/posts/?feedView=all

https://preview.redd.it/njg8ea2z6zhh1.png?width=2964&format=png&auto=webp&s=5b281471047d26f60eb766bcd4b01db694eae901

https://preview.redd.it/guza4xfy6zhh1.png?width=2611&format=png&auto=webp&s=f43d031fbe7a291fcecacc9a574e75b8d4021b1b

Snowflake added a new Account Usage view: SNOWFLAKE.ACCOUNT_USAGE.QUERY_METERING_HISTORY.

why this matters?

For standard warehouses, Snowflake already has QUERY_ATTRIBUTION_HISTORY.

The important change is that QUERY_ATTRIBUTION_HISTORY does not include jobs executed by Adaptive Warehouses. Snowflake now has a separate view for that: QUERY_METERING_HISTORY.

That matters because Adaptive Warehouses are supposed to make warehouse management more automatic. But as more of the compute behavior becomes adaptive, teams still need to understand:

• Which queries consumed credits?
• Which users, roles, tags, or query patterns drove the cost?
• How much was compute vs cloud services?
• Which workloads became expensive under Adaptive Warehouse behavior?

The new view includes fields like:

• QUERY_ID
• WAREHOUSE_NAME
• USER_NAME
• ROLE_NAME
• QUERY_HASH
• QUERY_PARAMETERIZED_HASH
• QUERY_TAG
• CREDITS_USED_COMPUTE
• CREDITS_USED_CLOUD_SERVICES
• CREDITS_USED

A few important details:

• It is available in SNOWFLAKE.ACCOUNT_USAGE
• It covers the last 365 days
• Its not real time! -> Latency can be up to 1 hour
• A long-running query can have multiple rows, one per metering hour (this makes cost attribution an issue, but can be worked around)
• Some columns can be NULL while the query is still running

To me, this is a meaningful step for Snowflake FinOps because it closes an important visibility gap around Adaptive Warehouses that we were trying to provide our customers.

So as for me, in Seemoredata we think that query level metering is super important, knowing which query burned credits is useful and the first step for cost and performance optimization. Especially in the adaptive warehouse that does a great job for performance optimization.

But when you look at optimization beyond performance, mainly the balance of cost to SLA, it usually also requires knowing workload intent, business context, SLAs, recurrence patterns, downstream impact, and what guardrails are needed.

So we are now actively working on adding this new piece of information to our SmartPulse - Autonomous Resource Management for Snowflake engine.

Have any of you started playing around with the Adaptive warehouse? interested to hear how you see it from a usability perspective.

My writing is stream of consciousness; fiction....

how do I get it in front of people to get more subscribers and constructive feedback?

I understand that in this thread you cant post your substack links...

any help here would be much appreciated. thanks!

שלום. התחלתי לפרסם חלקים מהספר שאני כותב פה

ממש אשמח לתגובות...

אני לא מנסה למכור שום דבר רק באמת להבין אם הכתיבה שלי עוברת מסך

תודה מראש