u/Lucas-Holmes-722

Fourteen warehouses, one per team, because that is how finance got its split. Nine of them served a single dashboard each and sat idle using nearly 40% of compute.

It took three weekends to merge them down to 4 by workload and the BI team fought it the whole way. Ran the same query before and after.

SELECT warehouse_name,
       ROUND(SUM(credits_used_compute),1) AS compute,
       ROUND(SUM(credits_attributed_compute_queries),1) AS attributed,
       ROUND(100*(1-SUM(credits_attributed_compute_queries)
             /NULLIF(SUM(credits_used_compute),0)),1) AS idle_pct
FROM snowflake.account_usage.warehouse_metering_history
GROUP BY 1 ORDER BY 4 DESC

                 COMPUTE   ATTRIBUTED   IDLE_PCT
before, 14 wh      4,210        2,530       39.9
after, 4 wh        2,890        2,540       12.1

Attributed barely moved and we stopped paying for 1,320 credits of nothing.

Finance used to read the per-team figure straight off warehouse names. Now the 2,540 splits by query tag and the 350 idle sits on one warehouse that four teams share. I can spread it or leave it unowned and neither is a measurement.

What do you put in the per-team column after a consolidation?

I just did a security review of two cloud cost tools, buyer-side this time.

Both came back clean. SOC 2 Type II and ISO 27001 ready, both read only, PointFive on Azure is a service principal scoped to billing and resource metadata with no data plane access. Googling is pointfive safe beforehand turned up nothing a security lead would accept.

None of the 300 questions touched what the tool ends up holding. A month in, it has a full inventory of the estate and our monitoring holds much the same without anybody making me fill out a form about it.

Would you expect inventory scope to show up in a system description?

Managing 40 or so devs. Finance asked for AI spend by team back in July and I did what everyone here would do, went and got it myself. Claude Code writes a session file per project on each machine with the token counts in it, so a nightly job ships those totals up and a small dashboard reads them. Took about a week. Per repo, per person, even the cache reads against output, I can see how much is context overhead. Genuinely good.

The problem is that is one tool. Cursor does not expose anything close to that, so that chunk of the bill is a seat cost I have stopped trying to attribute. Copilot gives me an org total and nothing underneath it. Meaning the dashboard I proudly showed finance covers maybe a third of what we spend, and the other two thirds is a shrug.

I have looked at the enterprise options and they feel sized for a company four times ours. I am also aware the job I wrote has one owner, me, and the last internal tool I built like this quietly broke about six months after I stopped looking at it.

What I still have not found is anything that covers the other two

Did an audit before a refactor and the scary part was how much is running that no module owns at all. Stuff clickopsed during an incident three years ago that quietly became load bearing. Leftovers from a stack that half destroyed and left a few resources alive on dependencies.

No one touches it because no one knows what it does and the engineer who built it left years ago. So it just bills, quarter after quarter. Ours was a few hundred resources.

If you inherit an account, scan for anything unmanaged and untagged before you trust your own IaC. It's a bigger pile than you'd think.

Not fishing for tool recs, just flagging it in case someone's about to refactor and assumes state is the whole picture. It isn't.

We are a fair way down the everything-as-code road. Modules for the infra, pipelines that plan and apply themselves, drift caught without anyone watching. It mostly runs itself, which was the entire point of the last two years.

Our container base images never fit that model. A while back, security wanted minimal hardened images, which led us to build a set off a slim base and stood up a rebuild pipeline. It works though doesn’t maintain itself the way the rest of the stack does. Every upstream CVE, every base bump, every request for a package I left out.

It has quietly become the least automated, most me-shaped part of an otherwise hands-off platform. Strange place to end up when the whole reason for any of this was to stop being the bottleneck.

There are maintained catalogs now, some of them free and I am starting to think a base image should just be a pinned dependency like everything else in this repo. Half hoping someone who already made the jump tells me it was fine. Half expecting to hear about the new problem I have not thought of yet.

Every time snowflake cost optimization lands on me it comes down to the same couple of boring things. Credits were up a good bit this quarter, finance flagged it and it wasn't more queries or more users. Usage barely moved.

Went through query history and it was mostly one warehouse that a BI tool kept awake almost all the time because auto-suspend was set to 3600 seconds instead of 60. Whoever set that has left. Two other warehouses were sized a couple of steps bigger than the work ever needed.

Dropped the suspend and resized the two big ones, that part was easy. What I keep going back and forth on is how far down you can size a warehouse before it starts hurting, because too small and the queries just queue and run longer and you give the savings straight back.

Where do you draw the line on warehouse size and are you catching the auto-suspend stuff as it happens or only when the bill jumps?

 We reserved a pool of GPUs a while back for training and inference and I finally pulled the utilization for a capacity review last week. Under 30% on average. We are paying for all of it and using less than a third.

Some of it makes sense. A few boxes run batch jobs a couple times a day and you cannot shrink those, they have to be there when the job fires. But the rest is just idle. Two of them turned out to be pinned by notebooks people opened and walked away from, one had been up for weeks.

Finance keeps on me about the gpu cost. I want to trim it too. Problem is the second I bring up handing any GPUs back the ML team is not having it. Their whole thing is if a GPU is not free the experiments queue and they lose half a day. So it stalls there. Every time.

What I keep getting stuck on is I cannot see who owns what. DCGM gives me per-node utilization. It does not tell me who reserved the idle box or whether they still need it, so it stays vague and nobody wants to be the one who hands a GPU back.

How do you decide when a reserved GPU is safe to give back?

Not sure if this is the right sub for a cost gripe, but I gotta ask.

Our Databricks DBU spend has quietly become one of our top three cloud line items and I'm struggling to explain to anyone why. A big part of it seems to be people running everything on all-purpose clusters instead of job clusters because it's what they know, plus interactive clusters with auto-terminate set way too high, so they idle for hours between runs.

The usage UI gives me totals but tying a DBU number back to a specific notebook, job, or team is far more manual than it should be. I've started exporting the system tables and building my own attribution in, of course, yet another dashboard nobody will look at.

Before I go further down that road, is there a saner way people are tracking Databricks cost per team and catching the idle cluster waste? Appreciated.

Every vendor in our stack has an AI story now and they all swear theirs catches more with fewer false alarms but board doesn't buy that. They want to know if the money we spent made us any safer and I couldn't answer that with a straight face.

We track finding counts, MTTR, coverage numbers and all it tells me is that the tool is busy. A noisy scanner throws up the same green dashboard as one that surfaces the three things worth fixing.

What I'm after is closer to how you'd grade any classifier. How often it's right when it flags something and how hard it is to see what it misses entirely. precision and recall if you want the terms for it. No vendor will hand that over on a test set we both agree on, so you take the datasheet on faith right up until you've signed.

For security leaders here who report to a board or an audit committee, what do you present to demonstrate that a tool is earning its place?