u/SaitejBuilds

I’ve been digging deeper into evals and AI release workflows, and there’s one part I’m especially interested in.
Teams can already compare prompts/models on quality, latency, and other eval metrics.
But I’m curious how people handle the tradeoff between **quality and cost**.
For example, suppose a new model:
improves task success from 85% to 90%
but doubles the cost per request
Is that a good change?
The answer probably depends on the actual customer outcome, not just the eval score or token cost individually.
I’m experimenting with comparing a baseline and candidate on the same test set, then looking at **cost per successful outcome** rather than cost per request.
The goal is to answer something closer to:
**“Did this change improve the product enough to justify what it costs?”**
For people running LLM features in production, how are you making this decision today?
Is this already part of your eval pipeline, handled manually, or mostly monitored after deployment?

I’ve been digging deeper into evals and AI release workflows, and there’s one part I’m especially interested in.
Teams can already compare prompts/models on quality, latency, and other eval metrics.
But I’m curious how people handle the tradeoff between quality and cost.
For example, suppose a new model:
improves task success from 85% to 90%
but doubles the cost per request
Is that a good change?
The answer probably depends on the actual customer outcome, not just the eval score or token cost individually.
I’m experimenting with comparing a baseline and candidate on the same test set, then looking at cost per successful outcome rather than cost per request.
The goal is to answer something closer to:
“Did this change improve the product enough to justify what it costs?”
For people running LLM features in production, how are you making this decision today?
Is this already part of your eval pipeline, handled manually, or mostly monitored after deployment?

I’ve been digging deeper into evals and AI release workflows, and there’s one part I’m especially interested in.
Teams can already compare prompts/models on quality, latency, and other eval metrics.
But I’m curious how people handle the tradeoff between quality and cost.
For example, suppose a new model:
improves task success from 85% to 90%
but doubles the cost per request
Is that a good change?
The answer probably depends on the actual customer outcome, not just the eval score or token cost individually.
I’m experimenting with comparing a baseline and candidate on the same test set, then looking at cost per successful outcome rather than cost per request.
The goal is to answer something closer to:
“Did this change improve the product enough to justify what it costs?”
For people running LLM features in production, how are you making this decision today?
Is this already part of your eval pipeline, handled manually, or mostly monitored after deployment?

I've been spending a lot of time building different products lately and one thing I'm realizing is that actually making the thing work is only half the battle

You can spend weeks fixing bugs, improving the ui adding features etc and still have nobody care enough to actually use it or pay for it

I'm starting to think distribution and understanding what people really want is harder than the building itself. For people here who already got their first 10 or 100 paying users, what actually worked for you in the beginning?

cold outreach? reddit? seo? talking to people directly? something else?

Would love to hear what worked before you had an audience.

I’ve been building something called AI Economics CI.
The idea came from a simple problem.
I changed an AI feature, and technically everything worked. All the API calls succeeded.
But the actual result got worse.
Customer success dropped from 87.5% to 66.7%, and the new version also cost more.
So even though nothing “broke,” it was still a bad change.
That made me ask:
Why do we only check whether the AI works?
Why not also check whether it became more expensive or less useful before we ship it?
So I built a tool that compares the old version with the new version and gives a simple result:
Safe to merge
Block merge
Collect more evidence
It’s still very early, and I’m trying to understand whether other people building AI products run into the same problem.
Do you usually catch cost or quality issues before release, or only after users start noticing them?

I just launched my first mobile app, DriveFlo, on the Google Play Store—and it completely changed how I think about building products.
I used to believe the hard part was getting an idea into a working app. But once I actually shipped DriveFlo, I realized something important:
building the app is only half the challenge… shipping it is a whole different game.
The moment it went live, I ran into a wave of things I hadn’t fully prepared for—bugs that only show up in real usage, edge cases I never considered, release friction, UI tweaks, and a long list of small but important fixes that don’t show up during development.
And now I’m starting to understand the next big lesson:
distribution is just as hard—if not harder—than building.
You can spend months creating something genuinely useful, but if people never discover it, none of it really matters.
So right now I’m shifting focus from adding features to figuring out how to actually get the first real users.
For anyone here who’s launched an app before:
what actually worked for getting your first 100 users?
organic content? reddit? app store optimization? paid ads? something else?

Checkout: https://drivefloapp.com

I’ve been building something called AI Economics CI.
The idea came from a simple problem.
I changed an AI feature, and technically everything worked. All the API calls succeeded.
But the actual result got worse.
Customer success dropped from 87.5% to 66.7%, and the new version also cost more.
So even though nothing “broke,” it was still a bad change.
That made me ask:
Why do we only check whether the AI works?
Why not also check whether it became more expensive or less useful before we ship it?
So I built a tool that compares the old version with the new version and gives a simple result:
Safe to merge
Block merge
Collect more evidence
It’s still very early, and I’m trying to understand whether other people building AI products run into the same problem.
Do you usually catch cost or quality issues before release, or only after users start noticing them?

I've been spending a lot of time building different products lately and one thing I'm realizing is that actually making the thing work is only half the battle

You can spend weeks fixing bugs, improving the ui adding features etc and still have nobody care enough to actually use it or pay for it

I'm starting to think distribution and understanding what people really want is harder than the building itself. For people here who already got their first 10 or 100 paying users, what actually worked for you in the beginning?

cold outreach? reddit? seo? talking to people directly? something else?

Would love to hear what worked before you had an audience.