Over reliance on LLMs: Has anyone else run into the problem where LLM costs look fine during MVP, then quietly become painful as usage grows?
We keep seeing startups route even simple backend and application logic through frontier models. This includes extracting fields, classifying support tickets, normalizing messy records, matching entities, converting text into JSON, scoring categories, summarizing highly templated notes, and deciding the next workflow step. To my surprise, I recently discovered there is a well regarded (technical prowess) unicorn startup whose engineering team also fell into this tarpit.
Early on, this makes complete sense. Prompting an LLM is much faster than designing schemas, writing parsers, building classifiers, maintaining ETL jobs, or figuring out the proper backend logic. It lets you ship before you fully understand the shape of the problem.
The problem starts when the product actually works. Usage increases. The same prompt chains run thousands or millions of times. Latency starts to matter. Costs creep up. Outputs remain non-deterministic. Core backend logic ends up hidden inside prompts, but the team is no longer sure how to turn those prompts into production-grade software.
My team consists of SWEs, MLEs and applied researchers with more than a decade of traditional ML and NLP experience, so we dealt with it by migrating/replacing many parts of our app post-MVP with more production ready approaches. However, I'm not sure this is applicable to the majority of other founders we see today, many of who are not even technical.
For teams running LLMs in production, how much of your LLM traffic is truly open-ended reasoning, and how much is repetitive extraction, classification, normalization, transformation, or workflow routing? Have costs become a real issue as usage scaled? Have you successfully replaced LLM calls with traditional backend logic, smaller models, or ETL pipelines, or is inference now cheap enough that it is better to keep everything as prompts?