I built a tiny logger to prove which feature was eating my OpenAI budget. Fell into a rabbit hole of people getting hit way harder
This started as a small itch.
OpenAI gave me one number every month: total spend. I wanted to know which feature in my app was spending it. I've got a code-gen feature, a summarizer, and a couple of smaller things running in the background. My money was on the code one, obviously, but I had no way to actually check.
So I built a tiny thing for myself. Every response already comes back with a usage object, so after each call I log the token counts plus a feature name, then price them myself. One line per call. I only wanted visibility, nothing more.
The "price them myself" part was way more annoying than I expected:
- cached reads bill at a different rate than normal input, and cache writes can cost more than normal input
- reasoning tokens bill as output even though you never see them
- providers disagree on whether cached tokens sit inside the prompt count or next to it. I double-counted them for a while before catching it
- audio tokens can cost several times the text rate on the same model, so blending them with text gives you garbage
Then I got curious whether anyone else cared, and went looking. Turns out my small itch was the door into a rabbit hole:
- someone on Google's AI dev forum got around €290 in search charges attached to like €10 of actual LLM usage, and their spend cap never caught it
- a dev on DEV wrote that the job that blew up their bill was green on every dashboard they had. A timeout plus auto-retry on long-context calls was quietly doubling costs, and it took weeks to figure out
- one startup reported a $113k AI bill in a single month
For me it was curiosity. For them it was money leaving before anyone could see where it was going.
Full honesty: I have little to no experience building products. What I'm decent at is digging into other people's problems, so I kept going. More features, more providers and models, learning what proxies and gateways actually are, reading how Langfuse and Helicone handle this. The logger eventually turned into an actual product (LLMtrack), but honestly I'm more curious how you all deal with it:
- Do you track LLM cost per feature or per customer at all, or just watch the provider total?
- If one feature started costing 5x more today, how long until you'd notice?
- Anyone who rolled their own: did you hit the cached-token double-count thing too, or was that just me?
Oh, and yes, it was the code feature eating most of my money, about 70% of the bill. I knew it. Funny part: by tokens, the summarizer was way bigger (129M vs ~30M), and it cost $14 total. If I'd only looked at token counts, I'd have blamed the wrong thing.