We thought we were paying the AI to make a spreadsheet. Turns out we were paying it to have a meeting with itself.
The expensive part wasn’t building the spreadsheet. It was the AI thinking out loud.
We spent a couple of days last week digging into how Shoulder Surf generates spreadsheets from meetings, and we found something we really weren’t expecting. Previously, when someone asked for something like a risk register or action tracker, we let the AI handle the entire job. It would figure out the content, write some Python, build the Excel file, inspect the result, make changes, and eventually give us the finished spreadsheet. It worked, so we never thought too hard about it.
Then we tested a different approach. Instead of asking the AI to actually build the spreadsheet, we asked it to give us the structured content that belonged in the spreadsheet, and then our own code generated the Excel file. Same meeting. Same request. Same AI model. The new approach was 3.6x cheaper and 2.8x faster, and somehow produced more useful content too.
That sent us down a rabbit hole. The old process consumed about 212,886 tokens of conversation context. The new one used 4,433. The expensive part wasn’t Excel. The expensive part was the AI repeatedly rereading the meeting while it worked. It would write some code, run it, inspect what happened, reread the conversation to remember what it was doing, make another change, and repeat. Every lap costs money. The actual code execution is almost free by comparison.
AI should author. Code should compute.
If we’re building a risk register from a meeting, the hard part is understanding what people said and turning that into useful rows. That’s exactly what the AI is good at. But once we know the rows, we don’t need an AI to decide how to put them into cells, apply formatting, freeze panes, or write formulas. Regular code is dramatically better at that.
There are still cases where the sandbox makes sense. If somebody uploads a large CSV and wants calculations, pivots, or hundreds of generated combinations, letting code run in a sandbox is exactly what we want. So we kept both approaches. The interesting part was figuring out where the line belongs.
Then we accidentally made the output worse.
Our first version of the new system generated 24 scenarios where the old system generated 47. That obviously wasn’t great, but the reason turned out to be surprisingly simple. The old system naturally gets multiple chances to think about the problem because it keeps looping through the file-generation process. The new system gets one clean pass, so once it thinks it’s finished, it stops.
We added a short instruction telling it to explicitly consider edge cases. 24 became 46. There was no clever engineering breakthrough there. We basically asked it to think a little harder.
We also stopped trusting the AI with formatting.
We have rules for how Shoulder Surf spreadsheets should look, and we gave those rules to the AI. It ignored some of them. Across two test runs, we found five separate formatting violations. It also failed to freeze panes on all 13 tabs where we asked it to.
So now we don’t ask. Our renderer applies those rules automatically. That turned out to be a useful distinction: a rule in a prompt is a suggestion. A rule in code is a rule.
Same thing with formulas. We don’t let the model write Excel formulas anymore. The AI can tell us, conceptually, “this column should total those rows,” and our code writes the actual formula and cell references. An AI-generated formula can be wrong while looking perfectly reasonable, and I’d much rather have something fail loudly than quietly put the wrong number in a spreadsheet somebody sends to their boss.
The biggest problem had nothing to do with spreadsheets.
Before any of this happens, Shoulder Surf first has to recognize that someone is actually asking for a document. We tested our old detection logic against 216 real requests. It recognized 5.
That was painful. The feature wasn’t failing. People weren’t even getting into the feature.
We had been looking for obvious words like “spreadsheet,” “export,” or “xlsx,” but that’s not how people actually ask for things. They say things like, “Give me everything that could blow up in our faces,” which is a risk register, or “Show me who’s on the hook for everything,” which is basically an action register.
You can’t solve that very well by adding more keywords, so we replaced the keyword rules with a tiny intent classifier. Our detection rate went from about 21% to 98% on our test set, and it costs roughly five hundredths of a cent per request. That may have been the highest-value change in the entire project.
One of our favorite results was when the system refused to make a document.
We tested nine different document types against real meetings. Six worked cleanly. Two refused. One request asked for an options comparison, but the meeting hadn’t actually compared any alternatives. Another asked for a budget, but there were no financial figures in the meeting.
So the system essentially said there wasn’t enough information to make the document. That’s exactly what we want. An empty spreadsheet is mildly annoying. A spreadsheet containing invented financial information is dangerous.
The damage from that kind of mistake doesn’t happen inside the app. It happens a week later when somebody presents the spreadsheet in a meeting and everyone assumes the numbers came from somewhere.
We also caught a smaller version of this. In one budget document, four rows were labeled “Stated” even though there were no amounts attached to them. The AI had interpreted “Stated” as “someone mentioned this topic.” We meant “someone actually stated a number.” Reasonable misunderstanding. Completely wrong result. Now, if something is labeled “Stated” but contains no actual value, we automatically downgrade it.
And then our tests taught us something embarrassing.
Our first 25 routing tests all passed. Usually that feels great. In retrospect, it should have made us suspicious.
The same people who wrote the routing logic wrote the tests shortly afterward, so naturally the tests covered all the situations we had already thought about. We started deliberately looking for situations we hadn’t thought about and found four real bugs.
One of them would have quietly generated a generic table instead of the specialized document somebody asked for. That’s the worst kind of product bug because nothing crashes, nobody reports it, and the product is just slightly worse every time it happens.
That whole exercise changed how we’re thinking about AI features. A lot of the expensive or unreliable parts aren’t actually the impressive AI parts. They’re the boring machinery surrounding them.
Sometimes the best thing you can do is let the AI do less.
And sometimes the biggest bug isn’t that the AI gave the wrong answer. It’s that your system never let the question reach the AI in the first place.