Fine-tuned a 1.7B model that beats gpt-5.4 on merchant extraction and runs 300x cheaper.
I took Qwen3-1.7B and fine-tuned it on one narrow task: turning messy bank transaction descriptors into clean merchant names + categories. Stuff like "TST-BLUE FORK 8841 HAMILTON" → Blue Fork Kitchen / Restaurants & Dining.
I built a sealed 60-row eval from my own real bank statements and ran the same scorer across everything:
- tuned 1.7B → 91.7% category / 78.3% merchant
- base Qwen3-1.7B → 63.3% / 66.7%
- gpt-5.4-nano → 85.0% / 56.7%
- gpt-5.4 → 96.7% / 70.0%
So it beats nano across the board and actually beats gpt-5.4 on merchant extraction (78.3 vs 70.0), while trailing it a bit on category.
where it failed: obscure local merchants it had never seen. It got the name perfect every time but whiffed on category, because that's not reasoning, it's just a lookup. So I bolted on a merchant directory: resolve each unknown once, cache it forever. Model does parsing, directory does long-tail recognition, and they split cleanly along the model's failure line. Combined accuracy hits ~98% category, past gpt-5.4.
Cost on a single L4: ~125k req/hr at ~$0.006–0.008 per 1k transactions. Roughly 6x cheaper than nano, 300x cheaper than gpt-5.4. And for bank data, the fact that nothing leaves your own hardware is honestly the biggest win.
Takeaway: for narrow, high-volume tasks, a small fine-tuned model + your own data + a real eval beats reaching for a frontier model. You don't need frontier scale for most of this stuff.
I'm starting to do this kind of build for companies, so if you've got a narrow high-volume task drowning in API costs, my DMs are open, but mostly just wanted to put the numbers out there. Happy to get into the weeds on the pipeline in the comments.
