Small models fail tool-calling for different reasons — and sometimes it's an upstream chat-template bug, not the model. I built an MLX tool to tell them apart.
Everyone benchmarks tool-calling with one number: "Model X gets 71% of function calls right." That number can't tell you why the other 29% failed — and the "why" is what decides what you do next.
So I built Toolhound, an MLX-native diagnostic that runs entirely on your Mac and attributes every tool-calling failure to one of four causes:
- `framework_template_bug` — the chat template mangled the tool tokens
- `framework_parser_gap` — the model emitted a rescuable call, the framework parser missed it
- `model_format_failure` — the model can't emit a parseable call
- `model_decision_failure` — valid format, wrong tool/args
What surprised me (Qwen2.5-0.5B / 1.5B, Llama-3.2-3B, 4-bit, on an M2 Pro):
- Qwen2.5-0.5B mostly fails on an upstream chat-template bug — Qwen2.5's template renders its tool-call example with doubled braces `{{"name": ...}}`, and the small model copies it literally. That's not the model's fault. args-correct 29%.
- Qwen2.5-1.5B parses fine (96%) but fails on judgment — wrong tool/args. args-correct 71%.
- Llama-3.2-3B formats perfectly, but wrong arg types + false abstentions. args-correct 61%.
Same benchmark, opposite root causes. A plain accuracy score hides that — and the smallest model's failures aren't even fixable by a better model.
Other things it does:
- 95% bootstrap CIs on every metric (temp=0, so no seed hand-waving — the CI comes from resampling the case set)
- Reports attribution under both a strict and a lenient parser, so you can see the verdict doesn't flip
- Quantifies bf16-vs-q4 damage without confounding it with template differences (asserts identical template first)
- v2 benchmarks existing zero-training fixes (PA-Tool is wired in). Honestly, on my demo run PA-Tool didn't beat baseline on any metric — it flags a result "credible" only when its CI is disjoint from baseline's, and it wasn't (it even hurt 1.5B's arg accuracy). I'd rather the tool tell me that than rubber-stamp it.
https://github.com/Code-byte404/toolhound
Feedback very welcome — especially: which models should I add next, and are the abstention "trap" cases too easy/hard? There are `good first issue's if anyone wants to add a model or help file the template bugs it finds upstream.