r/oMLX

Everyone benchmarks tool-calling with one number: "Model X gets 71% of function calls right." That number can't tell you why the other 29% failed — and the "why" is what decides what you do next.

So I built Toolhound, an MLX-native diagnostic that runs entirely on your Mac and attributes every tool-calling failure to one of four causes:

- `framework_template_bug` — the chat template mangled the tool tokens

- `framework_parser_gap` — the model emitted a rescuable call, the framework parser missed it

- `model_format_failure` — the model can't emit a parseable call

- `model_decision_failure` — valid format, wrong tool/args

What surprised me (Qwen2.5-0.5B / 1.5B, Llama-3.2-3B, 4-bit, on an M2 Pro):

- Qwen2.5-0.5B mostly fails on an upstream chat-template bug — Qwen2.5's template renders its tool-call example with doubled braces `{{"name": ...}}`, and the small model copies it literally. That's not the model's fault. args-correct 29%.

- Qwen2.5-1.5B parses fine (96%) but fails on judgment — wrong tool/args. args-correct 71%.

- Llama-3.2-3B formats perfectly, but wrong arg types + false abstentions. args-correct 61%.

Same benchmark, opposite root causes. A plain accuracy score hides that — and the smallest model's failures aren't even fixable by a better model.

Other things it does:

- 95% bootstrap CIs on every metric (temp=0, so no seed hand-waving — the CI comes from resampling the case set)

- Reports attribution under both a strict and a lenient parser, so you can see the verdict doesn't flip

- Quantifies bf16-vs-q4 damage without confounding it with template differences (asserts identical template first)

- v2 benchmarks existing zero-training fixes (PA-Tool is wired in). Honestly, on my demo run PA-Tool didn't beat baseline on any metric — it flags a result "credible" only when its CI is disjoint from baseline's, and it wasn't (it even hurt 1.5B's arg accuracy). I'd rather the tool tell me that than rubber-stamp it.

https://github.com/Code-byte404/toolhound

Feedback very welcome — especially: which models should I add next, and are the abstention "trap" cases too easy/hard? There are `good first issue's if anyone wants to add a model or help file the template bugs it finds upstream.

M5 128 GB 40 GPU Cores

For local Apple Silicon inference with Qwen oQ4 models, long contexts, and agentic workloads, MTP appears to be a net negative. Disable it unless your own benchmarks prove otherwise.

My halftop

I'll preface by saying i'm not a developer.
i'm just curious and eager to learn more on LLMs and coding.

I have opencode setup wit oMLX on a m1 max (40c) 64GB
i've been going through the oMLX benchmarks and looking through best options for Qwen (general coding) and Gemma (general research/reasoning)
https://omlx.ai/benchmarks

This is where i think i'm getting confused.
I'll apologize in advance if my qtns are somewhat amateurish.

i get i should be looking at the larger models (e.g 30B)
I understand a higher quant is preferred for coding (e.g 8bit)
with context though, shouldn't i be looking at higher context for coding sessions. If that is the case, doesn't that in turn lead to a larger KV cache size and chew in more onto memory.

Update: yes I now realize that oMLX has its own panel. This is redundant if you're only using oMLX as a server, but my tool also control llama.cpp, which doesn't have one. So skip for oMLX.

------------

I got tired of opening Terminal every time I wanted to start my server or switch models, so I built a menu bar app. I've been using this for a little while now and felt it was good enough to share with others, who hopefully been thinking the same thing.

GitHub: https://github.com/cporto/llm-menubar
Download (DMG): https://github.com/cporto/llm-menubar/releases/tag/v0.2.1

What it does:

• Start / stop / restart your server from the menu bar
• Switch models — unloads the current one automatically, loads the new one
• llama.cpp + oMLX — switch backends from the menu
• Opens the server dashboard in your browser on start — llama.cpp's web UI or oMLX's admin panel
• First-run wizard — finds your binary, picks your models folder, sets up launchd for you
• Remembers your last model and auto-loads it on launch
• Live status — animated tray icon with elapsed timers while starting and loading
• One-model-at-a-time — keeps RAM in check, unloads before loading

Runs on launchd, not as a child process — server keeps running if you quit the app, and the app picks up a server that's already running.

Hey everyone! It’s been a while, and I’m back with oMLX 0.4.5.dev1.
https://github.com/jundot/omlx/releases

I’ve been steadily committing changes since 0.4.4, but it was a little hard to decide where to cut the next dev release. I also wanted this release to include a meaningful attempt from the MLX kernel side, so it took a bit longer than usual. I hope you’ll understand.

The biggest change in this release is mainly relevant to people using an M3 Ultra, so apologies if this does not apply to your setup yet. - I’m also working on optimizing Gemma in a similar direction, so please stay tuned.

This release focuses on performance improvements for GLM-5.2, which I personally think is a big step forward for local AI, and MiniMax-M3, which has turned out to be a surprisingly useful model in practice.

Previously, these models “worked,” but honestly, I don’t think the long-context speed was where it needed to be for real use. With custom kernels, oMLX now gets a major speedup in long-context prefill. I also ran basic Needle in a Haystack tests and coding tests through Claude Code, and confirmed that quality did not collapse with the optimized path.

I hope this is a meaningful improvement for people using local LLMs in setups similar to mine.

Another major change is API-visible model profiles. You can now expose presets like 'qwen3-8b:thinking' or 'qwen3-8b:non-thinking' and call them directly through the API with the settings you want. Huge thanks to github pablomoralesm for this work: https://github.com/jundot/omlx/pull/1838

As always, this release was only possible because many people contributed their valuable time. I’m deeply grateful.

Thank you as well to everyone using oMLX, sharing feedback, reporting issues, and helping make the product better. It’s great to keep building local AI together!

Got tired of clicking around a menu bar every time I wanted to start my server or switch models, so I built a CLI. Been using this for a while now and figured someone else has probably also been quietly wishing their menu bar icon was just… text in a terminal.

GitHub: https://github.com/omlxMaster/llm-cli
Install: brew install llm-cli (or whatever)

What it does:
- llmctl start / stop / restart — your server, from the terminal where it belongs

- llmctl switch <model> — unloads the current one automatically, loads the new one

- llmctl backend llama.cpp|omlx — switch backends with a flag instead of a dropdown

- Prints the dashboard URL on start instead of opening a browser tab you didn’t ask for

- First-run wizard — llmctl init finds your binary, picks your models folder, sets up launchd for you

- Remembers your last model in a config file and auto-loads it on launch

- Live status via llmctl status — no animated tray icon, just numbers, because elapsed time is a number

- One-model-at-a-time — keeps RAM in check, unloads before loading

Runs on launchd, not as a child process — server keeps running if you quit your terminal, and the CLI picks up a server that’s already running. No Dock icon, no menu bar real estate, no mouse required.

Hey, Built this after hitting some dependency hell with the CUDA/Triton requirements on Colab. Covers SISO, MIMO, and Hybrid Attention-Mamba configs with verified numerical parity against the PyTorch reference (max error < 10⁻⁵, 12 passing tests).

Key things that work: exponential-trapezoidal discretization, complex rotary states, chunked prefill with cache consistency, mixed-precision LoRA fine-tuning, all MLX, no CUDA needed.
If you have any feedback or see any inconsistencies, hmu!

Note: haven't really tested it so far on public trained checkpoints (will do the next days/week) so this is mainly useful for local architecture experimentation and fine-tuning on toy data until the authors drop weights.

→ github.com/Jada42/mlx-mamba3

p.s. this is obviously a joke