r/oMLX

You may have seen the clip: Qwen3.8-27B at 70 tok/s on a Mac, billed as 4.6× faster, day-one in oMLX.

I ran that on a Mac Studio M3 Ultra (60 GPU cores, 256 GB).

What I actually got, Qwen3.8-27B 4-bit, thinking off:

• No draft (plain AR): about 22 tokens/sec • Official MTP (already running here): about 63 • DFlash 2 (oMLX 0.6.3rc1): about 65

So speculation is real — roughly 3× vs a dumb decode. DFlash 2 vs the MTP that’s already on this Studio: a wash.

Where the launch numbers come from: • 70 tok/s is an M5 Max MacBook Pro demo, not a Studio • 4.6× is Muse Glimmer on an NVIDIA H200, not Qwen 27B on a Mac • Inco’s own Qwen3.8 table is 2.7–3.4× vs AR, and only about 1.2× vs MTP

Thinking-on didn’t change the story (DFlash ~62 vs MTP ~56 on a short prompt).

If you already run official MTP on a Studio, I would not rebuild the stack for DFlash 2. If you’re still decoding autoregressively, turn on MTP — that’s the jump.

Happy to share setup notes in the comments.

I have a m1max with 64GB of RAM. I'm not sure how to tune it effectively and when the auto tuner runs it errors out with

Prefill would require ~61.47 GB peak (current 45.55 GB + KV+SDPA 15.91 GB) but metal_cap ceiling is 54.62 GB. Raise kernel iogpu.wired_limit_mb in Terminal (currently caps Metal at 58.00 GB), or reduce context length.

Hi I run qwen 3.8 oq4e version and accuracy benchmark of MathQA is only 41% and LiveCodeBench is 52%. Not sure if it’s normal? They look low to me… recall 3.6 has slightly higher score on both. Maybe quantisation issue?

Good speed, decent quality for some usecases.

So unfortunately I’ve tried the basic 27B q6 mlx from mlx-community and it only does like 8t/s. This model overthinks a lot so that mixed with 8t/s makes a simple task seem like days.

Any ideas how to speed it up? Or did anyone test q4 vs q5 vs q6?

Hello everyone,

Before getting M5 Max I was very excited for running LLM fully locally.
The machine is great, no doubt. But what bothers me is me thinking that it won't last long / not intended for long LLM inference sessions.

When Energy Mode (Battery, Power) is set to Automatic -> running LLM inference essentially draws 140W. This way the 140W power brick essentially provides just enough power. Machine heats up (no surprise here), and fans spin like crazy / full speed.

Then Low Power Mode is used, the LLM inference is running at about ~1/3 the speed of Automatic. Fans works very quite, and machine is pleasure to use.

Why the post?

My power brick gets overheated in like 2-3hrs and drops charging from 140W to 100W. This makes LLM inference to slowly draw battery (which is bad for battery). Similarly, charging via USB-C is capped at 100W as well..

I wonder if there is way to create an Energy Mode which sits between Low Power and Automatic.

This way to have a bit worse LLM inference speeds than at Automatic, but greater speeds than at Low Power mode.

Has anyone experimented with this? Any success?

This might not be so popular, as MacBook inference is already not that quick, but I'd like my macbook to live a longer live :)
(Using API for inference is not the solution I am considering at this point, if you wonder)

Thanks a lot!

So this just dropped - https://x.com/zhijianliu_/status/2089836737132650504?s=20

Advertises a new DFlash implementation with 70tok/s on MBP M5.

Collection here: https://huggingface.co/collections/incoai/dflash-2

I'll believe it when I see it! Testing now, will update with results.

Edit: Ran a somewhat naive test by simply switching Lightning MTP & ANE off and Dflash on (pointing at incoai/Qwen3.8-27B-DFlash2, kept all default parameters) for my current oQ8e config.

Unfortunately, I'm getting nowhere near the advertised speed:

oMLX - LLM inference, optimized for your Mac
https://github.com/jundot/omlx
Benchmark Model: Qwen3.8-27B-oQ8e-mtp
Engine: Auto
Context: Code (Mixed)
================================================================================

Single Request Results
--------------------------------------------------------------------------------
Test                                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128                          1258.0       56.16   814.0 tok/s    17.9 tok/s       8.403   137.1 tok/s    28.34 GB
pp4096/tg128                          5634.5       55.90   727.0 tok/s    18.0 tok/s      12.747   331.4 tok/s    29.80 GB
pp8192/tg128                         11888.0       57.51   689.1 tok/s    17.5 tok/s      19.208   433.1 tok/s    30.42 GB
pp16384/tg128                        26389.3       58.80   620.9 tok/s    17.1 tok/s      33.882   487.3 tok/s    31.67 GB
pp32768/tg128                        56956.4       61.20   575.3 tok/s    16.5 tok/s      64.766   507.9 tok/s    34.17 GB
pp65536/tg128                       199754.7      111.66   328.1 tok/s     9.0 tok/s     214.038   306.8 tok/s    39.20 GB

Anybody seeing anything different? Code (mixed) might be a somewhat non-standard benchmark choice, but I wouldn't expect performance to tank like this.

Edit 2: Corrected Dflash target from GGUF to non-quantized version.

Edit 3: Clarified oMLX Dflash config.

Good, but thinking budget needs to be set on oMLX, otherwise it buuuurns tokens.

10 issues. Highlights: TurboQuant engine crashes, distributed timeouts, dashboard polling.

🐛 **Bugs**

**Engine & Quantization**
• #2780 **TurboQuantMSEState missing `ndim` crash** — Engine crashes when `turboquant_kv` enabled (Qwen3.5 SDPA split).
• #2778 **Engine crash `ndim` missing** — Same crash during MTP verification with TurboQuant active.

**Distributed & Cluster**
• #2720 **Cluster tab auto-polls `/plan`** — 10s polling loop triggers 400s and disruptive page redirects.
• #2712 **Timeout hard-coded to 300s** — No prefill heartbeats kills long distributed requests.
• #2727 **Coordinator locked to workstation role** — Can't switch to headless; loses reserved 32 GiB.

**Sampling & Profiles**
• #2801 **`presence_penalty` + VLM-MTP = 500** — Applying profile with penalty to VLM-MTP model fails.

**UI & Networking**
• #2800 **Wired-limit rounding loop** — UI rounds threshold to impossible value (124519) on 128 GiB Mac.
• #2520 **Remote MCP connection fails** — MCP server unreachable on v0.5.5+.

✨ **Features & RFEs**

• #2825 **Benchmarks omit model settings** — Published notes lack settings, hurting reproducibility.
• #2711 **`thinking_budget` rejected in distributed** — Clusters get unbounded or no reasoning; bypassed anyway.

Total: 10 issues.

Just fresh from the presses! oMLX 0.6.1 build 2323 with NPU support.

Checkpoint: Qwen3.8-27B-oQ4e-fp16-mtp (scottlowry/Qwen3.6-27B-oQ4e-fp16-mtp)

Lightning MTP - on

Qwen ANE prefill - on

ANE Prompt Block 2048

MLP on ANE 53%

MLP Layer limit - 64

Use both ANE - off

Turbo Quant - off

# Context: Code (Mixed)

# Single request results

Test TTFT(ms) TPOT(ms) ppTPS tgTPS E2E(s) Throughput PeakMem

pp 1024 / tg 512 1202.8 15.1 851.3 66.2 8.9 171.7 22.4 GB

pp 4096 / tg 512 6408.2 14.1 639.2 70.9 13.6 337.8 23.5 GB

pp 8192 / tg 512 14886.6 18.4 550.3 54.4 24.3 358.0 24.1 GB

pp 16384 / tg 512 28808.4 21.4 568.7 46.8 39.8 424.7 25.4 GB

pp 32768 / tg 512 56761.6 22.0 577.3 45.6 68.0 489.2 28.0 GB

pp 65536 / tg 512 131379.6 22.2 498.8 45.0 142.8 462.5 33.2 GB

pp 131072 / tg 512 341823.6 29.7 383.4 33.8 357.0 368.5 43.9 GB

pp 200000 / tg 512 656975.3 40.9 304.4 24.5 678.0 295.8 54.9 GB

# Batch results

Batch tgTPS ppTPS avgTTFT(ms) E2E(s) Speedup

1x baseline 66.2 851.3 1202.8 8.9 1.00x

2x 55.7 611.1 2670.3 21.7 0.84x

Quality testing:

- Seraphim Serapis Tool-Eval-Bench 89/100 - same as 8 bit version

- 0rand/DragonScale Bench - 98/100 - very similar quality as DeepSeek v4 Flash 0731 (4bit/8bit) on 2xDGX Spark Cluster and OpenAI GPT 5.6 Luna

Hi, can someone explain to me how to properly setup the reasoning effort in pi agent to reflect the levels of the model, I believe that omlx supports it now right ?

I put in models.json this:

"thinkingLevelMap": {
            "off": "off",
            "low": "low",
            "medium": "medium",
            "xhigh": "xhigh"     
          }

but when I launch it in pi I still have off-minimal-low-medium-high-xhigh, and I'm not sure if there is a rela change between the levels. Off works btw.

Should I be doing something with kwargs or use a different template to be sure that the thinking level are correct? For example I've read about the froggric one, I'm a bit confused about it

Thanks

https://preview.redd.it/3lfvsocpqzjh1.png?width=889&format=png&auto=webp&s=21268253d9b477b479a135bf8f004bf8f67e7bbc

https://preview.redd.it/u8xz669qqzjh1.png?width=889&format=png&auto=webp&s=ac7f982a40960dc4f130860fed084ff0f2d78891

This is more of an appreciation post than anything else. I've been closely monitoring performance improvements for the new Qwen, and as expected with open source AI, the optimizations are rolling in. In particular, prefill has jumped across nearly all context lengths, and tg has improved for longer contexts.

See plots for a comparison across the past three non-rc/dev releases. Aside from enabling the new Qwen-specific ANE setting in v0.6.1, the relevant configs are identical across runs:

• Specific Quant: ZQ-Dev/Qwen3.8-27B-oQ8e-mtp
• Hardware: M5 Max (40c) 128GB
• Benchmark: Code (mixed)
• Lightning MTP: Enabled
• Prefill Priority: Speed

Thanks to u/cryingneko & co. You guys are clearly cooking.

Edit (18AUG2026) - Updating post with results from the update that dropped today (v0.6.2). Running the exact same benchmark and config, things seem to have regressed a bit:

https://preview.redd.it/bldun5ery6kh1.png?width=3633&format=png&auto=webp&s=a13dd1658011670a6f5577fc1578b081b3236efd

That being said, some of this may be explained by the new ANE-tuning feature, plus the fact that it doesn't currently support oQ8/oQ6 quants, so I didn't use it. This is actively being worked on though, so hopefully Q8 ppl can take advantage in the next build.

For those of you that want a functional 24GB Mac laptop while having this overthinking creature boosting your ideas.

With and without MTP (for the desperate)

I have been running qwen 3.6 35bMOE and qwen 3.8 27b at q4 with a q6 turboquant kv cache, with lighntning MTP Enabled for the past few days. After the latest update, I cannot use lightning MRP with turboquant, or the model will prompt process repeatedly in a loop or output gibberish. The only way to fix was to downgrade to v 0.6.

I don't know wtf the devs were thinking pushing 0.6.1 to "Stable" but holy fuck is it anything but!

Qwen 3.8 27b is known to be overthinking a lot as the default setting is XHigh. How do I set a lower level in harnesses like Pi or OpenCode? Thanks!

Hello im new to local ai and i just want to get the most information possible about it

For context im a software developper that never really used ai before and what to give it a try with my new machine (so basically im trying to achieve agentic result)

I already have omlx and some model installed
But im still trying to understanding all the concept and specification relatated to mlx etc

I understand that quantification shrink the model size trying to keep every essential data to run faster
Still dont know if their is really any advantage to not run every model in their 4bit variant ?

I learned that their is some model training on basic model idk if their are some reputated training like heratic, uncensored, abliterate or some name like opus etc

I see some like quantification method i guess like AXQ optiQ oQ etc idk what is it and the better one

I know the goal of A3B look like the best type of mode to get in every circomstence (maybe im wrong) with the most adequat part of model being active to answer the question)

And idk the best models im currently trying:
- gemma-4-26B-A4B-it-qat-OptiQ-4bit
- Qwen3.6-35B-A3B-OptiQ-4bit
- Qwen3-Coder-30B-A3B-Instruct-MLX-4bit
Maybe i will need some smaller model for lighter task i really dont know

I see some people here talk about fp16 for m1/m2 serie why ?

Any help is appreciate^^
Thanks for reading this