Dear oMLX developers,

First, thanks for an amazing tool.

Here is an issue that might be worth investigating. I've been playing with oMLX running Qwen 3.8 27B - 4 bit MTP. I've set up the model with 96k context, 16k max_tokens, and 8k reasoning limit. Qwen is on xhigh thinking. I'm using it to generate a large web page that is a fluid dynamics visualizers. On my MacBook Pro M5 Max, the full run with many turns takes about 90 mins. It is well behaved throughout - no memory thrashing (took some tuning with the GDN settings).

In analyzing the log, it seems that on many long generation runs, MTP is quitting early - after several hundred tokens - on a many thousand token generation run. Claude suggested that performance might be better if MTP re-armed more often. Here is a table that lists context length (prompt length) (growing over about an hour or so), and then generation length for a turn, and MTP status and success for that turn. Stop means that MTP got through the whole run, and parked means that it stopped after x tokens in the run.

Full post-GDN table, every completion I can pair with an MTP outcome:

Prompt	Gen	tok/s	MTP	Accept
69	10,207	29.0	—	—
1,182	252	13.7	stop	75.8%
8,729	8,599	23.9	parked at 640	71.5%
17,772	61	28.6	stop	70.5%
17,915	4,964	33.7	stop	93.8%
22,900	1,979	29.9	stop	87.3%
24,899	363	27.2	stop	82.2%
41,338	1,085	29.7	—	—
42,443	141	23.6	stop	83.9%
44,123	1,100	28.8	stop	94.2%
45,243	289	25.9	stop	93.1%
45,552	102	23.3	stop	83.8%
46,646	314	17.0	parked at 134	74.3%
46,980	570	22.2	stop	80.3%
47,998	120	21.3	stop	78.6%
48,201	7,559	15.9	parked at 101	79.3%
55,780	132	18.2	parked at 79	81.6%
56,230	2,243	24.0	stop	92.9%
58,493	4,763	15.5	parked at 499	85.6%
63,276	5,865	15.2	parked at 499	90.9%
69,161	3,296	14.9	parked at 155	81.6%
72,477	280	19.0	stop	85.5%
72,777	713	21.0	stop	87.3%
73,510	79	22.3	stop	86.8%
73,618	188	23.4	stop	92.3%
73,835	7,814	14.5	parked at 145	74.1%

thanks for considering!

I've got a Mac Studio m3 256GB ram using Omlx 0.6.2, Hermes 0.20 using Qwen3.8-27b 8bit . I have it doing some CRM (Odoo) work for me. 3.6 was great and I was / am very excited for 3.8 , it does a better job overall just takes a while to get there, sometimes much longer.

This is the first task it's just not getting there and I can see in WebUI it's just overthinking and going around and around never really getting anywhere.

I turned thinking / Reasoning effort down to Minimal. Other than that not much has changed from 3.6 Anyone have any ideas or suggestions to get the intelligence without as much over thinking?

I need recommendations on the model settings to get optimal performance in oMLX. I am using the latest 0.6.2 version. I have a Macbook Pro with M1 Max and 64GB ram.

• What context window should I use? Should I use 128k or 256k?
• Which one will deliver better performance, TurboQuant or Lightning MTP?
• If I use Lightning MTP, what would be the recommended context window?

I am using omlx and local models particularly for agentic coding via VS Code and OpenCode.

Ornith seems to be better.

TL;DR: On an M3 Ultra, Ornith-1.5-35B-A3B (4-bit MLX) decodes 4.6× faster than Qwen3.8-27B (8-bit MLX) and scores slightly higher on a small hard eval. It also beats Qwen3.8-27B with speculative decoding, while running autoregressive.

Setup

• Mac Studio, M3 Ultra, 256 GB unified memory
• mlx-lm 0.31.3 / mlx 0.32.1
• Ornith-1.5-35B-A3B, MLX 4-bit
  • 18.2 GiB download
  • 20.2 GB peak
• Qwen3.8-27B, MLX 8-bit
  • 27.5 GiB download
  • 29.7 GB peak
• Machine was shared and had other load. Numbers are a floor, not a best case.

Throughput

mlx_lm.benchmark -p 512 -g 512 -n 3, identical invocation for both:

Model	Decode tok/s	Prefill tok/s	Peak mem
Ornith-1.5-35B-A3B 4-bit	107.9	2162	20.2 GB
Qwen3.8-27B 8-bit	23.4	408	29.7 GB

Trial spread was 1.5% and 0.35%, respectively.

Prefill is the bigger story: 5.3×.

A 20K-token prompt took Ornith ~25s end to end versus ~125s for Qwen3.8-27B. If your workload re-reads long contexts, that dominates.

It also beats Qwen with speculative decoding

I spent a day trying to make Qwen3.8-27B fast before testing Ornith:

Qwen3.8-27B 8-bit config	Decode tok/s
Autoregressive	23.4
MTPLX, native MTP head, depth 2	65.7 (3.01×)
DFlash2 block-diffusion drafter, block 5	79.3 (3.37×)
Ornith, plain autoregressive	107.9

Both speculative stacks work and are genuinely impressive. DFlash2's 3.37× on Apple Silicon is close to its published 3.43× on an H200.

Ornith just beats them without needing either, with no drafter and no third-party runtime, because mlx-lm already ships qwen3_5_moe.py.

Quality: 12 hard cases, thinking enabled

Scoring is mechanical. Code tasks are executed against hidden assertions and pass only on a full suite.

Task	Ornith-35B-A3B	Qwen3.8-27B
code_exec (4, execution-scored)	4/4	4/4
multihop (3, two facts ~20K apart)	3/3	3/3
logic (3)	2/3	2/3
tool_schema (2, nested JSON)	2/2	1/2
Total	11/12	10/12
Wall time for the set	166s	498s

One logic item was ambiguous. Two vals gave the same "wrong" one, so discount it: 11/11 vs 10/11.

Qwen's other miss was invalid JSON on a nested tool call. For agent use, that is the failure mode that actually breaks loops.

Caveats, and they are not small

• Not precision-matched. 4-bit vs 8-bit. Some of the gap is quantisation; the rest is likely 3B active vs 27B dense. I have not run the 4-bit Qwen control.
• n=12. An 11 vs 10 spread is one item.
• Vendor benchmarks disagree with me. On SWE-bench Pro, the only benchmark both publish, Qwen3.8-27B is ahead: 61.7 vs 59.6.
• Thinking must be on. With enable_thinking: false, Ornith went 0/5 on arithmetic and recovered to 4/4 with it on. My first eval drew a conclusion that was purely an artifact of my own test design.
• 122B comparison still running.

The bit that surprised me

MoE is not a handicap here. It is the reason this works.

With ~3B active parameters per token, memory tracks total parameters while speed tracks active parameters.

Ornith gets:

• 4.6× the decode throughput
• 5.3× the prefill throughput
• 32% less peak memory

Also, Ornith-1.5 is architecturally Qwen's exact vocab size, i.e. a self-improvement-trained fork of Qwen's older MoE architecture.

Beating Qwen's newer dense model with it is a nice result for the training approach.

MIT licence, and it is multimodal.

I run small models awen3.5-9b and gemma-4-12b for some tasks. Recently tried GGUF format and looked pretty much okay. Getting 15-20 tok/sec and 100-150 on prefil with MTP.

Decide to try again oMLx. I cannot get better performance than ggufs. Maybe I am missing small models MTP versions. Or faster models. Or my settings bad?
So question: what models you guys using on similar hardware how much you are getting ?

Use cases: parsing, email drafting and general assistant.

My Device: mac m2 24gb ram

Hi everyone,

I'm trying to run the Muse-Glimmer model locally on Apple Silicon using the MLX framework for an agentic workflow. However, I’ve hit a massive roadblock: persistent KV caching is not working.

As a result, every single new follow-up request from the agent triggers a full prompt re-evaluation. On longer context windows, this makes the performance unusable.

Here is what I have already tried and the results:

1. LM Studio: No luck. I couldn't find a compatible model format/architecture that would even boot up there.
2. ExecuTorch (Meta's official method): It runs, but KV caching is missing. Every single step forces a full context recalculation.
3. mlx-vlm (standard run): The model starts up fine, but context caching does not work.
4. llama.cpp: The only framework where everything works perfectly and the KV cache is properly retained. However, there is a massive downside — the prompt processing speed is painfully slow compared to native MLX performance.
5. mtplx: Didn't even bother trying, as I previously failed to get KV caching working on this stack even for standard Qwen models.

My question to the community:
Has anyone successfully managed to get KV caching working for Muse-Glimmer on pure MLX ormlx-vlm?

Any code snippets, fork links, or ideas would be greatly appreciated!

You may have seen the clip: Qwen3.8-27B at 70 tok/s on a Mac, billed as 4.6× faster, day-one in oMLX.

I ran that on a Mac Studio M3 Ultra (60 GPU cores, 256 GB).

What I actually got, Qwen3.8-27B 4-bit, thinking off:

• No draft (plain AR): about 22 tokens/sec • Official MTP (already running here): about 63 • DFlash 2 (oMLX 0.6.3rc1): about 65

So speculation is real — roughly 3× vs a dumb decode. DFlash 2 vs the MTP that’s already on this Studio: a wash.

Where the launch numbers come from: • 70 tok/s is an M5 Max MacBook Pro demo, not a Studio • 4.6× is Muse Glimmer on an NVIDIA H200, not Qwen 27B on a Mac • Inco’s own Qwen3.8 table is 2.7–3.4× vs AR, and only about 1.2× vs MTP

Thinking-on didn’t change the story (DFlash ~62 vs MTP ~56 on a short prompt).

If you already run official MTP on a Studio, I would not rebuild the stack for DFlash 2. If you’re still decoding autoregressively, turn on MTP — that’s the jump.

Happy to share setup notes in the comments.

I have a m1max with 64GB of RAM. I'm not sure how to tune it effectively and when the auto tuner runs it errors out with

Prefill would require ~61.47 GB peak (current 45.55 GB + KV+SDPA 15.91 GB) but metal_cap ceiling is 54.62 GB. Raise kernel iogpu.wired_limit_mb in Terminal (currently caps Metal at 58.00 GB), or reduce context length.

Good speed, decent quality for some usecases.

Hi I run qwen 3.8 oq4e version and accuracy benchmark of MathQA is only 41% and LiveCodeBench is 52%. Not sure if it’s normal? They look low to me… recall 3.6 has slightly higher score on both. Maybe quantisation issue?

Hello everyone,

Before getting M5 Max I was very excited for running LLM fully locally.
The machine is great, no doubt. But what bothers me is me thinking that it won't last long / not intended for long LLM inference sessions.

When Energy Mode (Battery, Power) is set to Automatic -> running LLM inference essentially draws 140W. This way the 140W power brick essentially provides just enough power. Machine heats up (no surprise here), and fans spin like crazy / full speed.

Then Low Power Mode is used, the LLM inference is running at about ~1/3 the speed of Automatic. Fans works very quite, and machine is pleasure to use.

Why the post?

My power brick gets overheated in like 2-3hrs and drops charging from 140W to 100W. This makes LLM inference to slowly draw battery (which is bad for battery). Similarly, charging via USB-C is capped at 100W as well..

I wonder if there is way to create an Energy Mode which sits between Low Power and Automatic.

This way to have a bit worse LLM inference speeds than at Automatic, but greater speeds than at Low Power mode.

Has anyone experimented with this? Any success?

This might not be so popular, as MacBook inference is already not that quick, but I'd like my macbook to live a longer live :)
(Using API for inference is not the solution I am considering at this point, if you wonder)

Thanks a lot!

So unfortunately I’ve tried the basic 27B q6 mlx from mlx-community and it only does like 8t/s. This model overthinks a lot so that mixed with 8t/s makes a simple task seem like days.

Any ideas how to speed it up? Or did anyone test q4 vs q5 vs q6?

Good, but thinking budget needs to be set on oMLX, otherwise it buuuurns tokens.

So this just dropped - https://x.com/zhijianliu_/status/2089836737132650504?s=20

Advertises a new DFlash implementation with 70tok/s on MBP M5.

Collection here: https://huggingface.co/collections/incoai/dflash-2

I'll believe it when I see it! Testing now, will update with results.

Edit: Ran a somewhat naive test by simply switching Lightning MTP & ANE off and Dflash on (pointing at incoai/Qwen3.8-27B-DFlash2, kept all default parameters) for my current oQ8e config.

Unfortunately, I'm getting nowhere near the advertised speed:

oMLX - LLM inference, optimized for your Mac
https://github.com/jundot/omlx
Benchmark Model: Qwen3.8-27B-oQ8e-mtp
Engine: Auto
Context: Code (Mixed)
================================================================================

Single Request Results
--------------------------------------------------------------------------------
Test                                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128                          1258.0       56.16   814.0 tok/s    17.9 tok/s       8.403   137.1 tok/s    28.34 GB
pp4096/tg128                          5634.5       55.90   727.0 tok/s    18.0 tok/s      12.747   331.4 tok/s    29.80 GB
pp8192/tg128                         11888.0       57.51   689.1 tok/s    17.5 tok/s      19.208   433.1 tok/s    30.42 GB
pp16384/tg128                        26389.3       58.80   620.9 tok/s    17.1 tok/s      33.882   487.3 tok/s    31.67 GB
pp32768/tg128                        56956.4       61.20   575.3 tok/s    16.5 tok/s      64.766   507.9 tok/s    34.17 GB
pp65536/tg128                       199754.7      111.66   328.1 tok/s     9.0 tok/s     214.038   306.8 tok/s    39.20 GB

Anybody seeing anything different? Code (mixed) might be a somewhat non-standard benchmark choice, but I wouldn't expect performance to tank like this.

Edit 2: Corrected Dflash target from GGUF to non-quantized version.

Edit 3: Clarified oMLX Dflash config.

Just fresh from the presses! oMLX 0.6.1 build 2323 with NPU support.

Checkpoint: Qwen3.8-27B-oQ4e-fp16-mtp (scottlowry/Qwen3.6-27B-oQ4e-fp16-mtp)

Lightning MTP - on

Qwen ANE prefill - on

ANE Prompt Block 2048

MLP on ANE 53%

MLP Layer limit - 64

Use both ANE - off

Turbo Quant - off

# Context: Code (Mixed)

# Single request results

Test TTFT(ms) TPOT(ms) ppTPS tgTPS E2E(s) Throughput PeakMem

pp 1024 / tg 512 1202.8 15.1 851.3 66.2 8.9 171.7 22.4 GB

pp 4096 / tg 512 6408.2 14.1 639.2 70.9 13.6 337.8 23.5 GB

pp 8192 / tg 512 14886.6 18.4 550.3 54.4 24.3 358.0 24.1 GB

pp 16384 / tg 512 28808.4 21.4 568.7 46.8 39.8 424.7 25.4 GB

pp 32768 / tg 512 56761.6 22.0 577.3 45.6 68.0 489.2 28.0 GB

pp 65536 / tg 512 131379.6 22.2 498.8 45.0 142.8 462.5 33.2 GB

pp 131072 / tg 512 341823.6 29.7 383.4 33.8 357.0 368.5 43.9 GB

pp 200000 / tg 512 656975.3 40.9 304.4 24.5 678.0 295.8 54.9 GB

# Batch results

Batch tgTPS ppTPS avgTTFT(ms) E2E(s) Speedup

1x baseline 66.2 851.3 1202.8 8.9 1.00x

2x 55.7 611.1 2670.3 21.7 0.84x

Quality testing:

- Seraphim Serapis Tool-Eval-Bench 89/100 - same as 8 bit version

- 0rand/DragonScale Bench - 98/100 - very similar quality as DeepSeek v4 Flash 0731 (4bit/8bit) on 2xDGX Spark Cluster and OpenAI GPT 5.6 Luna

10 issues. Highlights: TurboQuant engine crashes, distributed timeouts, dashboard polling.

🐛 **Bugs**

**Engine & Quantization**
• #2780 **TurboQuantMSEState missing `ndim` crash** — Engine crashes when `turboquant_kv` enabled (Qwen3.5 SDPA split).
• #2778 **Engine crash `ndim` missing** — Same crash during MTP verification with TurboQuant active.

**Distributed & Cluster**
• #2720 **Cluster tab auto-polls `/plan`** — 10s polling loop triggers 400s and disruptive page redirects.
• #2712 **Timeout hard-coded to 300s** — No prefill heartbeats kills long distributed requests.
• #2727 **Coordinator locked to workstation role** — Can't switch to headless; loses reserved 32 GiB.

**Sampling & Profiles**
• #2801 **`presence_penalty` + VLM-MTP = 500** — Applying profile with penalty to VLM-MTP model fails.

**UI & Networking**
• #2800 **Wired-limit rounding loop** — UI rounds threshold to impossible value (124519) on 128 GiB Mac.
• #2520 **Remote MCP connection fails** — MCP server unreachable on v0.5.5+.

✨ **Features & RFEs**

• #2825 **Benchmarks omit model settings** — Published notes lack settings, hurting reproducibility.
• #2711 **`thinking_budget` rejected in distributed** — Clusters get unbounded or no reasoning; bypassed anyway.

Total: 10 issues.