u/onil_gova

M5 128 GB 40 Core GPU oMLX 0.4.5.dev1

For local Apple Silicon inference with Qwen oQ4 models, long contexts, and agentic workloads, MTP appears to be a net negative. Disable it unless your own benchmarks prove otherwise.

Notice 27B might have gains for short context, but TG takes a huge hit at long context.

M5 128 GB 40 GPU Cores

For local Apple Silicon inference with Qwen oQ4 models, long contexts, and agentic workloads, MTP appears to be a net negative. Disable it unless your own benchmarks prove otherwise.

Reason 458 why local LLMs are going to be a necessity

edit: For those requesting the source check out their technical report look at page 13