u/A-Rahim — reddlx

▲ 256 r/AIProgrammingHardware+3 crossposts

Qwen3.8-27B is now up to ~3× faster on Apple Silicon with mlx-dspark

mlx-dspark is an MLX port of DeepSeek's DSpark speculative-decoding drafters (the DeepSpec release), plus z-lab's DFlash, with one lossless verify loop. v0.10.0 adds Qwen3.8-27B via RadixArk's drafter, the first SpecForge/SGLang-packaged head it loads.

Numbers (M4 Pro 48 GB, medians of 3, greedy, output ids identical to plain decoding):

8-bit target: 2.45× mean at the auto-picked cap — 3.00× math / 2.38× code / 1.96× chat, 8.3 → 20.3 tok/s (code runs hit 3.18×). Peak ~29 GB.
4-bit target: 1.74× at 25.3 tok/s in ~18 GB (same drafter auto-resolves).
Fun property: 8-bit + drafter (20-27 tok/s) beats plain 4-bit (14.6 tok/s) — 8-bit quality at better-than-4-bit speed.

"Lossless" is checked, not asserted: the target verifies every drafted token, and the Mac app's Race view runs speculative vs plain on the same prompt and diffs the token ids (video is that view).

Everything is pip install mlx-dspark (OpenAI-compatible server + Anthropic Messages API, so it can back Claude Code with a local model), and there's a native Mac app (DMG/Homebrew).

Repo: github.com/ARahim3/mlx-dspark

I'd appreciate any feedback you might have after using it.

u/A-Rahim — 4 days ago

▲ 81 r/LocalLLM+1 crossposts

Meta's Muse Glimmer 30B now runs up to ~3.3x faster on Mac with mlx-dspark

Been tinkering with speculative decoding on Apple Silicon for a while, and this week I got Meta's new Muse Glimmer 30B working in my project mlx-dspark. On my M4 Pro, the 8-bit model goes from 8.2 tok/s to 18-26 tok/s depending on content. Math is the best case at 3.27x, code 2.5x, chat 2.22x. Output is byte-identical to normal decoding since the target verifies every token, so there's no quality tradeoff; it's just faster.

Meta's own DFlash numbers on Mac are 1.5x (M4 Max) / 1.8x (M5 Max), but those are on the 4-bit build, so not really apples-to-apples. 4-bit for me is ~1.7x at ~25 tok/s and only needs ~18GB. The 8-bit run peaks around 40GB, so you want a 48GB Mac for it. Basically, you get 8-bit quality at 4-bit speed.

Repo: github.com/ARahim3/mlx-dspark

I'm happy to hear feedback, and I'm curious about what other M-series chips get.

u/A-Rahim — 8 days ago

▲ 11 r/mlxAI+2 crossposts

mlx-dspark: DeepSeek's DSpark drafter running lossless on a Mac (native MLX, ~1.6×, OpenAI server + benchmarks)

DeepSeek published DSpark, the speculative-decoding drafter they built for DeepSeek-V4 (it's in their DeepSpec repo, with pretrained drafter checkpoints on HF). There was no MLX port, so none of it could run on a Mac. I wrote one.

GIF - left: normal decoding, right: DSpark. Same output, faster.

The part that matters: it's lossless. DSpark is an EAGLE-style drafter, so the main model still verifies every drafted token, and the output is identical to normal decoding (greedy is byte-for-byte up to floating-point ties, and the temperature mode is a verified exact sample from the target). You get the same text, faster. Works today on Qwen3 4B/8B/14B and Gemma-4 12B. New drafters keep showing up on HF in a few different formats; the DeepSpec-style ones run as-is with --drafter, and there's a compatibility table in the README for the rest.

Numbers on my M4 Pro, warm, 8-bit instruct targets, measured against the official mlx_lm/mlx_vlm tools:

https://preview.redd.it/aaoha8dzktbh1.png?width=2240&format=png&auto=webp&s=5eb18b31caff1ade9f2d40ef546b6764d57bcc8a

So roughly 1.4-1.6x single user, up to ~2x on code/math with Gemma. Tbh, I went in expecting the 2-4x that gets quoted for speculative decoding everywhere. You don't get that on a Mac. Turns out the paper never claimed it either, their real figure is ~1.6-1.85x per user in batched serving. The reason is specific to Apple Silicon: verify cost grows with every extra token you check per step (multi-token verify falls off MLX's fast quantized-GEMV path), so even a perfect drafter tops out around 2.2x here, and short draft blocks beat long ones. The full cost model is in the repo if you'd like to review it.

It's not just a benchmark script either. There's an OpenAI-compatible server (LM Studio and the openai SDK works against it) with streaming, tool calls, prefix caching (made follow-up turns on long chats ~13x faster), continuous batching where a finished request returns immediately and its slot picks up the next one, KV-cache quantization for long contexts, and a drafter-free n-gram lookup mode so any model gets some speedup even without a drafter.

I also ported z-lab's original DFlash (the block-diffusion drafter) so both can run under the same lossless loop on the same machine. The winner turned out to be model-dependent, which surprised me: on Gemma-4 12B, where verify is expensive, DFlash's 16-token block wins code/math (~2.1x, accepting ~6 tokens a step). On Qwen3-8B, DSpark just wins everywhere (~1.6x) and DFlash's big block is a wash. I double-checked that against dflash-mlx runner on the identical target and drafter, and it agrees.

Repo: https://github.com/ARahim3/mlx-dspark

Credit to DeepSeek's DeepSpec team and to z-lab for open-sourcing the drafters and the papers. Happy to answer questions, and PRs for more model adapters are very welcome.

reddit.com

u/A-Rahim — 1 month ago