I've been working on getting the 95.5 GiB Qwen3.8-Flash-Next model to run fast on a single 64GB Mac. In my earlier post, I shared a custom expert-streaming fork of llama.cpp . It worked, but decode capped out around ~23–27 tok/s and slowed down as context grew.

Today I'm releasing Slipstream: a compiled C++ Metal inference engine with native SSD expert streaming and speculative drafting for Apple Silicon.

The main result: If you already downloaded my original V3 model (34k+ downloads), you don't need to re-download anything. You can run that exact checkpoint on Slipstream for a 1.76x speedup: 41–52 tok/s (up from 23.1 tok/s in llama.cpp) on the same 64GB Mac.

Even better: decode speed doesn't collapse at long context. Across 3,086 live requests in real coding sessions, it stays flat at 33–44 tok/s all the way out to 130,000 tokens.

Previous posts for context:

• Qwen3.8-Flash-Next in llama.cpp on r/Qwen_AI
• Splash Engine native 8-bit on r/LocalLLaMA

Open source resources:

• Slipstream Engine on GitHub
• Original V3 Model on Hugging Face
• Optional Swift KV-Sparse Model on Hugging Face

1. How to run your existing V3 model on Slipstream

If you have the model from the last post (~/models/qwen38-flash-next-v3), you can point Slipstream directly at it.

Step 1: Clone & build (under 1 minute)

git clone https://github.com/npanj/slipstream.git
cd slipstream
make -j4

Step 2: Download the model (if you don't already have it)

# Downloads the 3 GGUF shards + MTP draft head (~95.5 GiB total)
huggingface-cli download nitinpanj/qwen38-flash-next-v3 \
    --local-dir ~/models/qwen38-flash-next-v3

Step 3: Raise wired GPU memory limit & serve

# Raise wired GPU memory limit once per boot (required on 64 GB Macs):
sudo sysctl iogpu.wired_limit_mb=59392

# Serve your existing model:
./slipstream serve --model ~/models/qwen38-flash-next-v3 --port 8090

>First Run Note: On first launch, Slipstream detects the multi-shard GGUF files and prepares optimized streaming package files into <model>/prepared/ (~5–7 minutes). Subsequent launches load in ~10–15 seconds.

The server exposes a standard OpenAI-compatible API (http://127.0.0.1:8090/v1/chat/completions) ready for curl, Oh My Pi (omp), Claude Code, or OpenCode.

2. Speed: llama.cpp Fork vs. Slipstream (Same V3 Checkpoint)

Here is a direct head-to-head comparison running the exact same 95.5 GiB model files across 6 reasoning and coding tasks on the same M5 Pro (64 GB unified memory, temperature 0.0):

Domain / Task	Prompt Task	llama.cpp Fork	Slipstream	Speedup	llama.cpp TTFT	Slipstream TTFT
Math Reasoning	GSM8K (eggs problem)	24.0 tok/s	43.6 tok/s	1.82x	4,024 ms	2,337 ms
Math Derivation	MATH-500 series ($p - q$)	24.3 tok/s	43.1 tok/s	1.77x	1,655 ms	1,587 ms
Constraint Logic	3-chair deduction	25.4 tok/s	46.0 tok/s	1.81x	1,469 ms	1,042 ms
Python Coding	merge_intervals ($O(N log N)$)	19.7 tok/s	35.0 tok/s	1.77x	1,507 ms	1,070 ms
Systems Coding	Rust CSV parser	22.7 tok/s	37.5 tok/s	1.65x	1,257 ms	859 ms
Tech Writing	Multi-head attention	22.5 tok/s	39.4 tok/s	1.75x	1,267 ms	843 ms
AVERAGE	Across all 6 tasks	23.1 tok/s	40.8 tok/s	1.76x	1,863 ms	1,290 ms

https://preview.redd.it/vh1danbnzwsh1.png?width=3000&format=png&auto=webp&s=aabbd3bcc8611f108d474c1a1b3ae81085c0d9c8

What made Slipstream faster:

1. Asynchronous layer-ahead prefetch (fcntl(F_RDADVISE)): In llama.cpp, synchronous page reads for missed expert matrices stalled the GPU on NVMe latency (~475 ms per chunk). In Slipstream, non-blocking read-ahead hints stream upcoming expert layers from SSD into RAM while the GPU is still executing the previous layer, cutting prefill staging latency by 28%.
2. Hybrid MTP + Prompt Lookup speculation: During tool calls and code generation, Prompt Lookup Decoding (PLD) matches prompt anchors in under 50 ns with 0 allocations, preventing draft rejections. This lifted tool-calling decode from 5.6 tok/s to over 45 tok/s.
3. Metal GPU-mapped n-gram tables: llama.cpp faulted on the 26.8 GiB n-gram table during prefill. Slipstream maps and gathers n-gram embeddings directly in Metal kernels.

3. Context Scaling: Real Telemetry up to 130,000 Tokens

On standard Transformers, decode slows down sharply as context grows because the KV cache swells and memory bandwidth saturates.

Qwen3.8-Flash-Next avoids that through its hybrid architecture:

• 48 recurrent linear DeltaNet layers (fixed $128 \times 128$ hidden state, $O(1)$ memory growth with context).
• Only 16 full-attention layers.

Here is actual telemetry collected across 3,086 live requests during real agent coding sessions on my M5 Pro (64 GB):

Context Range (Tokens)	Live Runs	Average Decode	Median (p50)	Peak Decode	Average TTFT	Notes
< 1,000	314	41.5 tok/s	41.9 tok/s	59.8 tok/s	2.16 s	Short baseline
1k – 4,000	21	41.0 tok/s	42.5 tok/s	64.5 tok/s	5.26 s	Small documents
4k – 8,000	58	43.6 tok/s	43.2 tok/s	67.2 tok/s	7.36 s	Code review turns
8k – 16,000	117	43.6 tok/s	44.6 tok/s	58.2 tok/s	7.91 s	Multi-file context
16k – 32,000	562	38.2 tok/s	40.9 tok/s	58.0 tok/s	13.59 s	Deep agent session
32k – 64,000	1,029	35.0 tok/s	37.5 tok/s	55.6 tok/s	13.24 s	Large repo refactor
64k – 96,000	650	32.4 tok/s	34.7 tok/s	53.9 tok/s	12.81 s	Multi-turn transcript
96k – 130,000	364	32.9 tok/s	33.3 tok/s	43.8 tok/s	7.95 s	Cache-hit deep turns

https://preview.redd.it/qruz5nmpzwsh1.png?width=3300&format=png&auto=webp&s=032c1cb918dc548599e9f15ecd72b66d46828863

Takeaway: Decode speed stays between 33 and 44 tok/s all the way out to 130k tokens. Even at 130k context, it generates tokens faster than stock llama.cpp did on a 500-token prompt.

4. Optional: Swift KV-Sparse Model Variant

If you want higher reasoning accuracy and lower KV cache memory, I also put together an optional Swift variant of this model: Swift-Qwen3.8-Flash-Next-V3.

What Swift changes:

• KV-Sparse Attention: Replaces standard dense attention with KV-sparse layers distilled from Swift-1.5, cutting down RAM pressure at long contexts.
• Spliced Q8 Donor Backbones: Slices 686 high-precision Q8 donor tensors into resident backbone layers for sharper representations.
• Concise Reasoning: Distilled to eliminate repetitive thinking loops in deep contexts.

Both models run on Slipstream using the exact same engine command. Here is how they compare across 145 paired evaluation problems (temperature 0.0, seed 1234):

Domain / Benchmark	Items	Original Flash-Next V3	Swift-Flash-Next V3	Accuracy Delta	Original Decode	Swift Decode
AIME 2025	20	45.0% (9/20)	45.0% (9/20)	0.0%	44.3 tok/s	44.3 tok/s
MATH-500 (L4–5)	35	60.0% (21/35)	62.9% (22/35)	+2.9%	44.8 tok/s	44.8 tok/s
GPQA Diamond	35	45.7% (16/35)	54.3% (19/35)	+8.6%	44.8 tok/s	44.8 tok/s
GSM8K	25	96.0% (24/25)	96.0% (24/25)	0.0%	45.6 tok/s	45.6 tok/s
HumanEval	25	92.0% (23/25)	92.0% (23/25)	0.0%	40.6 tok/s	40.6 tok/s
Hard Systems Logic	5	100.0% (5/5)	100.0% (5/5)	0.0%	39.2 tok/s	39.2 tok/s
OVERALL	145	67.6% (98/145)	70.3% (102/145)	+2.8%	43.9 tok/s	44.4 tok/s

https://preview.redd.it/gt1edfvrzwsh1.png?width=3000&format=png&auto=webp&s=d42ce576fb52c2067a3422bafa594058444cf714

To run the Swift model instead:

huggingface-cli download nitinpanj/Swift-Qwen3.8-Flash-Next-Q4_0-Q8out-v3-GGUF \
    --local-dir ~/models/swift-qwen38-flash-next-v3

./slipstream serve --model ~/models/swift-qwen38-flash-next-v3 --port 8090

5. Foundation for Qwen4

The core primitives in Slipstream:

• 512-route sparse MoE streaming with SSD prefetch
• QSA (Quasi-Sparse Attention) indexer & selection kernels
• Hyper-connection mixing and per-layer embedding gathers
• Metal GPU-mapped n-gram table gathers
• Single-lane speculative verification with PLD & MTP

...were built around this hybrid architecture. If Qwen4 adopts a similar blueprint (hybrid linear recurrence + sparse attention + routed MoE experts), Slipstream should be able to run Qwen4 locally on consumer unified memory hardware on day one.

6. Hardware Tested & Porting to NVIDIA / AMD

• Hardware tested: All testing and benchmarking were done on an Apple MacBook Pro (M5 Pro, 64 GB unified memory, 2 TB SSD).
• CUDA / ROCm ports: I don't have access to modern NVIDIA or AMD GPU hardware, so I can't build or test CUDA/ROCm backends myself.
• If you have hardware and want to help port this: If anyone in the community has NVIDIA or AMD hardware and wants to help bring expert streaming and hybrid speculation to Linux/Windows, I'm happy to help collaborate on the port. Feel free to open an issue on the repo or DM me.

7. Credits & Upstream

• Splash Team (Incoai): Full credit to the creators of Splash. Their C++ Metal speculative decoding design and memory architecture provided the foundation for this work. I will prepare a clean PR proposing these Flash-Next and SSD streaming extensions to the Splash upstream repo in case they want to incorporate them.
• ds4 Team: For their insights on Metal router numerical precision (Taylor polynomial softplus expansion) and streaming scheduling designs.
• Qwen Team: For training Qwen3.8-Flash-Next and releasing the hybrid linear MTP architecture.
• ukisai: For the Swift-1.5 distillation work enabling KV-sparse reasoning.
• bartowski & unsloth: For donor quants and quantization tooling.
• mihailescu2m: For the initial expert streaming work in llama.cpp.

https://preview.redd.it/nulsv53o8vqh1.png?width=4500&format=png&auto=webp&s=74765dbd409f4c221640f9f6000a685f6fdbb242

Spent weekend benchmarking the Splash engine (by Incoai) and extending its architecture to native 8-bit on Apple Silicon (M5 Pro, 64 GB unified memory).

Splash is a compiled C++ and Metal speculative decoding engine designed specifically for Apple Silicon. Upstream Splash pioneered a blisteringly fast speculative decoding pipeline for 4-bit models (~60 tok/s). However, aggressive 4-bit quantization hits a nasty "reasoning cliff" on competition-grade math and multi-step derivations.

We wanted to bring Splash's speed to true uncompressed 8-bit weights without losing its speculative decoding advantages. By extending Splash's architecture to support native 8-bit tiled Metal kernels (schema 5, MDFL0008), we were able to sustain 37–55 tok/s with zero quantization degradation.

• Model weights (27 GB native 8-bit): https://huggingface.co/nitinpanj/Qwen3.8-27B-Splash-HQ
• Q8 Metal runtime fork (Splash extension): https://github.com/npanj/splash/tree/q8
• Prebuilt release (no compiling): https://github.com/npanj/splash/releases/tag/1.0-q8
• Benchmark suite & raw JSON logs: https://github.com/npanj/splash-plus
• Benchmark & Context Scaling Plot: https://raw.githubusercontent.com/npanj/splash-plus/main/benchmark_and_context_scaling.png

Note on compatibility: Official upstream Splash 1.0 (incoai/splash) hardcodes package validation to 4-bit schemas (splash-packed-q4, schema 3/4). This fork adds schema 5 (splash-packed-q8, MDFL0008) loading and compiled Metal Q8 tiled decode kernels, while keeping 100% backwards compatibility with upstream Splash's official Q4 models. Proposed upstream: [incoai/splash#94](https://github.com/incoai/splash/pull/94).

Speeds on Apple Silicon (M5 Pro, 64 GB Unified Memory)

Evaluated at temperature=0.0 across 5 standardized task domains:

Task / Domain	Prompt Description	Splash-Q4 (Official 4b)	Splash-HQ (Native 8b)	Splash-Q8 (Compressed)	MTPLX-Q8 (MTP D3)	Stock MLX / llama.cpp (AR)
Math & Logic	Algebraic derivation	83.3 t/s	54.8 t/s	52.7 t/s	28.5 t/s	9.9 t/s
Coding & Algos	merge_intervals $O(N log N)$	75.5 t/s	34.7 t/s	40.3 t/s	28.8 t/s	9.9 t/s
Constraint Reasoning	3-chair spatial permutation	59.3 t/s	39.3 t/s	37.4 t/s	27.7 t/s	9.9 t/s
Domain Knowledge	FlashAttn vs PagedAttn	39.0 t/s	21.9 t/s	22.8 t/s	23.8 t/s	9.9 t/s
Nuanced Writing	Memory bandwidth constraint	46.2 t/s	33.7 t/s	29.5 t/s	23.4 t/s	9.9 t/s
AVERAGE	Across all 5 domains	60.7 t/s	36.9 t/s	36.5 t/s	26.5 t/s	9.9 t/s
Speedup vs AR	Relative to 9.9 t/s baseline	6.13x	3.73x	3.69x	2.68x	1.00x

A few notes on the comparisons:

• Splash-HQ vs MTPLX (+39% overall, +92% math): Both run on the exact same 8-bit base weights. But Splash’s compiled C++ Metal backend executes with significantly lower dispatch overhead than Python/MLX DraftCore, getting 36.9 vs 26.5 tok/s overall, and hitting 54.8 tok/s on structured math reasoning.
• The Precision-Speed Paradox: Uncompressed native 8-bit (Splash-HQ, 27 GB) actually ran slightly faster on average than compressed 8-bit (Splash-Q8, 17 GB)—36.9 vs 36.5 tok/s. In speculative decoding, decode speed is $\text{Draft Speed} \times \text{Acceptance Rate}$. Aggressive compression flattened logits and lowered draft acceptance; native 8-bit produced sharper logits, fewer verification rollbacks, and higher net throughput despite reading more bytes from memory.

Context Scaling: What Happens Up to 256k Context (Live Telemetry to 190k)

Qwen3.8 is architecturally specified with a native 256k context window (262,144 tokens). Most Transformers fall off a cliff in decode speed as context grows because the KV cache balloons.

However, Qwen3.8 uses a hybrid architecture: 48 recurrent linear DeltaNet layers (fixed $128 \times 128$ hidden state, $O(1)$ memory growth with context) and only 16 full-attention layers.

On a 64 GB Mac, we pushed it live in an active server session all the way out to 190,016 tokens to see if decode speed degraded under real usage:

Context Length (Tokens)	Cached Tokens	Generated Output	TTFT (Prompt Prefill)	Decode Speed	Notes
65	0	50	0.8s	35.7 tok/s	Short prompt baseline
16,433	15,040	232	4.0s	49.0 tok/s	Prefix cache hit
34,605	29,376	2,771	15.6s	27.1 tok/s	Long response generation
83,379	76,320	435	28.9s	30.1 tok/s	Deep context code review
106,212	98,752	29,487	34.0s	24.8 tok/s	Massive batch generation
157,961	157,056	400	6.1s	43.5 tok/s	Cache hit at 158k tokens
180,082	143,360	3,446	228.5s	33.3 tok/s	Extended reasoning session
187,613	186,720	425	6.5s	31.9 tok/s	Cache hit at 187k tokens
188,546	147,456	1,083	268.2s	21.1 tok/s	Partial prefill recompute
190,016	151,552	1,115	227.4s	32.0 tok/s	Max context reached (64GB RAM)

(See the visual plot in the repo: benchmark_and_context_scaling.png showing the full 51-point scatter and rolling trend line).

The big takeaway on context: Decode speed does not collapse. Thanks to Splash's memory handling and the hybrid architecture, it stays between 21 – 33 tok/s across the entire range.

The actual bottleneck at 150k+ context is cold prefill (TTFT). When the prefix cache hits, TTFT at 187k context is just 6.5 seconds. But on a cold cache miss, prefilling 180k+ tokens on a 27B model on Apple Silicon takes ~4–5 minutes. If you are using agent harnesses (like Oh My Pi, Claude Code, or curl), make sure client SSE idle timeouts are set high enough so the client doesn't drop the connection during cold prefills.

The "Reasoning Cliff" on Competition Math

Throughput numbers don't matter if math derivations hallucinate. We tested extended CoT reasoning on MATH-500, AIME 2025, and GPQA Diamond:

• On MATH-500 Problem 0 (evaluating $\sum_{j=1}^(\infty) \sum_{k=1}^(\infty) \frac{1}{(j+k)^(3}) = p - q$), both stock Splash-Q4 and compressed Splash-Q8 fell off a cliff: they suffered numerical drift halfway through the algebraic series manipulation and output wrong values.
• Upgrading to Splash-HQ (full uncompressed 8-bit across all 64 layers) or Splash-Mixed (where only the top 8 sensitive layers, 56–63, are 8-bit) completely eliminated the cliff and cleanly derived $p - q$.
• Upgrading just the deepest 8 layers restored the full symbolic precision while keeping RAM manageable.

https://preview.redd.it/29mgrn0c9vqh1.png?width=2400&format=png&auto=webp&s=47bd228ef9096349e1c563a7d3ce0cc69743c505

Setup Recipe (No Compiling Needed))

Needs: Apple Silicon Mac, macOS 26.4 or later, 48 GB unified memory (64 GB recommended; the weights alone are 27 GB).

The GitHub repo holds the C++ and Metal runtime engine, while the 27 GB model weights are hosted on Hugging Face. You don't need to manually download model files with git-lfs or separate scripts—Splash has a built-in package downloader.

1. Install the prebuilt engine (one line)

 curl -fsSL https://raw.githubusercontent.com/npanj/splash/q8/install-q8.sh | sh

No Xcode, Homebrew or pip needed. It installs a splash-q8 command and doesn't replace an existing Homebrew splash.

2. Launch the server (Automatic Download on First Run)

When you run the command below, Splash automatically detects missing model artifacts, connects to Hugging Face, streams the 27 GB files with progress bars, verifies the manifest SHA-256 hashes, and boots the engine:

 splash-q8 serve --model nitinpanj/Qwen3.8-27B-Splash-HQ

(Once downloaded, subsequent runs load instantly from local disk offline).

(Optional: If you prefer to pre-download the model files beforehand via Hugging Face CLI instead, you can run:)

huggingface-cli download nitinpanj/Qwen3.8-27B-Splash-HQ

3. Connect your client

The server exposes a standard OpenAI-compatible /v1/chat/completions endpoint on http://127.0.0.1:8000:

# Test via curl
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nitinpanj/Qwen3.8-27B-Splash-HQ",
    "messages": [{"role": "user", "content": "Explain why uncompressed 8-bit weights improve speculative decoding acceptance."}],
    "temperature": 0.0
  }'

# Or connect Oh My Pi (OMP)
omp --model splash/nitinpanj/Qwen3.8-27B-Splash-HQ

Building from source instead? You need full Xcode, not just the Command Line Tools. On Xcode 26, first run xcodebuild -downloadComponent MetalToolchain, then make -j4.

Practical Gotchas & Details

1. Memory headroom at 150k+ context: On a 64 GB Mac, model weights take ~27 GB. As context pushes towards 180k–190k, working memory climbs to ~42 GiB. Metal's memory governor will pause allocation growth when system free RAM dips below ~50 MB (Memory: growth paused). If you don't need 190k context, you can pass --max-context 131072 to cap it cleanly.
2. Backwards compatibility: `splash-q8` also serves the official 4-bit models. This fork preserves all upstream Splash 4-bit dense and MoE schemas (splash-packed-q4, splash-packed-q4-moe), so you can serve official models like incoai/Qwen3.8-27B-Splash or incoai/Qwen3.6-35B-A3B-Splash directly.
3. Only tested on Apple Silicon (Unified Memory): Everything here relies on unified memory bandwidth and Metal tiled shaders; not tested on CUDA or CPU.

Credits & Attribution

Full credit to the Incoai team for creating Splash (https://github.com/incoai/splash). Their C++ Metal speculative decoding architecture is what makes these speeds possible on Apple Silicon in the first place—this fork simply extends their work to support native 8-bit weights and custom Q8 tiled kernels. Also huge credit to the Qwen team for base weights and MTP architecture, and Youssofal for MTPLX reference benchmarks.

Updated: setup so that compilation is not needed

Updated (10/1): you can now find follow up work for Qwn3.8-Flash-next here: https://www.reddit.com/r/LocalLLaMA/comments/1wva7l2/running_955_gib_qwen38flashnext_at_4152_toks_on_a/

I have MacBook with 64 gb ram. I am looking for mlx optimized version with mtp. Where can I find it? Also how are people running mlx models locally?

Which coding harness people like most with QWEN3.8. I used codex and Claude code but they were very slow. I tried using pi, but i read that it is not good with planning like Claude. I am wondering which harness are people using that is practically fast and reliable