TL;DR

• 3× modded RTX 2080 Ti 22 GB (66 GB VRAM total) runs Qwen3.8-Flash-Next 125B MoE (~6B active) at 66.6 tok/s decode / 742 tok/s prefill with Strata’s layer-split — and 932 tok/s prefill on a fresh 83K-token prompt.
• First same-harness perplexity comparison of the two competing quant families on this model (neither vendor publishes this head-to-head): UD-Q4_K_XL 4.1263 > IQ3_S 4.1967 > IQ3_XXS 4.2173, paired t-test over 72 chunks: Q4’s lead is significant (t=4.45 / 5.33); S vs XXS is not (t=1.05).
• Strata layer-split beats NVLink peer mode by +83% and beats a standard llama.cpp partial-offload layout by 2.5× decode / 4.4× prefill on identical hardware.
• 512K context is nearly free: 59.9 vs 66.6 tok/s at 262K (-10%, within noise of these runs), RSS +4 GB.

HARDWARE

Models (all GGUF, from HF):

Strata splits the 48 layers into 3 stages (layer-split), keeps hot experts GPU-resident in an adaptive per-card cache (97–100% of routed mass), streams the rest from a RAM arena, PLE/n-gram table reads direct from NVMe, KV = int8 with a 32768-cell VRAM window.

SPEED (3-RUN AVERAGES, ~3K-TOKEN PROMPTS, KV INT8, CTX 262144 UNLESS NOTED)

Decode tok/s

Q4_K_XL   layer-split [0,1,2]      ████████████████████████████████████████66.6
IQ3_XXS   layer-split [0,1]        ███████████████████████████████████████▊63.2
IQ3_XXS   layer-split [0,1,2]      ███████████████████████████████████████ 61.3
Q4_K_XL   layer-split [0,1,2] 512K ████████████████████████████████████▍   59.9
IQ3_S     layer-split [0,1,2]      ███████████████████████████████████▍    58.3
Q4_K_XL   layer-split [0,1]        ███████████████████████████████████▍    57.6
IQ3_S     layer-split [0,1]        █████████████████████████████████▍      56.1
IQ3_XXS   1 GPU                    ████████████████████████████            50.5
IQ3_S     1 GPU                    ███████████████████████████             48.6
Q4_K_XL   NVLink peer 512K         █████████████████████                   36.6
llama.cpp partial-offload 512K     █████████████████                       27.0
Q4_K_XL   1 GPU                    ███████████████                         25.7

Prefill tok/s (short prompts)

IQ3_XXS   layer-split [0,1,2]      ████████████████████████████████████████ 742
IQ3_S     layer-split [0,1,2]      ███████████████████████████████████████▍ 726
IQ3_XXS   layer-split [0,1]        ███████████████████████████████████████▊ 700
IQ3_S     layer-split [0,1]        ████████████████████████████████████▍    671
IQ3_XXS   1 GPU                    ████████████████████████████             525
Q4_K_XL   layer-split [0,1,2]      ███████████████████████▍                 448
IQ3_S     1 GPU                    ███████████████████████▎                 426
Q4_K_XL   layer-split [0,1,2] 512K ███████████████████████                  424
Q4_K_XL   layer-split [0,1]        ████████████████████                     377
Q4_K_XL   1 GPU / peer             ████████████                            ~232
llama.cpp partial-offload          ██████                                   102

One-shot prefill on a fresh 83K-token prompt (this is the fun one)

Q4_K_XL layer-split [0,1,2] 512K  ████████████████████████████████████████ 932.5 tok/s
Q4_K_XL NVLink peer         512K  █████████████████████▌                   490.2 tok/s

(83,306 fresh tokens, 0 cache reuse — the 3-stage split parallelizes the prompt path; decode at 83K context: 58.9 tok/s, expert-cache hit 97.4%, KV VRAM-hit 92.8%.)

Expert-cache hit rate (avg) + per-model footprint @ layer-split [0,1,2]

Ablations (on Q4 split3 — what NOT to do)

Observations

1. Layer-split >> NVLink peer mode: same model, 2 cards: peer 36.4 → split 57.6 tok/s (+58%); 3rd card adds +16% even though it’s the x8 slot (split hand-off goes through pinned host memory — NVLink direct is still an open Strata issue #338).
2. The 1-GPU ranking inverts on 3 GPUs: 1 card → IQ3_XXS 50.5 > IQ3_S 48.6 > Q4 25.7 (smaller blobs = higher cache hit). 3 cards → Q4 66.6 > XXS 61.3 > S 58.3 with all caches near-full — pointing at per-format GPU kernel paths (Strata ships dedicated Q4_K/Q5_K/Q5_1/Q8_0 expert kernels since 0.1.31) plus draft-acceptance variance, not at cache misses.
3. 512K context costs ~5-10% vs 262K (rope linear ×2): prefill 424 vs 448 (-5.5%), decode 59.9 vs 66.6 (round-spread is ±6 tok/s on both — read as noise to ~5%); draft acceptance dipped 73%→63% at 512K (possible mild effect of the scaled rope on drafting, or just run variance).

ACCURACY

Perplexity — wikitext-2-raw test, measured here

Method: llama-perplexity (build 349), test split = 4,358 rows / ~294K tokens, 72 chunks × ctx 4096, q8_0 KV, flash-attn on, byte-identical GPU placement flags for all three models (same -ot expert pins as the speed tests). Lower = better.

excess mean-log-PPL vs best (paired over the same 72 chunks, lower = better):

UD-Q4_K_XL  4.1263 █                                  (best)
GSQ IQ3_S   4.1967 ███████   +0.0704  (+1.71%)
GSQ IQ3_XXS 4.2173 █████████ +0.0910  (+2.21%)

• A duplicate full Q4 run produced bit-identical results (deterministic harness), so differences above ~0.01 mean-log are real, not noise.
• The paired test (same chunks for every model) is far more sensitive than the tool’s unpaired ±0.024.

Published task numbers (ISTA’s evaluation — not re-run here)

unsloth’s own metrics for the UD ladder (token-level, different scale — do not cross-read with task scores): Q4_K_XL top-1 recovery 92.26%, KLD 0.0469 · Q5 93.68/0.0304 · Q6 94.09/0.0271.

Reading of it: on distribution fit (ppl) the 4-bit UD wins measurably; on published tasks the IQ3_S claims base parity. Both can be true — ppl and task accuracy measure different things, and we have no BF16 ppl baseline here (354 GB). For “which do I run”: Q4 = best decode + best ppl; IQ3_S = smallest file / fastest load / published task parity; IQ3_XXS = best prefill, cheapest bits.

LLAMA.CPP COMPARISON (SAME BOX, SAME MODEL FILE)

The 103.7 GiB model can’t fit 66 GB VRAM + 512K KV + 28.8 GB PLE, so llama.cpp runs the usual partial-offload layout:

--n-cpu-moe 48                       # 48 expert layers
--split-mode layer --tensor-split 1,1,1
-ot 'per_layer_token_embd=CPU,       # PLE/n-gram -> CPU
     blk.(8|9|1[0-4])...=CUDA0,      # 7 expert blocks -> GPU0
     blk.(2[0-7])...=CUDA1,          # 8 -> GPU1
     blk.(3[8-9]|4[0-3])...=CUDA2'   # 6 -> GPU2
     # = 21 blocks on GPU, 27 expert layers on 16 CPU threads
--ctx-size 512000 --cache-type-k q8_0 --cache-type-v q8_0
--flash-attn on --threads 16 + MTP draft sidecar

Measured: 102 prefill / 27.0 tok/s decode (draft acceptance ~79.6%).

This is not “llama.cpp is slow” — it’s a different trade: Strata’s layer-split runs the same stage decomposition, but each stage computes its hot experts from GPU memory (adaptive cache, 97–100% routed mass) instead of gathering 27 layers’ experts from the CPU pool every token. Same split, different expert story → 4.4× prefill, 2.5× decode. Caveat: our build doesn’t have qwen4exp’s re-enabled -sm tensor path (upstream #28569), so tensor-split wasn’t an option here.

CAVEATS

1. Speed prompts are synthetic (~3K tokens) + one 12K deep probe, 3 runs each; some configs show ±5–10% round spread (draft-acceptance variance) — treat sub-5% differences as noise.
2. Absolute ppl depends on the chunking mode; the valid claim is the cross-model ordering under identical settings.
3. Task tables are ISTA’s published numbers, not our measurements.
4. GPUs are power-capped at 167 W (efficiency-tuned); a 250 W run would likely shift sustained clocks a bit.
5. PPL ≠ task accuracy — both numbers are above so you can weight them yourself.

Configs, per-run JSONs and ppl logs are on hand — happy to post any specific one.

In the past few days, I have gained the following experiences:

Although the performance of the Qwen 3.8 27B model is not bad, but...

Let's start at the very beginning. In terms of raw computing performance, coding based on my local measurements and in practice, it seems that the Qwen3.6-27B-Q4_K++ models are almost on the same level as the Qwen3.8 27B model. The big difference is that the 3.8 thinks a lot. This is where its advantage comes into play. The tool calling is better, so it is more efficient overall. However, it is not necessarily smarter.

If I set the thinking value to med or min for the Qwen 3.8 27B instead of xhigh, it consistently underperforms in the tests compared to the Qwen 3.6 27B.

So Qwen 3.8 27B is only smarter if the reasoning level is xhigh, but then it is incredibly slow and generates a lot of tokens. And the worst thing is that even though the KV cache is Q8 quantized, it still requires a brutal amount of RAM/VRAM compared to Qwen 3.6.

And here comes the point that needs to be discussed.

Qwen 3.6 27B runs with 200k+ ctx on my 22 GB GPU in good quality.

Qwen 3.8 27B does not. This is not enough for it, there is a maximum of 100k ctx in the VRAM and the model is only Q4 quality, the KV cache Q8, because below it the model will be brutally stupid, a lobotomy. Obviously the best would be F16.

So I think that for planning and ideation, thinking, Qwen 3.8 27B is suitable, but for implementation 3.6 27B is perfect and MUCH faster. But...

This is where Qwen 3.8 Flash Next comes into play.

While on the 2080 Ti card (22GB Modded) on the i9-7980XE CPU with quad channel DDR4 (128GB) RAM:

the Qwen 3.8 27B model runs at Q4 quantum quality, ctx 100k, KV Q8 settings at approx. 35-45 t/s with MTP,

the Qwen 3.8 Flash Next (Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS) with STRATA, ngram can easily fit on 1 GPU with 256k ctx (KV Q8) and 64 GB RAM is enough for it. The speed is currently 60-75 t/s and the strata is developing at full speed.

A fast SSD is probably also important for this, but I'm not sure, because I think the ngram table is in the RAM now. By the way, I only have Gen3 PCI-e ports, so I solved the SSD acceleration by connecting 2 average SSDs to nvme ports and creating an array with RAID0, and then their speed added up.

Experiences about the Qwen 3.8 FN:

It doesn't overthink things. It can think a lot, but not always, only when justified. It's precise, follows instructions, smart. I'll risk saying that based on my usage habits, it's smarter than the GLM 5.3 Flash.

Since its speed is also better and the VRAM requirement is less than the Qwen 3.8 27B, the question now is which model should I choose? I think the answer is clear.

Do you have similar experiences?

I spent the day benchmarking Qwen3.6-27B-MTP-APEX on a modest 2× RTX 3060 12GB + i7-6700 system. Started at 16.7 t/s with a basic layer-split config and ended up at 56 t/s at 96k context — a 3.3× speedup from where I started, and the context window went from 32k to 96k in the process.

This post walks through every measurement, every wrong turn, and the final config that actually works. The single command at the bottom replaces my previous setup.

TL;DR

* Final production config: Qwen3.6-27B Q4\_K\_S + tensor split + MTP (`n_max=2, p_min=0.44`) → 56 t/s at 96k ctx, F16 KV cache, 3.3× faster than the basic layer-split setup * The killer flag: `--split-mode tensor --tensor-split 1,1` — without it, I was stuck at 17 t/s * The hidden trap: Short benchmark prompts inflate `mean_draft_len` and make high `n_max` look amazing. Real workloads cap out at \~3-4.

Hardware

CPU:  Intel Core i7-6700 @ 3.40 GHz (Skylake, 4C/8T, 14nm)
Mobo: ASUS Z170 PRO GAMING (Intel Z170, LGA 1151)
RAM:  64GB DDR4 — DIMM\_A1: Samsung 32GB @ 2720 MT/s
                  DIMM\_B1: Team Group 32GB @ 2720 MT/s
                  (mismatched sticks, both XMP'd to 2720 from 2400 stock)
GPU:  2× NVIDIA GeForce RTX 3060 Lite Hash Rate (GA106), 12GB VRAM each
      (Intel HD 530 used for monitor output)
PSU:  (whatever I had lying around — total system power \~350W under load)
OS:   Linux Mint 22.3 "Zena" (Ubuntu 24.04 base, Linux 6.x kernel,
     Cinnamon desktop, X11 session)
CUDA: 12.0 (nvidia-driver-535)
llama.cpp: self-built from source, version 10218 (commit de699957b)
         built with -DGGML\_CUDA=ON, NCCL disabled

This is a $400 dual-3060 setup I picked up second-hand. Not impressive on paper. Let's see what it can do.

The GPUs run in `tensor-split 1.05,0.95` mode because the first card has a slightly better silicon lottery result and can take 5% more of the tensor-parallel split. No NVLink — just PCIe 3.0 x8/x8.

The journey: 9 measurements, 9 surprises

Every measurement below used the same prompt set: short Shakespeare quote (`. * 1300` ≈ 16k chars ≈ 4000-4500 tokens), 256-token generation, warm cache unless noted. All on Qwen3.6-27B-MTP-GGUF.

*Stage 1: Q4\_K\_XL, layer split (basic config)*

Config: --split-mode layer --tensor-split 1.05,0.95 --cache-type-k q8\_0 --cache-type-v q8\_0 Result: 16.73 t/s

The default setup. Predictable. The Q4\_K\_XL model is 17 GB on disk, which combined with q8\_0 KV at 32k ctx uses \~24 GB VRAM — basically all of it.

*Stage 2: Try MTP (no tensor split)*

Config: + --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.44
Result: 26.96 t/s (+61%)

MTP helps on this dense model? Apparently. The 81.9% draft acceptance rate pays off because dense forward passes are cheap enough that the draft overhead doesn't exceed the savings. Interesting.

*Stage 3: Tensor split (the breakthrough)*

Config: --split-mode tensor --tensor-split 1,1 (no KV quant — F16 KV)
Result: 28.46 t/s (+70% over baseline)

Tensor split distributes every matmul across both GPUs simultaneously instead of running layers sequentially. Combined with F16 KV cache, this is 70% faster than layer split.

But wait — tensor split means I can't use quantized KV cache. Trade-off accepted because tensor split was worth more than the KV quant savings.

*Stage 4: Tensor split + MTP (the holy-cow moment)*

Config: tensor split + draft-mtp n\_max=2 + F16 KV Result: 39.13 t/s on 7349-token prompt, 55.97 t/s on 2341-token prompt draft acceptance 85%, mean draft len 3.00

This was the moment. 2.33× over baseline. MTP + tensor split, who would have thought.

*Stage 5: Switch to Q4\_K\_S, push context to 96k*

The Q4\_K\_XL at 32k ctx was using 17 GB model + 7 GB F16 KV ≈ 24 GB VRAM. OOM at 64k. But Q4\_K\_S is 16 GB instead of 17 — fits at 96k!

Config: Q4\_K\_S + tensor\_mtp at 96k ctx
Result: 56.4 t/s, draft acceptance 100%, mean draft len 3.00

Identical throughput to 32k, but 3× the context. The lower quant also makes the model more confident — acceptance goes from 85% to 100%.

*Stage 6: n\_max sweep (cold cache, short prompts)*

Hypothesis: if `mean_draft_len = 3.00` at `n_max = 2`, then `n_max = 4` or `5` or `6` should give `mean_draft_len = 5`, `6`, `7` — even faster.

Whoa! `n_max = 7` looks like 512 effective tok/s (62× the predicted per-second, because every step generates \~8 tokens). With `n_max ≥ 3` the server crashes (cuBLAS bug in b10218), but a workaround saves it:

Code

GGML\_CUDA\_DISABLE\_GRAPHS=1 ./llama-server ...

This works on every n\_max from 3 to 7. Stable through 1024-token generations. The dream config.

*Stage 7: Production reality check*

Then I tested with diverse real-world prompts (not the same Shakespeare quote × 1300) and longer generations (512 and 1024 tokens instead of 256):

**The dream is dead.** In production:

* `mean_draft_len` plateaus around 3.0-4.1 regardless of `n_max` * Higher `n_max` only adds recursive draft forward overhead, lowering tg t/s without meaningfully raising mean\_len * n\_max = 3 wins (highest tg at 41.6 t/s, needs workaround) * n\_max = 2 is even better if you remove the workaround (see Stage 8)

*Stage 8: n\_max = 2 without the workaround*

The cuBLAS crash only triggers at `n_max ≥ 3` (with this build). With `n_max = 2` you don't need the workaround, and the `GGML_CUDA_DISABLE_GRAPHS=1` cost (\~5-10%) goes away.

Code

n\_max = 2, no workaround, real workloads:
  tg = 56 t/s, mean\_len = 3.00, eff\_tps = 168

**Same 56 t/s and 168 effective tok/s as the cold-cache measurement. This is the real production number, and it's also the safest config.**

**The complete picture**

A 3.3× tg-t/s speedup over the simple layer-split baseline, with 3× the context window (96k vs 32k) and 100% draft acceptance.

The production command

This is what I actually run on my box:

\~/llama.cpp/build/bin/llama-server \\
  --model /mnt/Data/Models/unsloth/Qwen3.6-27B-MTP-GGUF/Qwen3.6-27B-Q4\_K\_S.gguf \\
  --jinja \\
  --split-mode tensor --tensor-split 1,1 \\
  --ctx-size 98304 \\
  -fa on \\
  --batch-size 1024 --ubatch-size 256 \\
  --threads 2 \\
  --n-gpu-layers 99 \\
  --parallel 1 \\
  --spec-type draft-mtp \\
  --spec-draft-n-max 2 \\
  --spec-draft-p-min 0.44 \\
  --host 0.0.0.0 --port 8081

**Bulletproof, no workarounds needed, 56 t/s, 96k context. This is the configuration I'd actually deploy.**

Lessons learned (what I'd tell past-me)

1. Don't trust short-prompt benchmarks for n\_max sweeps. Repetitive test text makes the MTP head look overconfident. Always measure with diverse prompts and longer generations before locking in a config.
2. Q4\_K\_S gives you more context headroom than Q4\_K\_XL on VRAM-tight boxes. On 24 GB total VRAM, the difference between 17 GB and 16 GB model files is the difference between 49k and 96k max context. Throughput is essentially identical.
3. Tensor split is mandatory on dual GPU. Without it, I was stuck at 17 t/s. The forward pass simply doesn't parallelize well across PCIe for matmuls.
4. The workaround `GGML_CUDA_DISABLE_GRAPHS=1` costs \~5% throughput but prevents the cuBLAS crash when MTP is enabled at `n_max ≥ 3`. Track the upstream issue at `ggml-org/llama.cpp#25061`.
5. `p_min = 0.44` is the sweet spot. I tried 0.7, 0.85 — when acceptance is already 100%, raising p\_min doesn't help (and slightly hurts because the model rarely hits the gate).
6. My own benchmark data was misleading me initially. 39 t/s on 7349-token prompts vs 56 t/s on 2341-token prompts — same model, same config. The difference was prefill cost, not throughput. The number that matters is `effective tokens/second = tg × mean_draft_len`, not raw `predicted_per_second`.

The configurations that didn't work (saving you the trouble)

For completeness, here's what I tried that did not help:

If anyone has ideas on how to push past 56 t/s on this box, I'm all ears. The mean\_draft\_len plateau is the obvious next thing to investigate — does anyone know if there's a way to make the MTP head more confident on diverse prompts? Or is this fundamentally a limit of single-model self-speculation?

*Hardware: 2× RTX 3060 12GB + i7-6700 + 64GB DDR4. Model: Qwen3.6-27B Q4\_K\_S with baked-in MTP head. llama.cpp b10218. All measurements on the 2× RTX 3060 box described above. YMMV on other hardware.*

I spent the day benchmarking Qwen3.6-27B-MTP-APEX on a modest 2× RTX 3060 12GB + i7-6700 system. Started at 16.7 t/s with a basic layer-split config and ended up at 56 t/s at 96k context — a 3.3× speedup from where I started, and the context window went from 32k to 96k in the process.

This post walks through every measurement, every wrong turn, and the final config that actually works. The single command at the bottom replaces my previous setup.

TL;DR

• Final production config: Qwen3.6-27B Q4_K_S + tensor split + MTP (n_max=2, p_min=0.44) → 56 t/s at 96k ctx, F16 KV cache, 3.3× faster than the basic layer-split setup
• The killer flag: --split-mode tensor --tensor-split 1,1 — without it, I was stuck at 17 t/s
• The hidden trap: Short benchmark prompts inflate mean_draft_len and make high n_max look amazing. Real workloads cap out at ~3-4.

Hardware

CPU:  Intel Core i7-6700 @ 3.40 GHz (Skylake, 4C/8T, 14nm)
Mobo: ASUS Z170 PRO GAMING (Intel Z170, LGA 1151)
RAM:  64GB DDR4 — DIMM_A1: Samsung 32GB @ 2720 MT/s
                  DIMM_B1: Team Group 32GB @ 2720 MT/s
                  (mismatched sticks, both XMP'd to 2720 from 2400 stock)
GPU:  2× NVIDIA GeForce RTX 3060 Lite Hash Rate (GA106), 12GB VRAM each
      (Intel HD 530 used for monitor output)
PSU:  (whatever I had lying around — total system power ~350W under load)
OS:   Linux Mint 22.3 "Zena" (Ubuntu 24.04 base, Linux 6.x kernel,
     Cinnamon desktop, X11 session)
CUDA: 12.0 (nvidia-driver-535)
llama.cpp: self-built from source, version 10218 (commit de699957b)
         built with -DGGML_CUDA=ON, NCCL disabled

This is a $400 dual-3060 setup I picked up second-hand. Not impressive on paper. Let's see what it can do.

The GPUs run in tensor-split 1.05,0.95 mode because the first card has a slightly better silicon lottery result and can take 5% more of the tensor-parallel split. No NVLink — just PCIe 3.0 x8/x8.

The journey: 9 measurements, 9 surprises

Every measurement below used the same prompt set: short Shakespeare quote (. * 1300 ≈ 16k chars ≈ 4000-4500 tokens), 256-token generation, warm cache unless noted. All on Qwen3.6-27B-MTP-GGUF.

Stage 1: Q4_K_XL, layer split (basic config)

Config: --split-mode layer --tensor-split 1.05,0.95 --cache-type-k q8_0 --cache-type-v q8_0 Result: 16.73 t/s

The default setup. Predictable. The Q4_K_XL model is 17 GB on disk, which combined with q8_0 KV at 32k ctx uses ~24 GB VRAM — basically all of it.

Stage 2: Try MTP (no tensor split)

Config: + --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.44
Result: 26.96 t/s (+61%)

MTP helps on this dense model? Apparently. The 81.9% draft acceptance rate pays off because dense forward passes are cheap enough that the draft overhead doesn't exceed the savings. Interesting.

Stage 3: Tensor split (the breakthrough)

Config: --split-mode tensor --tensor-split 1,1 (no KV quant — F16 KV)
Result: 28.46 t/s (+70% over baseline)

Tensor split distributes every matmul across both GPUs simultaneously instead of running layers sequentially. Combined with F16 KV cache, this is 70% faster than layer split.

But wait — tensor split means I can't use quantized KV cache. Trade-off accepted because tensor split was worth more than the KV quant savings.

Stage 4: Tensor split + MTP (the holy-cow moment)

Config: tensor split + draft-mtp n_max=2 + F16 KV Result: 39.13 t/s on 7349-token prompt, 55.97 t/s on 2341-token prompt draft acceptance 85%, mean draft len 3.00

This was the moment. 2.33× over baseline. MTP + tensor split, who would have thought.

Stage 5: Switch to Q4_K_S, push context to 96k

The Q4_K_XL at 32k ctx was using 17 GB model + 7 GB F16 KV ≈ 24 GB VRAM. OOM at 64k. But Q4_K_S is 16 GB instead of 17 — fits at 96k!

Config: Q4_K_S + tensor_mtp at 96k ctx
Result: 56.4 t/s, draft acceptance 100%, mean draft len 3.00

Identical throughput to 32k, but 3× the context. The lower quant also makes the model more confident — acceptance goes from 85% to 100%.

Stage 6: n_max sweep (cold cache, short prompts)

Hypothesis: if mean_draft_len = 3.00 at n_max = 2, then n_max = 4 or 5 or 6 should give mean_draft_len = 5, 6, 7 — even faster.

Whoa! n_max = 7 looks like 512 effective tok/s (62× the predicted per-second, because every step generates ~8 tokens). With n_max ≥ 3 the server crashes (cuBLAS bug in b10218), but a workaround saves it:

Code

GGML_CUDA_DISABLE_GRAPHS=1 ./llama-server ...

This works on every n_max from 3 to 7. Stable through 1024-token generations. The dream config.

Stage 7: Production reality check

Then I tested with diverse real-world prompts (not the same Shakespeare quote × 1300) and longer generations (512 and 1024 tokens instead of 256):

The dream is dead. In production:

• mean_draft_len plateaus around 3.0-4.1 regardless of n_max
• Higher n_max only adds recursive draft forward overhead, lowering tg t/s without meaningfully raising mean_len
• n_max = 3 wins (highest tg at 41.6 t/s, needs workaround)
• n_max = 2 is even better if you remove the workaround (see Stage 8)

Stage 8: n_max = 2 without the workaround

The cuBLAS crash only triggers at n_max ≥ 3 (with this build). With n_max = 2 you don't need the workaround, and the GGML_CUDA_DISABLE_GRAPHS=1 cost (~5-10%) goes away.

Code

n_max = 2, no workaround, real workloads:
  tg = 56 t/s, mean_len = 3.00, eff_tps = 168

Same 56 t/s and 168 effective tok/s as the cold-cache measurement. This is the real production number, and it's also the safest config.

The complete picture

A 3.3× tg-t/s speedup over the simple layer-split baseline, with 3× the context window (96k vs 32k) and 100% draft acceptance.

The production command

This is what I actually run on my box:

~/llama.cpp/build/bin/llama-server \
  --model /mnt/Data/Models/unsloth/Qwen3.6-27B-MTP-GGUF/Qwen3.6-27B-Q4_K_S.gguf \
  --jinja \
  --split-mode tensor --tensor-split 1,1 \
  --ctx-size 98304 \
  -fa on \
  --batch-size 1024 --ubatch-size 256 \
  --threads 2 \
  --n-gpu-layers 99 \
  --parallel 1 \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  --spec-draft-p-min 0.44 \
  --host 0.0.0.0 --port 8081

Bulletproof, no workarounds needed, 56 t/s, 96k context. This is the configuration I'd actually deploy.

Lessons learned (what I'd tell past-me)

1. Don't trust short-prompt benchmarks for n_max sweeps. Repetitive test text makes the MTP head look overconfident. Always measure with diverse prompts and longer generations before locking in a config.
2. Q4_K_S gives you more context headroom than Q4_K_XL on VRAM-tight boxes. On 24 GB total VRAM, the difference between 17 GB and 16 GB model files is the difference between 49k and 96k max context. Throughput is essentially identical.
3. Tensor split is mandatory on dual GPU. Without it, I was stuck at 17 t/s. The forward pass simply doesn't parallelize well across PCIe for matmuls.
4. The workaround GGML_CUDA_DISABLE_GRAPHS=1 costs ~5% throughput but prevents the cuBLAS crash when MTP is enabled at n_max ≥ 3. Track the upstream issue at ggml-org/llama.cpp#25061.
5. p_min = 0.44 is the sweet spot. I tried 0.7, 0.85 — when acceptance is already 100%, raising p_min doesn't help (and slightly hurts because the model rarely hits the gate).
6. My own benchmark data was misleading me initially. 39 t/s on 7349-token prompts vs 56 t/s on 2341-token prompts — same model, same config. The difference was prefill cost, not throughput. The number that matters is effective tokens/second = tg × mean_draft_len, not raw predicted_per_second.

The configurations that didn't work (saving you the trouble)

For completeness, here's what I tried that did not help:

If anyone has ideas on how to push past 56 t/s on this box, I'm all ears. The mean_draft_len plateau is the obvious next thing to investigate — does anyone know if there's a way to make the MTP head more confident on diverse prompts? Or is this fundamentally a limit of single-model self-speculation?

Hardware: 2× RTX 3060 12GB + i7-6700 + 64GB DDR4. Model: Qwen3.6-27B Q4_K_S with baked-in MTP head. llama.cpp b10218. All measurements on the 2× RTX 3060 box described above. YMMV on other hardware.

System Specs:

CPU: Intel Core i7-6700 @ 3.40 GHz (4 cores, Skylake 14nm)

Motherboard: ASUS Z170 PRO GAMING (Intel Z170, LGA 1151)

RAM: 64 GB DDR4

• DIMM_A1: Samsung 32GB @ 2720 MT/s (M471A4G43AB1-CWE)
• DIMM_B1: Team Group 32GB @ 2720 MT/s (TEAMGROUP-SD4-3200)
• DIMM_A2/B2: Empty
• Note: Mismatched sticks (Samsung + Team Group). 2720 MT/s is overclocked via XMP (official Z170 DDR4 limit is 2133, XMP typically 2400-2666, 2720 is OC'd).

GPU:

• GPU 0: NVIDIA GeForce RTX 3060 Lite Hash Rate (GA106) — 12GB VRAM
• GPU 1: NVIDIA GeForce RTX 3060 Lite Hash Rate (GA106) — 12GB VRAM
• Intel HD Graphics 530 (integrated, used for monitor output)

The Setup:

I've been experimenting with MoE models, specifically trying to offload certain parts to CPU/SystemRAM to see if it helps. Currently testing the SC117/Ornith-1.0-35B-MTP-APEX-I-Balanced.gguf model.

It's looking pretty good — I'm getting a comfortable ~33-35 t/s generation speed. But here's the thing: when I enable MTP (Mixture of Tokens Parallelism), speed drops to max ~28 t/s. The MTP heads consume additional memory, so when I load them, more of the model falls back to SystemRAM.

Right now I'm running without MTP. Theoretically, the APEX quantization is the best quality I can get from this model, and the speed is already usable. So I've reached a point where I'm wondering:

What would you optimize here, if anything? Or is this about the maximum I can squeeze out of this model on this hardware?

Current llama-server command:

Code· bash

~/llama.cpp/build/bin/llama-server \
  --model /mnt/Data/Models/SC117/Ornith-1.0-35B-MTP-APEX-GGUF/Ornith-1.0-35B-MTP-APEX-I-Balanced.gguf \
  --jinja \
  --n-gpu-layers 40 \
  --split-mode layer \
  --tensor-split 1.05,0.95 \
  --n-cpu-moe 5 \
  --ctx-size 131072 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --no-mmap \
  -fa on \
  --batch-size 1024 \
  --ubatch-size 256 \
  --threads 2 \
  --host 0.0.0.0 \
  --port 8081 \
  --temp 0.8

Results from the log:

• Prompt processing: ~303 t/s (extremely fast)
• Generation: ~33.3 t/s stable
• GPU memory: 11709MiB + 11711MiB = ~23GB used (out of 24GB total)

Warnings from the log:

• failed to fit params to free device memory: n_gpu_layers already set by user to 40, abort
• model has unused tensor blk.40.* (several tensors in layer 40 marked as unused)
• fused Gated Delta Net (chunked) not supported, set to disabled

TL;DR: 33 t/s without MTP, 28 t/s with MTP. The APEX quant seems optimal for quality. Should I bother with MTP or is this the ceiling for this model on dual 3060s?

Edit: Fixed the --n-gpu-layers 40 warning — it's expected since I'm explicitly setting it. The blk.40 unused tensors are likely because layer 40 (0-indexed) spills to CPU and some of its tensors aren't needed for this architecture.

A reflection on why apps (especially banking apps) refuse to run on hardware you actually own — and why this amounts to a quiet dismantling of your digital self-determination. I'd like people to understand what's happening beyond the screen.

^(These examples are mostly Hungarian, but the pattern is global. Please share examples from your own country in the comments — let's build a map of how this plays out worldwide.)

Introduction

Digital freedom is disappearing slowly, quietly — and most people don't even notice.

In recent years, more and more banking apps have decided they will simply not run on phones where the user has made their own choices about the software. If the bootloader is unlocked, if the device is rooted, if you're running an AOSP-based Android, or a Linux-based mobile OS (Ubuntu Touch, Mobian, /e/OS, GrapheneOS), the app just refuses to launch.

At first glance, this looks like a security measure. In reality, it's about a great deal more.

I. The phone is yours. Or is it?

You bought the phone. The hardware is yours. In principle, you should be the one deciding what operating system or software runs on it.

And yet we've reached a point where certain essential services — banking being the obvious example — are only conveniently accessible if the user runs exactly the software environment a handful of large corporations deem acceptable.

Banking apps today do far more than banking. They start by thoroughly inspecting your device:

• they check the bootloader state,
• they check system integrity,
• they check for root privileges,
• they pull the manufacturer, firmware, and hardware fingerprint (biometrics live in a different story),
• and they ask Google whether the device carries Google's certification.

Google gives all of this away for free through the Play Integrity API (formerly SafetyNet) — so for developers, "all" they have to do is integrate it. A few lines of code, and suddenly the app decides what you're allowed to run on your own hardware.

This has a cost

• it burns CPU time on your phone,
• it consumes energy — increasingly expensive these days,
• it eats storage — itself becoming a privilege,
• and it locks up development capacity inside the bank.

But the biggest cost isn't any of that. It's that the user loses the freedom to choose.

If a banking app only works on certain systems, the user eventually has no choice but to use those. This indirectly determines:

• which phone is worth buying,
• which operating system you may use,
• and when you have to replace a device that otherwise works perfectly.

So it's no longer the user who decides. It's the provider.

II. The monoculture trap

This has serious long-term consequences.

Developers go where the users are. If every important app only works properly on two or three platforms, developing for alternative systems becomes harder and harder. Fewer users, fewer developers, fewer apps — a self-reinforcing cycle kicks in.

In parallel, technological diversity shrinks — and from an information-security standpoint, that is not a good thing.

When almost everyone runs the same platform, a single critical vulnerability or serious security flaw can affect a huge number of users at the same time. Diversity isn't just healthy competition — it is, in a real sense, a more resilient ecosystem.

Just remember: Log4Shell, Heartbleed, and the Pegasus spyware were all especially dangerous because the affected software was everywhere. Diversity is the simplest and cheapest "immune system" we have.

III. GrapheneOS — the rebuttal the banks find most inconvenient

It is worth pointing out that not every alternative system is less secure. There are Android variants — the best known being GrapheneOS — that many security professionals consider genuinely high-security. GrapheneOS is:

• open source, fully auditable,
• considered a benchmark in the security research community,
• effectively resistant to everything a typical Android device can be tricked into running.

The independent security portal privsec dot dev maintains a tracker of how banking apps behave on GrapheneOS. The result: many of those apps don't block GrapheneOS because it's less secure — they block it because it lacks Google's certification. In other words, the decision is not a security decision, it's a platform-political one.

This is a good illustration that the current practice is not really about actual security risk — it's about a trust model that protects platforms, not users. The bank trusts Google, not the user. The bank hands the decision to Google, not to its own engineers.

IV. Personal data: the invisible tax

Then there's the data problem.

The big platforms no longer just know what you buy. They also know:

• when you wake up (the first screen-unlock of the day),
• where you move (GPS, cell-tower info, Wi-Fi),
• when you work (app-usage patterns),
• who you talk to, how often, in what mode,
• how your habits shift from month to month, year to year.

From this data, extremely accurate behavioural models can be built. Not rough statistics — individual-level predictions.

This data wealth is used for advertising and business decisions (e.g. telling a corner shop, based on foot-traffic patterns, how many staff and what opening hours are worth running), but also for:

• political campaign targeting — crowd reactions can now be modelled at the individual level,
• law-enforcement profiling — transactional and location data can be assembled into patterns of daily life.

Your digital footprint is your new real estate — except you don't own it, and you don't get the rent.

V. The squeeze on cash: the last link in the chain

On top of all this comes the gradual displacement of cash.

A growing number of large transactions can only be settled by bank transfer:

• In 2026, for agricultural land sales, full abolition of cash payment had been planned — only wire transfer or escrow with a lawyer would have been allowed. Under public pressure, the original plan was softened: in some narrow cases, cash is still technically usable, but under severely restricted conditions that turn it into a fiction of free choice.
• For real-estate purchases, mandatory bank transfer is also becoming the norm; in many cases the seller's side will no longer accept cash at all.

If you want to do this quickly, mobile banking becomes essentially unavoidable. And if you want to use mobile banking, the bank often tells you which operating system your phone must run. If you refuse, the only option is a branch visit. During office hours. With a queue. And a day of your life gone.

Budapest parking — 1 July 2026

The Budapest General Assembly decided on a phased removal of parking meters. In the summer of 2026, more than three thousand machines were taken off the streets, while fees were simultaneously raised by 30–50%. Cash and card payment at the meter are gone — parking can only be paid for by mobile phone: app, SMS, or voice call.

Notice the pattern:

1. First came the "convenient" app.
2. Then the provider made the app effectively mandatory.
3. The previous option (meter, cash) was scrapped.
4. The price was raised, because now you "have to" use it anyway.

The enforcement officer offers no way to comply lawfully — there is no ticket to buy from them. They only fine. But if you have cash, time, and no smartphone, there is nowhere to buy from. It forces everyone — the old, the young, the poor, the rich alike — to own a smartphone, have a bank card on file, and actually use it. This is the modern form of digital exclusion. The citizen is expected to know which zone they are in, the phone number, the exact wording of the SMS, and how to operate the device. The margin for error is zero — otherwise the fine arrives.

VI. Why is this the bank's preferred solution?

Banks obviously have legitimate security concerns. Nobody wants their account broken into, or strangers accessing their money.

But the real question is: is this really the only possible path?

The current practice essentially says: "If your phone does not run exactly the software environment we accept, you simply cannot use our service conveniently."

That is undeniably the easier path for the bank.

Banks have significant financial, development, and IT resources. In principle, they could:

• harden the application itself (internal code review, anti-reverse-engineering),
• strengthen server-side checks (anomaly detection, behaviour-based fraud analysis),
• protect communication (end-to-end encryption, hardware-backed key storage),
• improve authentication (FIDO2, hardware security keys, biometric signing),
• or even work together with communities and developers to securely support more operating systems.

Instead, in many cases, it seems simpler to exclude everyone who doesn't match the platforms they accept.

This has a cost. The bank doesn't pay it. The user does.

The user who, years after buying the device, would otherwise still be perfectly happy with their phone. The user who consciously chose an open-source operating system. The user who wants to decide for themselves what software to run on a device they actually own.

Naturally, this is the most profitable path — the one that costs the bank the least. As for what it costs millions of users — not to mention the global pollution — it seems nobody is really interested. Only the current business goal matters.

VII. Planned obsolescence

Many apps today require a specific minimum Android version. A five-, six-, or more-year-old phone that otherwise works perfectly can easily lose its usefulness — not because the hardware failed, but because a provider has decided to stop supporting it.

This isn't just a financial issue. It's an environmental one.

Every phone replaced before its time means:

• more electronic waste,
• more raw-material extraction (cobalt, lithium, rare earths),
• more energy used in manufacturing and shipping,
• and the carbon footprint baked into the device loses its meaning the moment it's thrown away early.

While many companies loudly communicate sustainability and "green" operations, they say much less about the fact that their own software support decisions have a major impact on how long an electronic device can actually be used.

Sustainability is not just about office recycling bins or reducing paper use. It's also about not forcing millions of people to replace perfectly functional devices purely because of software decisions.

The contrast: Linux Mint on a 3rd-generation Intel CPU

Think about it: on a 10+ year-old computer with a 3rd-generation Intel processor, a modern Linux distribution like Linux Mint 22 still runs without trouble. Browsing, programming, document editing, watching video — and even more demanding development or graphics work can be done with the right software. The same hardware is officially unsupported by Windows 11.

So the question isn't whether it's technically impossible to run a modern OS on that machine — other systems prove that it isn't. The question is which direction the software industry is steering users.

The same pattern shows up in professional software: there are well-optimized browser-based 3D design and modelling tools that still run on decade-old hardware, while comparable "protected" and heavily marketed expensive software requires a brand-new workstation bought at astronomical prices. Same logic, same pattern, same business model.

VIII. Meta, Instagram, Messenger — and what's coming next

I haven't gone into detail on the serious data-collection and behavioural-analysis risks yet. But Meta (Facebook, Instagram, Messenger, WhatsApp), Google, TikTok, and the other major platforms are turning exactly this danger into a daily reality — not in theory, but in everyday life.

Chat Control — tomorrow's problem, already here today

On top of all this comes a new layer: the EU's so-called "Chat Control" proposal.

The official goal is to combat online sexual abuse of children. The actual content of the draft, however, would mean:

• client-side scanning — your message would be run through an algorithm on your own device, before encryption, and compared against a central database,
• messaging providers (Signal, WhatsApp, Telegram, Messenger) would be obliged to roll this out for all users,
• all this while end-to-end encryption is kept in place — the proposal's logic being that encryption "is not broken," when in practice the opposite is true.

If your own device is the one scanning your own messages, that is the end of private communication. Not because "encryption is broken," but because encryption no longer protects you: the content is read by a third party's algorithm before it is sent. It's the same logic as a bank telling you which software you can run on your device: the provider checks, through your own device, without your consent, in the name of your own good.

The proposal triggered serious debate in 2024 and 2025, and returned to the agenda in 2026. The details keep shifting, but the direction is clear: private communication is being gradually turned into supervised communication.

IX. What can you do?

1. Deliberately rate down these apps in the App Store and Google Play. It doesn't matter whether you personally use them. The rating is immediately visible on the top charts, and it creates pressure.
2. Publicly boycott providers that only work on Google-certified, locked-bootloader, "managed" environments. Your money is your vote in a system that otherwise ignores you.
3. Formally raise the issue with your bank and provider: "Open up your service to AOSP, rooted devices, Ubuntu Touch, a proper web interface, hardware security keys, FIDO2…" They may say no. But the request will be on the record.
4. Hold decision-makers accountable at the political level: when a municipality or a state abolishes cash payment without offering an alternative, that is a form of digital exclusion that hits the elderly, the poor, the digitally restricted, and the privacy-conscious alike. That is not neutrality — that is serious discrimination.
5. Speak about it publicly. Share, comment, debate, write. Digital exclusion happens in silence — and silence is its main weapon.

X. Closing

Windows is a story of its own — as is the world of closed-source software. Age-verification mandates and similar absurdities are arriving there as well. iOS, Android, their ongoing "wonders," online age checks, the closure of social platforms, the narrowing of options and the increasingly heavy-handed steering of free choice, the masses shaped by media narratives — those are topics for another post that never quite gets written.

For now, we are here: the boycott of banking apps, the deliberate low ratings, public protest. That is the only way to bring the giants to their senses. Let the market dance to our song, not the other way around.

There are solutions that would simply warn the user of the risks, and then leave the decision in their hands. Just as we can decide which operating system to install on a computer, we should have the right to decide which software to use on a phone we actually own.

Technically, security does not have to mean the abolition of freedom — and, with it, the discrimination of ordinary people.

Tudjátok-e, hogy minek/kinek szólt a déli harangszó?