TL;DR

• 3× modded RTX 2080 Ti 22 GB (66 GB VRAM total) runs Qwen3.8-Flash-Next 125B MoE (~6B active) at 66.6 tok/s decode / 742 tok/s prefill with Strata’s layer-split — and 932 tok/s prefill on a fresh 83K-token prompt.
• First same-harness perplexity comparison of the two competing quant families on this model (neither vendor publishes this head-to-head): UD-Q4_K_XL 4.1263 > IQ3_S 4.1967 > IQ3_XXS 4.2173, paired t-test over 72 chunks: Q4’s lead is significant (t=4.45 / 5.33); S vs XXS is not (t=1.05).
• Strata layer-split beats NVLink peer mode by +83% and beats a standard llama.cpp partial-offload layout by 2.5× decode / 4.4× prefill on identical hardware.
• 512K context is nearly free: 59.9 vs 66.6 tok/s at 262K (-10%, within noise of these runs), RSS +4 GB.

HARDWARE

Part	Detail
CPU	Intel Core i9-7980XE (Skylake-X), 18C/36T @ 2.6 GHz base, AVX-512
Board	X299-E chipset (LGA2066), ASUS
RAM	128 GB DDR4-3600, quad-channel (8×16 GB G.Skill F4-3600C18, 2 DPC) — 125 GiB usable
Measured RAM BW	STREAM triad 73.6 GB/s, 2 GB-block write 103.6 GB/s (peak spec 115.2)
GPUs	3× RTX 2080 Ti 22 GB (11→22 GB mod) = 66 GB VRAM
GPU power	167 W cap each (67% of stock 250 W, tuned from an efficiency sweep), boost 2100 MHz
Topology	GPU0 x16, GPU1 x16 + NVLink→GPU2 (2×25.8 GB/s), GPU2 x8 (on a riser)
Storage	2× NVMe SSD RAID0, 953.6 GiB, ~2.6 GB/s read (model files)
Software	Linux, driver 580.178.04 / CUDA 13.0, Strata 0.1.38, llama.cpp build 349 (sm_75+sm_86)

Models (all GGUF, from HF):

Quant	bpw	Size	Source
UD-Q4_K_XL	~4.7	111.3 GB	unsloth Dynamic
GSQ-RCO IQ3_S	3.50	83.6 GB	ISTA-DASLab
GSQ-RCO IQ3_XXS	3.00	75.8 GB	ISTA-DASLab

Strata splits the 48 layers into 3 stages (layer-split), keeps hot experts GPU-resident in an adaptive per-card cache (97–100% of routed mass), streams the rest from a RAM arena, PLE/n-gram table reads direct from NVMe, KV = int8 with a 32768-cell VRAM window.

SPEED (3-RUN AVERAGES, ~3K-TOKEN PROMPTS, KV INT8, CTX 262144 UNLESS NOTED)

Decode tok/s

Q4_K_XL   layer-split [0,1,2]      ████████████████████████████████████████66.6
IQ3_XXS   layer-split [0,1]        ███████████████████████████████████████▊63.2
IQ3_XXS   layer-split [0,1,2]      ███████████████████████████████████████ 61.3
Q4_K_XL   layer-split [0,1,2] 512K ████████████████████████████████████▍   59.9
IQ3_S     layer-split [0,1,2]      ███████████████████████████████████▍    58.3
Q4_K_XL   layer-split [0,1]        ███████████████████████████████████▍    57.6
IQ3_S     layer-split [0,1]        █████████████████████████████████▍      56.1
IQ3_XXS   1 GPU                    ████████████████████████████            50.5
IQ3_S     1 GPU                    ███████████████████████████             48.6
Q4_K_XL   NVLink peer 512K         █████████████████████                   36.6
llama.cpp partial-offload 512K     █████████████████                       27.0
Q4_K_XL   1 GPU                    ███████████████                         25.7

Prefill tok/s (short prompts)

IQ3_XXS   layer-split [0,1,2]      ████████████████████████████████████████ 742
IQ3_S     layer-split [0,1,2]      ███████████████████████████████████████▍ 726
IQ3_XXS   layer-split [0,1]        ███████████████████████████████████████▊ 700
IQ3_S     layer-split [0,1]        ████████████████████████████████████▍    671
IQ3_XXS   1 GPU                    ████████████████████████████             525
Q4_K_XL   layer-split [0,1,2]      ███████████████████████▍                 448
IQ3_S     1 GPU                    ███████████████████████▎                 426
Q4_K_XL   layer-split [0,1,2] 512K ███████████████████████                  424
Q4_K_XL   layer-split [0,1]        ████████████████████                     377
Q4_K_XL   1 GPU / peer             ████████████                            ~232
llama.cpp partial-offload          ██████                                   102

One-shot prefill on a fresh 83K-token prompt (this is the fun one)

Q4_K_XL layer-split [0,1,2] 512K  ████████████████████████████████████████ 932.5 tok/s
Q4_K_XL NVLink peer         512K  █████████████████████▌                   490.2 tok/s

(83,306 fresh tokens, 0 cache reuse — the 3-stage split parallelizes the prompt path; decode at 83K context: 58.9 tok/s, expert-cache hit 97.4%, KV VRAM-hit 92.8%.)

Expert-cache hit rate (avg) + per-model footprint @ layer-split [0,1,2]

	Q4_K_XL	IQ3_S	IQ3_XXS
cache hit	96.9%	99.9%	100.0%
slots (of 24576)	15,218	24,390	24,576
load time	134 s	52 s	64 s
engine RSS	81.6 GB	55.8 GB	48.7 GB
VRAM / card	~21.5 / 22.5 GB (96%) all three

Ablations (on Q4 split3 — what NOT to do)

variant	prefill	decode	verdict
baseline	448	66.6	—
+ PLE table locked in RAM	447	66.8	neutral — costs 29 GB pinned + swap, rejected
+ full KV in VRAM (no streaming)	427	65.2	small loss (fewer expert slots), rejected

Observations

1. Layer-split >> NVLink peer mode: same model, 2 cards: peer 36.4 → split 57.6 tok/s (+58%); 3rd card adds +16% even though it’s the x8 slot (split hand-off goes through pinned host memory — NVLink direct is still an open Strata issue #338).
2. The 1-GPU ranking inverts on 3 GPUs: 1 card → IQ3_XXS 50.5 > IQ3_S 48.6 > Q4 25.7 (smaller blobs = higher cache hit). 3 cards → Q4 66.6 > XXS 61.3 > S 58.3 with all caches near-full — pointing at per-format GPU kernel paths (Strata ships dedicated Q4_K/Q5_K/Q5_1/Q8_0 expert kernels since 0.1.31) plus draft-acceptance variance, not at cache misses.
3. 512K context costs ~5-10% vs 262K (rope linear ×2): prefill 424 vs 448 (-5.5%), decode 59.9 vs 66.6 (round-spread is ±6 tok/s on both — read as noise to ~5%); draft acceptance dipped 73%→63% at 512K (possible mild effect of the scaled rope on drafting, or just run variance).

ACCURACY

Perplexity — wikitext-2-raw test, measured here

Method: llama-perplexity (build 349), test split = 4,358 rows / ~294K tokens, 72 chunks × ctx 4096, q8_0 KV, flash-attn on, byte-identical GPU placement flags for all three models (same -ot expert pins as the speed tests). Lower = better.

excess mean-log-PPL vs best (paired over the same 72 chunks, lower = better):

UD-Q4_K_XL  4.1263 █                                  (best)
GSQ IQ3_S   4.1967 ███████   +0.0704  (+1.71%)
GSQ IQ3_XXS 4.2173 █████████ +0.0910  (+2.21%)

comparison	mean-log diff	paired SE	t	verdict
Q4 vs IQ3_S	−0.0704	0.0158	−4.45	Q4 significantly better
Q4 vs IQ3_XXS	−0.0910	0.0171	−5.33	Q4 significantly better
IQ3_S vs IQ3_XXS	−0.0206	0.0196	−1.05	not significant (S nominally better)

• A duplicate full Q4 run produced bit-identical results (deterministic harness), so differences above ~0.01 mean-log are real, not noise.
• The paired test (same chunks for every model) is far more sensitive than the tool’s unpaired ±0.024.

Published task numbers (ISTA’s evaluation — not re-run here)

	BF16	IQ3_S	IQ3_XXS
Task avg (AIME25 / GPQA-D / LCB v6)	93.12	93.26	92.57 (99.4%)
AIME25	100.00	100.00	100.00
GPQA-Diamond	91.92	92.93	91.41
LiveCodeBench v6	87.43	86.86	86.29

unsloth’s own metrics for the UD ladder (token-level, different scale — do not cross-read with task scores): Q4_K_XL top-1 recovery 92.26%, KLD 0.0469 · Q5 93.68/0.0304 · Q6 94.09/0.0271.

Reading of it: on distribution fit (ppl) the 4-bit UD wins measurably; on published tasks the IQ3_S claims base parity. Both can be true — ppl and task accuracy measure different things, and we have no BF16 ppl baseline here (354 GB). For “which do I run”: Q4 = best decode + best ppl; IQ3_S = smallest file / fastest load / published task parity; IQ3_XXS = best prefill, cheapest bits.

LLAMA.CPP COMPARISON (SAME BOX, SAME MODEL FILE)

The 103.7 GiB model can’t fit 66 GB VRAM + 512K KV + 28.8 GB PLE, so llama.cpp runs the usual partial-offload layout:

--n-cpu-moe 48                       # 48 expert layers
--split-mode layer --tensor-split 1,1,1
-ot 'per_layer_token_embd=CPU,       # PLE/n-gram -> CPU
     blk.(8|9|1[0-4])...=CUDA0,      # 7 expert blocks -> GPU0
     blk.(2[0-7])...=CUDA1,          # 8 -> GPU1
     blk.(3[8-9]|4[0-3])...=CUDA2'   # 6 -> GPU2
     # = 21 blocks on GPU, 27 expert layers on 16 CPU threads
--ctx-size 512000 --cache-type-k q8_0 --cache-type-v q8_0
--flash-attn on --threads 16 + MTP draft sidecar

Measured: 102 prefill / 27.0 tok/s decode (draft acceptance ~79.6%).

This is not “llama.cpp is slow” — it’s a different trade: Strata’s layer-split runs the same stage decomposition, but each stage computes its hot experts from GPU memory (adaptive cache, 97–100% routed mass) instead of gathering 27 layers’ experts from the CPU pool every token. Same split, different expert story → 4.4× prefill, 2.5× decode. Caveat: our build doesn’t have qwen4exp’s re-enabled -sm tensor path (upstream #28569), so tensor-split wasn’t an option here.

CAVEATS

1. Speed prompts are synthetic (~3K tokens) + one 12K deep probe, 3 runs each; some configs show ±5–10% round spread (draft-acceptance variance) — treat sub-5% differences as noise.
2. Absolute ppl depends on the chunking mode; the valid claim is the cross-model ordering under identical settings.
3. Task tables are ISTA’s published numbers, not our measurements.
4. GPUs are power-capped at 167 W (efficiency-tuned); a 250 W run would likely shift sustained clocks a bit.
5. PPL ≠ task accuracy — both numbers are above so you can weight them yourself.

Configs, per-run JSONs and ppl logs are on hand — happy to post any specific one.

In the past few days, I have gained the following experiences:

Although the performance of the Qwen 3.8 27B model is not bad, but...

Let's start at the very beginning. In terms of raw computing performance, coding based on my local measurements and in practice, it seems that the Qwen3.6-27B-Q4_K++ models are almost on the same level as the Qwen3.8 27B model. The big difference is that the 3.8 thinks a lot. This is where its advantage comes into play. The tool calling is better, so it is more efficient overall. However, it is not necessarily smarter.

If I set the thinking value to med or min for the Qwen 3.8 27B instead of xhigh, it consistently underperforms in the tests compared to the Qwen 3.6 27B.

So Qwen 3.8 27B is only smarter if the reasoning level is xhigh, but then it is incredibly slow and generates a lot of tokens. And the worst thing is that even though the KV cache is Q8 quantized, it still requires a brutal amount of RAM/VRAM compared to Qwen 3.6.

And here comes the point that needs to be discussed.

Qwen 3.6 27B runs with 200k+ ctx on my 22 GB GPU in good quality.

Qwen 3.8 27B does not. This is not enough for it, there is a maximum of 100k ctx in the VRAM and the model is only Q4 quality, the KV cache Q8, because below it the model will be brutally stupid, a lobotomy. Obviously the best would be F16.

So I think that for planning and ideation, thinking, Qwen 3.8 27B is suitable, but for implementation 3.6 27B is perfect and MUCH faster. But...

This is where Qwen 3.8 Flash Next comes into play.

While on the 2080 Ti card (22GB Modded) on the i9-7980XE CPU with quad channel DDR4 (128GB) RAM:

the Qwen 3.8 27B model runs at Q4 quantum quality, ctx 100k, KV Q8 settings at approx. 35-45 t/s with MTP,

the Qwen 3.8 Flash Next (Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS) with STRATA, ngram can easily fit on 1 GPU with 256k ctx (KV Q8) and 64 GB RAM is enough for it. The speed is currently 60-75 t/s and the strata is developing at full speed.

A fast SSD is probably also important for this, but I'm not sure, because I think the ngram table is in the RAM now. By the way, I only have Gen3 PCI-e ports, so I solved the SSD acceleration by connecting 2 average SSDs to nvme ports and creating an array with RAID0, and then their speed added up.

Experiences about the Qwen 3.8 FN:

It doesn't overthink things. It can think a lot, but not always, only when justified. It's precise, follows instructions, smart. I'll risk saying that based on my usage habits, it's smarter than the GLM 5.3 Flash.

Since its speed is also better and the VRAM requirement is less than the Qwen 3.8 27B, the question now is which model should I choose? I think the answer is clear.

Do you have similar experiences?