I have a piece of software that I have been working on for a few months that I think is getting close to a 1.0 release, but I still need an icon for it. I have an idea that I have worked on for an icon, but I want help to get it perfect and in the correct format for Haiku. If anyone would like to help for a few bucks, send me a DM. TIA.

I benchmarked 13 models at 65K-128K context to find out what actually matters for agentic workloads — prefill dominates everything, and KV head count beats parameter count

I've been running local LLMs for agentic workflows (tool use, coding agents, RAG) and kept seeing people obsess over tg128 (token generation speed) as the headline performance metric. So I ran a structured long-context benchmark to figure out what actually matters when your context window is full. The answer surprised me.

---

Setup

• GPU: RX 7900 XT 20GB (Vulkan backend, RADV/Mesa)
• Backend: llama.cpp / llama-bench (build 9860)
• Flags: -ngl 99 (GTT spill), -fa on, -ub 2048 -b 16384, ASPM=performance, bare TTY to free VRAM
• 13 models: 5 dense, 6 MoE, 1 Mamba2 hybrid, 1 MLA MoE — ranging from 5GB to 18GB
• 3 KV cache tiers: Q8_0 K / Q4_0 V (aggressive), Q8_0 K / Q8_0 V (symmetric), F16 (baseline)
• Context sizes: 512, 4K, 16K, 65K, 131K — both pure prefill (pp) and prompt+gen (pg)
• Full run took ~21 hours across two sessions

---

Full prefill speed results (Q8_0 K / Q8_0 V KV cache, tokens/sec)

If you just want the raw numbers, here's every model tested. pp = pure prompt processing (prefill), tg128 = token generation (decode). Sorted by pp131K.

Model	Size	Type	pp512	pp4K	pp16K	pp65K	pp131K	tg128
Trinity-Mini	16G	MoE 3B/26B	2639	2924	2370	1419	923	150
Granite-4.0-H-Small	17G	Mamba2+MoE	1115	1271	1220	1043	875	71
Ornith-9B / Qwen3.5-9B	6G	Dense	2103	2220	1943	1274	873	92
Qwen3.6-35B-A3B	18G	MoE 3B/35B	2184	2736	2227	1268	802	110
Gemma-4-26B-A4B	14G	MoE 4B/26B	2523	2798	2076	1024	600	119
North-Mini-Code	15G	MoE 3B/30B	2155	2187	1568	900	579	134
Gemma-4-12B	7G	Dense	1492	1498	1145	595	350	66
Qwen3.6-27B	16G	Dense	693	681	602	406	285	32
Granite-4.1-8B	5G	Dense	1965	1807	1124	442	244	93
Ministral-3-14B	8G	Dense	1419	1325	916	404	232	67
Apriel-1.6-15B	9G	Dense	1332	1208	812	347	197	66
Devstral-24B	15G	Dense	829	796	628	313	---	42
GLM-4.7-Flash	16G	MoE (MLA)	1822	1054	358	---	---	---

A few things to note: Devstral-24B couldn't complete the 131K test (8 KV heads × 128 dim = 160 KB/token — KV cache alone is ~21GB at 131K). GLM-4.7-Flash crashed above 16K (MLA issue, see Finding 5). Ornith-9B is architecturally identical to Qwen3.5-9B.

---

Finding 1: At 65K+ context, prefill is 94–99% of wall-clock time. tg128 is nearly irrelevant for short agentic outputs.

Here's the wall-clock breakdown for a real agentic query — 65K context in, 300 tokens out (typical tool-use response). Sorted by total time:

Model	Type	Prefill	Decode	Total	Prefill %
Trinity-Mini (MoE 3B/26B)	MoE	46.2s	2.0s	48.2s	96%
Qwen3.6-35B-A3B (MoE)	MoE	51.7s	2.7s	54.4s	95%
Ornith-9B / Qwen3.5-9B	Dense	51.4s	3.3s	54.7s	94%
Gemma-4-26B-A4B (MoE)	MoE	64.0s	2.5s	66.5s	96%
Granite-4.0-H-Small (Mamba2)	Mamba2	62.8s	4.2s	67.1s	94%
North-Mini-Code (MoE)	MoE	72.8s	2.2s	75.0s	97%
Gemma-4-12B	Dense	110.2s	4.5s	114.7s	96%
Granite-4.1-8B	Dense	148.4s	3.2s	151.6s	98%
Qwen3.6-27B	Dense	161.4s	9.3s	170.7s	95%
Ministral-3-14B	Dense	162.0s	4.5s	166.5s	97%
Apriel-1.6-15B	Dense	188.9s	4.6s	193.5s	98%
Devstral-24B	Dense	209.5s	7.2s	216.6s	97%

Decode is 1–5% of the time you actually wait. If your agent makes a short tool call or writes a brief response, the only thing that matters is how fast you can process the context window.

This means benchmark reports that lead with tg128 are misleading for agentic use cases. pp65K / pp131K is the metric that matters. The pg(prompt, gen) blended metric is better but still obscures the split — a model with fast prefill + catastrophically slow decode can look mediocre on pg despite being excellent for short outputs.

---

Finding 2: KV head count is the dominant architectural factor for long-context prefill — not parameter count, not MoE vs dense

Prefill speed retention (% of pp4K speed) at increasing context, all models:

Model	Size	KV Heads	pp4K	16K	65K	131K	Type
Granite-4.0-H-Small	17G	Mamba2*	1271	96%	82%	69%	Mamba2+MoE
Qwen3.6-27B	16G	4×256	681	88%	60%	42%	Dense
Ornith-9B / Qwen3.5-9B	6G	4×128	2220	87%	57%	39%	Dense
Trinity-Mini	16G	4×128	2924	81%	49%	32%	MoE
Qwen3.6-35B-A3B	18G	4×128	2736	81%	46%	29%	MoE
Gemma-4-12B	7G	8×128	1498	76%	40%	23%	Dense
Gemma-4-26B-A4B	14G	4×256	2798	74%	37%	21%	MoE
North-Mini-Code	15G	4×128	2187	72%	41%	26%	MoE
Apriel-1.6-15B	9G	8×128	1208	67%	29%	16%	Dense
Ministral-3-14B	8G	8×128	1325	69%	31%	18%	Dense
Granite-4.1-8B	5G	8×128	1807	62%	24%	14%	Dense
Devstral-24B	15G	8×128	796	79%	39%	---	Dense
GLM-4.7-Flash	16G	MLA (1×576)	1054	34%	---	---	MoE (MLA)

*Granite-H-Small has 4 attention layers + 36 Mamba2 layers (recurrent state, no KV cache)

Ornith-9B / Qwen3.5-9B (9B dense, 4 KV heads × 128 dim = 64 KB/token KV) is 4.4× faster at 128K context than Apriel-15B (15B dense, 8 KV heads × 128 dim = 160 KB/token) — despite being the same dense class and half the size. The difference is purely KV cache architecture. Every attention pass has to scan the full KV cache, and 8 KV heads means 2.5× more data to scan per token.

Practical rule: When evaluating a model for long context, check n_kv_heads and head_dim in the config before looking at parameter count. Two models from the same family can differ by 3–4× at 128K if one has 4 KV heads and the other has 8.

---

Finding 3: Mamba2 hybrid models have near-flat prefill scaling. The architecture actually delivers.

I was skeptical of the Mamba2 hype, but the data is clear. Granite-4.0-H-Small (IBM, 4 attention layers + 36 Mamba2 layers) retains 69% of its pp4K speed at 131K context — every transformer model in the test dropped below 42%.

Model	pp4K	pp131K	Slowdown
Granite-H-Small (Mamba2)	1271	875	1.45×
Trinity-Mini (MoE)	2924	923	3.2×
Ornith-9B / Qwen3.5-9B (Dense GQA)	2220	873	2.5×
Granite-8B (Dense)	1807	244	7.4×

At 131K context, Granite-H-Small (17GB) ties Ornith-9B / Qwen3.5-9B (6GB) at ~875 t/s despite being 3× the file size. The Mamba2 layers use fixed recurrent state instead of growing KV cache, so only 4 attention layers contribute to KV growth.

The catch: its decode is slow (71 t/s) and its reasoning quality is low. But for the specific workload pattern of "huge context, short output" — which is exactly what agentic tool use looks like — the prefill scaling advantage is real and measurable. If someone trains a good model on this architecture, it could be a serious agentic contender.

---

Finding 4: F16 KV cache can be FASTER than Q8/Q4 quantized KV — the dequantization paradox

Conventional wisdom says quantize your KV cache (Q8_0 K / Q4_0 V) for speed — smaller cache, less bandwidth. I tested this head-to-head at 65K context, comparing F16 baseline against Q8_0 K / Q8_0 V (the results were identical to Q8K/Q4V within ±1% — V cache quantization choice turned out to be irrelevant):

Model	Type	Q8K/Q8V	F16	F16 advantage
Gemma-4-26B-A4B	MoE	1015	1554	+53%
Gemma-4-12B	Dense (7GB)	593	857	+44%
Qwen3.6-35B-A3B	MoE	1273	1573	+24%
Ornith-9B / Qwen3.5-9B	Dense (6GB)	1276	1544	+21%
Trinity-Mini	MoE	1429	1583	+11%
Granite-H-Small	Mamba2	1040	984	-5%
Ministral-3-14B	Dense (8KV)	409	335	-18%
Granite-4.1-8B	Dense (8KV)	447	358	-20%
Apriel-1.6-15B	Dense (8KV)	351	120	-66%

F16 wins for MoE models and small dense models. It loses badly for dense models with many KV heads. Here's why:

The Q8/Q4 KV cache dequantization is a compute operation that scales with context length. At 65K context, the attention kernel has to dequantize 65K × n_kv_heads × head_dim Q8/Q4 elements per token. This compute cost exceeds the bandwidth saved by halving the cache size.

Meanwhile, F16 KV doubles the cache but requires zero dequantization. For MoE models, the larger F16 cache causes GTT spill — but only the active parameters (~3B of 35B) cross PCIe, so the spill penalty is small. For dense models with 8 KV heads, the F16 cache is both larger and all weights spill — double penalty.

Updated rule:

• F16 wins: MoE models (tiny active spill footprint), small dense models (<10GB), efficient-GQA dense (4 KV heads)
• F16 loses: Dense + 8+ KV heads + >10GB (full weight spill + large KV = catastrophic)
• Q8K/Q4V vs Q8K/Q8V: Complete wash (±1%) across every model. V cache quantization choice is irrelevant — pick whichever.

Test F16 on your hardware at your actual working context. The conventional wisdom isn't always right.

---

Finding 5: MLA (Multi-head Latent Attention) degrades hard on Vulkan as context grows

GLM-4.7-Flash (1 KV head, 576 dim — MLA architecture) showed a steep prefill degradation:

Context	pp (t/s)	vs pp512
pp512	1822	100%
pp4K	1054	58%
pp16K	358	20%
pp65K	crashed	---

That's an 80% drop from 512 to 16K context. The 65K benchmark test crashed (Vulkan DeviceLost). However — I want to be clear about what this data does and doesn't show. I didn't test any context sizes between 16K and 65K, so the exact crash boundary is unknown. I'm also running this same model in daily use at 20K+ context without crashes, so it's not a hard wall at 16K — the failure point is somewhere above that.

What is clear from the data: MLA's prefill scaling on Vulkan is dramatically worse than standard attention. Whether that becomes a hard crash or just very slow depends on context size and VRAM headroom. The 66% drop from pp4K to pp16K is measured and real — MLA's KV compression/decompression kernel appears to scale poorly with context on the Vulkan backend. Don't extrapolate short-context MLA benchmarks to long context on Vulkan, and don't assume the degradation is linear — it looks like it accelerates. This may be backend-specific — CUDA or Metal may handle MLA's attention pattern better.

---

Finding 6: MoE models win the speed × intelligence composite for agentic work

Combining the speed data with Artificial Analysis Intelligence Index v4.1 scores into a composite (intelligence weighted by inverse wall-clock time). Score = AA Intel × (50s / wall_clock_at_65K), so 50s wall = 1.0× AA multiplier.

Rank	Model	AA Intel	Size	Wall (65K)	Type	Composite
1	Qwen3.6-35B-A3B	32	18G	54s	MoE	29.4
2	Trinity-Mini	24	16G	48s	MoE	24.9
3	Gemma-4-26B-A4B	26	14G	67s	MoE	19.6
4	North-Mini-Code	21	15G	75s	MoE	14.0
5	Ornith-9B / Qwen3.5-9B	15	6G	55s	Dense	13.7
6	Qwen3.6-27B	37	16G	171s	Dense	10.8
7	Gemma-4-12B	18	7G	115s	Dense	7.8
8	Granite-4.0-H-Small	7	17G	67s	Mamba2	5.2
9	Apriel-1.6-15B	19	9G	194s	Dense	4.9
10	Ministral-3-14B	15	8G	167s	Dense	4.5
11	Granite-4.1-8B	12	5G	152s	Dense	4.0
12	Devstral-24B	17	15G	217s	Dense	3.9

GLM-4.7-Flash excluded — no 65K data (MLA crash).

The smartest model (Qwen3.6-27B, AA=37) finishes 6th because it takes 3× longer than the MoE models. The top MoE (Qwen3.6-35B-A3B, AA=32) delivers 87% of the intelligence at 3× the speed. For agentic workflows where the model makes multiple short-output iterations, that tradeoff wins.

MoE models benefit twice: fast decode (read only active params) and cheap GTT spill (only active weights cross PCIe when context forces spillover). The combination makes them disproportionately strong at long context on constrained VRAM.

---

TL;DR — Practical takeaways for local agentic LLM deployment

1. Stop obsessing over tg128. At 65K+ context, prefill is 94–99% of wall-clock for short outputs. Benchmark pp65K / pp131K instead.
2. Check KV head count first. 4 KV heads × 128 dim (64 KB/token) scales dramatically better than 8 KV heads × 128 dim (160 KB/token) at long context — regardless of model size.
3. Test F16 KV cache at your working context. It can be 20–53% faster than Q8/Q4 for MoE and small dense models. The conventional "always quantize KV" wisdom breaks at long context on some architectures.
4. MoE models are the sweet spot for agentic long-context on consumer VRAM. Cheap GTT spill + fast decode + competitive intelligence. A 30B-class MoE delivers 80%+ of top dense model intelligence at 2–3× the speed.
5. Mamba2 hybrids are real. Near-flat prefill scaling to 131K. Currently let down by training quality, not architecture. Worth watching.
6. MLA degrades hard on Vulkan as context grows. GLM-4.7-Flash lost 80% of prefill speed from 512 to 16K context and crashed at 65K. It's usable at moderate context (I run it daily at 20K+), but the scaling curve is steep and non-linear. Don't extrapolate short-context MLA benchmarks.

---

All data collected on llama.cpp build 9860, Vulkan backend, RX 7900 XT 20GB. Your mileage will vary on other hardware — especially the F16 KV findings, which depend on your GPU's bandwidth vs compute ratio and your VRAM headroom. The architectural patterns (KV heads matter, prefill dominates, MoE scales better) should generalize.

Happy to share the raw JSONL or the benchmark script if anyone wants to reproduce.

I got Llama.cpp working on Haiku again, albeit with some caveats. Goes along great with my other projects HaiCode and HaiClaude. I have only tested it on the 64-bit Haiku, but if I get it working on the 32-bit hybrid image I will put it in an edit. Thanks go out to 3D Eyes on the Haiku discussion forum who did the original patch. While not all of it is relevant anymore, it was helpful for getting started.

So, after Anthropic breaking Claude Code for all future versions, I have decided to make an agentic coding agent just for us Haiku users. This is still very much a work in progress, and the TUI version is mostly broken, but hop on and enjoy the ride. BTW, fully expect to get downvoted to hell for posting something vaguely AI related. I would be pleasantly surprised if this was not the case. Enjoy!