r/unsloth

I wanted to share my recent experience and benchmarks

deploying the **Qwen3.6-27B-MTP-GGUF (Q8_0)** model locally

using speculative decoding on a dual-GPU budget enterprise

setup: **2x AMD Instinct MI50 (32GB HBM2 each, gfx906

architecture)**.

If you are looking for a cost-effective way to host 27B+

models locally with fast generation speeds, older AMD

enterprise cards (like the MI50) are absolute hidden

gems—though they do require some ROCm tuning to hit their

maximum potential.

Here is a full breakdown of the performance, the bottleneck

I ran into, and how I optimized the prefill (Prompt

Processing) by **over 50%**!

---

### 💻 Hardware & Env Specs

* **GPUs**: 2x AMD Instinct MI50 32GB HBM2 (PCIe Gen3 x16)

* **CPU**: Intel Xeon E5-2696 v4 (22C/44T)

* **RAM**: 64GB DDR4

* **Backend**: llama.cpp (inside Docker with ROCm 7.2.3,

custom compiled for gfx906)

* **Model**: `unsloth/Qwen3.6-27B-MTP-GGUF` (Q8_0 weights,

~28GB) loaded fully onto 2x GPUs via layer-split.

* **Speculative Decoding**: Native `draft-mtp` enabled with

`--spec-draft-n-max 2 -np 1`.

---

### 📊 Real-world Generation Speed & MTP Acceptance Rate

The generation speed is incredibly snappy thanks to

Unsloth's MTP architecture. Here are the speculative decoding

stats captured from a real long-context conversation:

* **Generation Speed (Decode)**: **27.96 - 28.57

tokens/sec**

* **Draft Acceptance Rate**: **76.6%** (489 accepted out of

638 generated drafts!)

* **VRAM footprint**: Super clean **~42% (13.7 GB)** VRAM

usage per GPU with a **64k context window** (`-c 65536`),

leaving tons of headroom.

Here is the raw stdout log of the inference run:

```text

24.17.528.975 I slot print_timing: id 0 | task 2484 |

eval time = 28897.12 ms / 808 tokens ( 35.76 ms per

token, 27.96 tokens per second)

24.17.528.976 I slot print_timing: id 0 | task 2484 |

total time = 135079.97 ms / 19574 tokens

24.17.528.977 I slot print_timing: id 0 | task 2484 |

graphs reused = 2955

24.17.528.978 I slot print_timing: id 0 | task 2484 |

draft acceptance = 0.76646 ( 489 accepted / 638 generated)

24.17.528.990 I statistics draft-mtp: #calls(b,g,a)

= 12 2993 2993, #gen drafts = 2993, #acc drafts =

2562, #gen tokens = 5985, #acc tokens = 4642

──────

### 🔍 Troubleshooting the Prefill (Prompt Processing)

Bottleneck

Initially, while generation was fast, my prefill speed was

painfully slow, hovering around 176.7 tokens/second. A ~18k

token prompt was taking 106 seconds just to prefill!

By digging into the logs and running llama-bench , I found

two main culprits:

1. Physical Batch Size Underutilization: The default --

ubatch-size 512 is too small to saturate the massive 3,840

stream processors on the Vega-based MI50 architecture.

2. PCIe Checkpoint Synchronizations: The llama.cpp server's

default context checkpointing ( --checkpoint-every-n-tokens

8192 ) was triggering a ~250MB state copy from VRAM to host

RAM over PCIe Gen3 every 8192 tokens. This forced the entire

GPU pipeline to stall and sync for 1.01 seconds per

checkpoint!

#### 🚀 The Fix:

I adjusted the startup parameters in my launching script to:

1. Max out the physical batch size: -b 2048 --ubatch-size

2048 to saturate the GPU.

2. Disable the PCIe context checkpoints: --checkpoint-every-

n-tokens -1 (since this is a sequential Claude CLI setup,

intermediate checkpoint rolling back is unnecessary).

3. Set -c 65536 to lower the overall KV-Cache footprint for

safety.

──────

### 📈 The Benchmark Results (Optimized vs Default)

Here are the actual實机 llama-bench results under different

physical micro-batch ( ubatch-size ) configurations on my

dual

MI50 server:

Prompt Size│ ubatch-size│ ubatch-size│ ubatch-size│ Speed Im

────────────┼────────────┼────────────┼────────────┼─────────

512 Tokens │ 196.47 t/s │ 264.44 t/s │ 271.74 t/s │ + 38.3%

2048 Tokens│ 194.20 t/s │ 273.71 t/s │ 290.00 t/s │ + 49.3%

8192 Tokens│ 195.12 t/s │ 279.79 t/s │ 295.66 t/s │ + 51.5%

Note: Disabling the checkpoints also completely eliminated

the 1-second stalls, meaning the end-to-end prefill for an

18k

token payload now finishes in ~63 seconds instead of 106

seconds!

### 💡 Takeaways

• MTP speculative decoding works incredibly well on ROCm with

llama-server . Over 76% acceptance rate makes a 27B model

feel like a much smaller model during generation.

• If you deploy ROCm on older architectures

(gfx906/MI50/Radeon VII), do not stick with default ubatch

sizes. Cranking up --ubatch-size 2048 coupled with --flash-

attn on yields massive prefill speedups.

• Watch out for context checkpoint overhead if you are using

long contexts over slower PCIe buses.

Big thanks to the Unsloth team for making these MTP weights

available! Let me know if you have any questions about ROCm

configurations or MI50 setup.

Platform: Ubuntu 26.04, RTX 5060Ti, NVIDIA Driver 595.71.05, CUDA 13.2

I downloaded the IQ3XXS version and tried to run it with llamacpp both ways - with and without the newly introduced spec-type argument. But in both the cases, the model produces random characters as output. Here's one of the commands that I used:

./llama-server -m ~/.lmstudio/models/unsloth/Qwen3.6-27B-MTP-GGUF/Qwen3.6-27B-UD-IQ3_XXS.gguf --no-mmproj -ngl 99 -c 8192 -np 1

I am able to run the regular models but not this one. Am I missing something or this quant has a problem?

Sample output:

1   .111 111111111 1049  . 21 1. 1.111 10 1.  .​1

11​..​ .  A .

Update: This is a known issue where compiling llamacpp against CUDA13.2 makes it produce garbage with all variants (MTP/Non MTP). Downgrading to CUDA12.8 solved it. However that isn't simple on Ubuntu 26.04.

I recently upgraded to Ubuntu 26.04 because of NPU support. However, the recommended CUDA toolkit with 26.04 is 13.2. I managed to install the older versions of the toolkit using deb installers on nvidia site but llamacpp compilation fails because of glibc incompatibility. Eventually had to use docker and setup nvidia/cuda:12.8.0-devel-ubuntu24.04 for compilation.

Now happy with a jump from 30tps to 40tps :)

I’m running Hermes Agent on a single NVIDIA DGX Spark using vLLM with:

docker run --gpus all \
--name qwen36-aggressive \
--restart unless-stopped \
-p 8000:8000 \
--ipc=host \
--ulimit memlock=-1 \
--ulimit stack=67108864 \
--shm-size=32g \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-e VLLM_ATTENTION_BACKEND=FLASHINFER \
-e FLASHINFER_DISABLE_VERSION_CHECK=1 \
-e VLLM_HTTP_TIMEOUT_KEEP_ALIVE=600 \
vllm/vllm-openai:cu130-nightly \
--model Qwen/Qwen3.6-35B-A3B-FP8 \
--served-model-name qwen36 \
--host 0.0.0.0 \
--port 8000 \
--tensor-parallel-size 1 \
--gpu-memory-utilization 0.75 \
--dtype auto \
--kv-cache-dtype fp8 \
--max-model-len 262144 \
--max-num-batched-tokens 32768 \
--max-num-seqs 4 \
--attention-backend flashinfer \
--enable-prefix-caching \
--enable-chunked-prefill \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder \
--trust-remote-code \
--reasoning-parser qwen3 \
--performance-mode throughput \
--default-chat-template-kwargs '{"preserve_thinking":true}' \
--override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}' \
--speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}'

It boots successfully and seems stable so far, but I’d love opinions from people running similar long-context / agentic setups.

Any feedback or suggestions are welcome.

I'm on a shared DGX-H200 that unfortunately does not have current gcc-toolset-13, and I cannot update it. When I build my local version of llama.cpp I have to set cmake flags to disable all AVX instructions. I'm installing unsloth-studio via the single script:

curl -fsSL https://unsloth.ai/install.sh | sh

How do I modify the build config of the llama.cpp within the install ?

quantize.imatrix.file Qwen3.6-35B-A3B-GGUF/imatrix_unsloth.gguf
quantize.imatrix.dataset unsloth_calibration_Qwen3.6-35B-A3B.txt

I find the gguf but not the unsloth_calibration_Qwen3.6-35B-A3B.txt file. Same for other LLMs.

For agent models, PinchBench and Tau2 may matter more than one more AIME headline。

I still think AIME and GPQA matter. They say something real about capability ceilings. For agent models, though, I reach first for execution-heavy, tool-heavy, multi-step signals. That is why Ring-2.6-1T caught my eye: PinchBench: 87.60, Tau2-Bench Telecom: 95.32, and ClawEval: 63.82 sit alongside AIME 26: 95.83, GPQA Diamond: 88.27, and ARC-AGI-V2: 66.18. For production-style agents, I care first about whether the model can keep a workflow moving, coordinate tools cleanly, and avoid spending deep reasoning on every intermediate step. The public high / xhigh framing fits that story too, with deeper reasoning available when you need it instead of dominating every path.

unsloth studio를 업데이트 한 이후부터 프롬프트 입력에서 한글 입력이 전혀 안된다.
운영 환경은 mac mini

unsloth studio의 버그야? 아니면 한글 같은 다국어 입력 문제 해결 방법이 있나??

The Unsloth team has been truly amazing in breadth and depth of releases. I’m super excited to try the 27b in particular.

The Qwen3 CoderNext model has actually surprised me in functionality when thinking is less valuable, like feeding Aider.

I would be grateful if this got MTP turned on with Unsloth’s high quality quants!!

Anyone else a fan?

I finally got around to installing Unsloth Studio and am giving it a whirl on my Windows machine. I have an RTX6k and am trying to load the latest QWEN MTP model you guys created. However no matter what I do, all models are loading in RAM instead of the GPU.

From what i see online it means there's a cuda driver mismatch most likely. But i installed 12.9 and updated all my environment variables. So I'm not sure why this is still happening.

Any thoughts or help?