I'm looking for the best open-weight LLM I can realistically run locally for OpenCode, with a target of at least 40+ output tokens/sec while using the model's full context window.

My workstation:

• AMD Threadripper PRO 9985WX, 64C/128T
• 512 GB DDR5-5600 ECC RDIMM, 8-channel
• 2× NVIDIA RTX PRO 6000 Blackwell 96 GB (192 GB total VRAM)
  • 1× Workstation Edition
  • 1× Workstation Max-Q
• ASUS Pro WS WRX90E-SAGE SE
• Linux
• Mainly using llama.cpp / LM Studio, but I'm also open to vLLM or SGLang if they make more sense

The main use case is agentic coding through OpenCode, so I'm prioritizing coding ability, tool use, long-context reliability, instruction following, and avoiding repetition/loops.

I'm fine with FP8, NVFP4/MXFP4, GGUF, etc. if the quality trade-off is reasonable. The model does not necessarily have to fit entirely in VRAM; CPU/RAM offloading is also an option, but I still want 40+ tok/s generation speed at full context.

I'm basically looking for the smartest model this machine can run at that speed, rather than the fastest small model.

What would you pick today?

I'd especially appreciate actual benchmarks from similar dual-96GB Blackwell setups rather than theoretical estimates.

I'm working on setting up Qwen 3.8 27B on my 4x3090 rig. I mostly used the config from Club3090 as a guideline.

One result I didn't expect was MTP significantly hurting performance at long context. My main usages is Hermes agent and if this is correct it suggests I should disable MTP all together or perhaps my settings are not optimal.

Benchmarks below - any thoughts?

Qwen3.8-27B-FP8 Comparative Benchmark Matrix

Speculative Setting |Short Prompt Decode (tok/s) |Short TTFT (s) |72k Long Prompt Decode (tok/s) |72k Long Warm TTFT (s) |72k Concurrency-2 Wall Time (s) |Steady VRAM / GPU
No MTP (Disabled) |67.5 |0.121 |57.3 |0.532 |15.25 |~20.5 GiB
MTP = 1 |80.5 |0.121 |15.6 |1.119 |48.05 |~20.7 GiB
MTP = 3 |106.3 |0.135 |22.6 |1.142 |32.08 |~21.2 GiB

I've been testing Qwen3.8-27B Q8 on a Ryzen AI Max+ 395 / Strix Halo system through Lemonade + llama.cpp, specifically to see whether the new MTP speculative decoding support actually improves generation speed.

I kept the prompt and output length identical between runs:

• 96 input tokens
• 1024 output tokens
• Temperature 0
• Same Qwen3.8-27B model/quant
• Flash Attention enabled
• --no-mmap
• Only backend/MTP settings changed

These are the results so far:

Backend	MTP setting	Generation speed	TTFT
Vulkan	Off	9.159 tok/s	0.758 s
Vulkan	n-max=1	6.579 tok/s	0.950 s
Vulkan	n-max=3	7.122 tok/s	0.764 s
ROCm	Off	6.534 tok/s	0.715 s
ROCm	n-max=3	4.689 tok/s	0.688 s

So on my machine:

• Vulkan + MTP n=3 is about 22% slower than Vulkan without MTP.
• Vulkan + MTP n=1 is about 28% slower.
• ROCm itself is about 29% slower than Vulkan without MTP.
• ROCm + MTP n=3 drops another ~28% versus ROCm without MTP.
• Overall, Vulkan without MTP is almost 2x the generation throughput of ROCm + MTP in this test.

For MTP I'm loading llama.cpp with:

--spec-type draft-mtp --spec-draft-n-max 3

(and also tested n-max=1 on Vulkan).

For ROCm, I'm using Lemonade's current stable ROCm backend. Lemonade reports the llama.cpp backend as b10397; the bundled ROCm/TheRock stack appears to be ROCm 7.13.x. I haven't tested ROCm 7.14 yet.

It's surprising to see that with MTP there's a pretty substantial regression on both Vulkan and ROCm.

I'd be interested to compare with other Strix Halo owners:

1. Are you seeing MTP actually improve Qwen3.8 throughput?
2. What --spec-draft-n-max value works best for you?
3. Are you using Vulkan or ROCm?
4. Which ROCm version / llama.cpp build?
5. Does ROCm 7.14 materially improve Strix Halo performance versus 7.13?
6. What Qwen3.8-27B quant are you using?
7. If you're getting a significant MTP speedup, what kind of draft acceptance rate are you seeing?

I'm mainly trying to figure out whether these numbers are normal for the current llama.cpp MTP implementation on Strix Halo, or whether something is wrong with my setup.

At least with my current stack, Vulkan with MTP disabled is very clearly the fastest configuration I've tested.

5090's used price been skyrocketing, then there's taobao sellers claiming to have the MI210 for just under RMB 20,000 ($3k) Is the MI210 good or it has to be bridged with quad cards to actually see some benefits?

I’m looking at the NVIDIA DGX Spark and ASUS Ascent GX10, both built around the GB10 platform, and I’m curious which one people would actually choose for a serious local AI setup.
Main workloads would be:

Local LLM inference
Fine-tuning / experimentation
Coding agents
Running multiple AI services locally
General development and research
If price were reasonably close, which would you pick?
I’m especially interested in real-world differences around thermals, noise, sustained performance, software/support, reliability, and overall value.
Would you go DGX Spark because it’s the NVIDIA reference system, or ASUS Ascent GX10 for another reason?

View Poll

Weights dropped today, Unsloth GGUFs followed right after. Here are numbers from a 128GB Strix Halo box. Vulkan/RADV, flash attention on, batch 2048/512, depths 0 and 32k, two runs per cell, drift check at the end was under 0.8 percent.

It loads on existing llama.cpp builds. Architecture string is the same as Qwen 3.6 (qwen35), 65 layers, 4 KV heads at dim 256, 262k native context. No update needed.

Weight quants, f16 KV cache:

Weights	Prefill d0 (t/s)	Gen d0 (t/s)	Prefill 32k (t/s)	Gen 32k (t/s)
UD-Q5_K_XL (20 GB)	347.0	10.5	252.4	9.4
Q8_0 (29 GB)	340.3	7.5	249.5	6.9

Compared to Qwen 3.6 27B in the same Q5 quant on the same build: within 2.5 percent everywhere. So switching costs nothing in throughput.

The Q8 row is there because someone asked for it under my last post. Q8 generates 29 percent slower than Q5 at depth 0 and 27 percent at 32k. That matches the file sizes: 29 vs 20 GB is a factor of 1.44, the decode gap is 1.41. Dense model, reads all weights every token, so generation scales with file size.

KV cache quantization, measured at 32k (at depth 0 all three KV types are identical to within half a percent):

Weights	KV cache	Prefill 32k (t/s)	Gen 32k (t/s)
UD-Q5_K_XL	f16	252.4	9.4
UD-Q5_K_XL	q8_0	250.9	9.9
UD-Q5_K_XL	q4_0	255.2	10.0
Q8_0	f16	249.5	6.9
Q8_0	q8_0	247.0	7.1
Q8_0	q4_0	243.9	7.2

Same pattern as on 3.6: prefill within 2 percent, generation gains 4 to 6 percent with quantized cache. Matches what I measured across nine models recently.

Both GGUFs have the MTP tensors, I haven't tested speculative decoding yet. Caveats: community fork build (10283, strix-halo-vulkan branch), throughput only. No idea yet whether 3.8 is actually better than 3.6 at anything.

If you have numbers on other hardware, especially the Q8 vs Q5 gap on something with more bandwidth, post them with your build commit.

Hi All,

Just joined. Absolute novice when it comes to ‘AI’ and its integration with hardware / devices. Can anyone in this sub point me towards courses / uni’s that are highly regarded in this field ?

I did some testing this morning, and I was surprised to find that Muse Glimmer actually comfortably fits on a single RTX 3090 with full context + DFlash + mmproj at Q4_K_XL, unlike Qwen3.6-27B and Gemma-4-31B.

Muse Glimmer supports up to 256k context according to Unsloth. Here is my command:

llama-server \
  --model Muse-Glimmer-30B-UD-Q4_K_XL.gguf \
  --mmproj Muse-Glimmer-30B-mmproj-kquant.gguf \
  --spec-draft-model Muse-Glimmer-30B-DFlash-kquant.gguf \
  --spec-draft-ngl 999 \
  --spec-draft-n-max 15 \
  --spec-type draft-dflash \
  -c 262144 \
  --override-kv muse-glimmer.context_length=int:262144,dflash.context_length=int:262144 \
  -ngl 999 \
  -fit off \
  --parallel 1 \
  --flash-attn on \
  --no-warmup \
  --cache-type-k f16 \
  --cache-type-v f16 \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 64 \
  --reasoning-preserve \
  --jinja \
  --host 127.0.0.1 \
  --port 8080

This fits in about 22GB to 23GB of VRAM, actually leaving a reasonable amount of unused memory.

On this RTX 3090, for Qwen3.6-27B and Gemma-4-31B, this is what I've been able to achieve using their Q4_K_XL models with MTP + mmproj, right at the limits of the RTX 3090's VRAM:

Model	F16 KV cache	Q8 KV cache
Qwen3.6-27B	70,000 tokens	125,000 tokens
Gemma-4-31B	52,000 tokens	81,000 tokens

Those small contexts have been borderline unusable on f16, and I don't enjoy using Q8 KV unless absolutely necessary, so I mostly use my slower DGX Spark to run these models at the full context.

On Muse Glimmer, there seems to be little reason to use my DGX Spark since it fits so nicely on the RTX 3090. Maybe I could run a bunch of parallel agents with full KV on the Spark.

Muse Glimmer also runs at between 64 tok/s and 124 tok/s in my testing under DFlash, depending on whether it is outputting prose or code. Either way, a pretty solid speed. I've seen about 1400 tok/s of prompt processing.

I also ran a two needle haystack test at about 150k tokens with one needle at the beginning and the other at the end, and the model retrieved them perfectly on the first try, so this is definitely not soft-capped to 128k context.

Hi! The current price of the RTX PRO 6000 is almost double what it was a year ago, and it’s now about 3× the price of a single DGX Spark.

I currently have one RTX PRO 6000 and was considering buying another one. But at the current price, I’m wondering whether I should sell my existing RTX PRO 6000, sell my PC as well, and put some extra money toward a setup with 8× DGX Spark plus a good switch.

Has anyone actually tried an 8× DGX Spark setup? I’d really appreciate some advice, especially regarding the largest models you can run at a usable speed when working with a codebase.

Someone with a Mac Studio with 500 GB of unified memory told me that they can only use models around 200 GB in size. Anything larger is basically unusable—you can chat with it, but waiting for it to scan and reason over a codebase takes forever. They said the main bottleneck isn’t token generation speed, but prompt processing speed (prefill).

For those who have experience with large unified-memory setups or multiple DGX Sparks, what has your experience been like? What’s the largest model you’ve found usable for coding?

So, I couldn’t resist joining the fun with Muse Glimmer. I am running on DGX Spark, which is notoriously slow for dense models, due to the low memory bandwidth.

I started the model with a very basic vllm setup and shot some prompts at it and got back the awesome 5-8 tok/s. However, I saw meta also published an assistant (dflash drafter) model alongside the main model.

Setup dflash with the assistant and started with 6 speculative tokens and pretty much doubled the generation speed (10-12 tok/s), even with a low acceptance rate.

Then I jumped to 12 speculative tokens and got in the 15-19 tok/s range, starting to become usable.

Played a bit more it seems 18 speculative tokens was causing too much drafting overhead so dropped down to 15 where I am currently and am getting 23ish tok/s.

Switched my hermes agent (which I use kind of like a personal assistant - managing calendar entries, reminders, tracking expenses, etc.) to the model and even though its slower than the 122b MoE Qwen I was running before it seems to be doing very well with tool calls and loading skills.

Will play around more with it and share the experience!

Edit:
Current vllm options:

--kv-cache-dtype fp8 \
--kv-cache-memory-bytes 4147483648 \
--max-num-batched-tokens 8196 \
--gpu-memory-utilization 0.92 \
--speculative-config '{"method":"dflash","num_speculative_tokens": 15, "model": "/models/muse-glimmer-dflash"}' \
--port 8000 \
--host 0.0.0.0 \
--load-format fastsafetensors \
--attention-backend triton_attn \
--enable-chunked-prefill \
--enable-auto-tool-choice \
--tool-call-parser muse_glimmer \
--reasoning-parser muse_glimmer \
--generation-config auto \
--override-generation-config '{"temperature": 0.7, "top_p": 0.8, "top_k": 20, "presence_penalty": 0.0, "repetition_penalty": 1.0}'

I'm curious what people are running into when using AI coding tools locally.

For me, the obvious issues seem to be GPU memory and system RAM, especially when running local models alongside an IDE, containers, databases, and other development tools.

At what point does hardware actually become the limiting factor?

Is it mostly VRAM, RAM, CPU performance, storage speed, or just having too many services running at once?

For those building AI development machines, what upgrade made the biggest practical difference to your workflow?

Our second blog post on writing NVFP4 blockscaled GEMM kernels for the NVIDIA RTX PRO 6000 Blackwell GPU is out! The blog iteratively optimizes a basic working NVFP4 GEMM kernel written in CuTe DSL to take it to speed-of-light, reaching over 80% TFLOP/s utilization for 16k square matrix shape. We give a detailed treatment of important optimization techniques such as threadblock swizzling, async and warp-specialized epilogue, and retiling for favorable wave quantization. Specific to blockscaled GEMM with scales consumed from registers, we also explain how to solve for bank conflicts that arise from the default choices of interleaved scale factor layouts.

We include complete code in the form of CuTe DSL kernels for all the optimizations discussed in the blog.

Hey everyone,

I'm building a workstation specifically to run local LLMs, perform local video generation and do 24/7 home lab stuff

My budget is ~$2.4k USD.

Here is the planned parts list:

• GPUs: 2x NVIDIA GeForce RTX 4060 Ti 16GB (Dual-Slot Twin Fan)
• CPU: Intel Core i5-14400F (10C / 16T)
• Motherboard: MSI B760 Gaming Plus WiFi ATX (Dual PCIe x16 slots)
• RAM: 128GB DDR5 (4x32GB) @ JEDEC 4800–5200MHz
• Storage: 2TB M.2 PCIe 4.0 NVMe SSD (7000 MB/s)
• PSU: 850W 80+ Gold ATX 3.1 Fully Modular
• Case: Montech AIR 903 MAX (High-Airflow ATX)
• Cooler: Thermalright Peerless Assassin 120 SE

Looking for advice on this.

Thanks!

Id love to get some feedback on it. Im by no means a writer so the grammar might not be perfect, but i do think it has quite some technical value to it.

If youre in ML or interested in AI Infra please give it a go; as i said i would love for some feedback.

Im not active on reddit, but my dms are always open on other platforms such as linkedin (gustavkeller).

Hi I am getting into ai image generation for making t-shirt and merch designs using a combination of Ai and imaging programs to make wallpaper, t-shirt and merch artwork and am currently studying on how much money I will have to save up to guild a Ai workstation. My requirements would be to generate images and upscale them to about 12" by 16" 300 dpi. I can also vectorize them. I am currently using Google Gemini for my images but need to upscale these images, I might try upscale. My end result will be to build a local AI image generation and upscaling workstation to achieve this. I am currently looking at using a Intel Ark B60 24GB vram or a B70 32Gb Vram GPU as they are the only GPU's that fall within the price range I can save up for over a period of about a year. Realistically I will have a limited budget and hope ram and memory prices will decrease by the time I am ready to build a workstation. I will continue to use Google Gemini unless someone can recommend a better option but want to move my Ai to local generation as it fits into my brand marketing plan which is generating the Ai locally. What are my best options to save up to build a Local Ai rig price wise?