I am wondering if I should buy an extra GPU for models like Qwen 3.8 27B, e.g. an R9700 or intel arc one. What are the pros and cons vs a second strix halo (I am on Bosgame M5) vs just running one solo?
Context: Currently I mostly use Qwen 3.5 122B Q4 Unsloth which isn’t bad at all but I still have to use, via cloud, DS4F or Luna a lot currently since those are stronger (won’t fail the task at hand, mostly C++) and are much much faster.
I only get the chance to play few games per week and the old calibration wouldn’t let me get enough rank confidence for a medal. Is the new one any faster or requires less games?
Hey I am running a few models on a strix halo box. Especially for the larger models (like Qwen 3.5 122B) they work okayish performance wise if the cache is utilised properly but a full cache miss at 100k context causes roughly 10-20 minute of PP time - which is extremely annoying.
I will first show what I have already configured (and it helps!), then describe what is still not working well. I am very interested on your input what I could still fine tune.
What I've configured so far (and what I understand it does):
It works a lot better than stock configuration this way, but I still find problematic:
I am hoping for some input from you guys how to better set up the cache and maybe work around the other described issues somehow!
Comparison was done on Strix Halo 128gb shared memory, Ubuntu 26.04, Lemonade Server, Vulkan backend.
I often run larger models like gpt-oss 120B or qwen but their performance seems to degrate quickly once in deep waters... ah.. deep context. The most important quality to me is prompt processing - we are talking existing code and context quickly fills up when analyzing it for a change request / bugfix. In existing code, I think 95-99% is PP and 1-5% is TG of the total time. I tried Nemotron Super (120B) recently and liked the quality, speed was decent but to my surprise I felt it handled deeper context (~100k) way better than what I am used to with similar models. To falsify that subjective impression, ran llama-bench with the three competitors in the 120B class (GPT-OSS, qwen 3.5, and Nemotron) and, mostly as a comparison, the popular smaller/weaker/faster Qwen 3.6 35B model. As a subjective baseline I set 100 TPS PP as "usable" and stopped the benchmark if the model fell below it. Also, I should mention that the max context varies by model: GPT-OSS can handle max ~128K, Qwen 3.5/6 can handle ~256K, but Nemotron up to 400k Tokens context depth.
My main conclusions are: My feeling was right, Nemotron Super handles deep context exceptionally well, compared to the others. The "speed king" GPT-OSS 120B looses speed so fast that Nemotron Super surpasses it in PP at 32K depth. QWEN 3.5 122B A10B is surpassed almost immediatelly at 16K depth. Even Qwen 3.6 35B A3B's PP is on par at the model's max context of ~256k context, surprisingly.
At token generation speed (IMO not as important), Nemotron Super starts usable (IMO >~10 TG TPS) but not yet really "fun" (IMO >~20 TG TPS) to use. It degrates slowly to "barely usable" according to that definition at ~400k context depth - which is stll impressive if you ask me. The most direct competitor Qwen 3.5 122B A10B is about as slow at 128k context. Note that I didn't enable MTP, though.
If you need high TG, Nemotron is not the best model for context below 128k; if you mainly need PP and a larger model, Nemotron seems a reasonable choice. The fallback if you don't need that large a model is obviously the smaller Qwen 3.6 variants like 35B.
Has anyone different results? Maybe with rocm? Any tweaking I didn't consider?