r/Qwen_AI

llama.cpp pr #27342 adds dflash2, so i rented an rtx 6000 and ran the same four prompts through four decoding setups on qwen3.8 27B

median results over the four tasks:

• baseline 47.4 tok/s
• mtp 114.7 tok/s
• dflash 99.3 tok/s
• dflash2 140.6. tok/s

so on average 3x for dflash2

though i have to point out that it's far from a 3x gain some of the time, on one of the test it struggled to achieve a 1.5x gain, it really just depends on the task you give to the model

>the races are sped up in some places, so that the video lasts roughly 30 seconds, but the tok/s and acceptance % on screen are the real

i'm from the atomic.chat team - we publish our own quants on hf and make a desktop and mobile app for running local models. so any feedback welcome - we're building this for you folks

about dflash2: https://inco.ai/blog/dflash2/

Just got Qwen3.8-27B (FP8) running on rented GPUs from Trooper AI. FP8 fits on 64 GB VRAM with decent results.

Stack:

- GPU: 2× RTX 4080 Super Pro (64 GB VRAM total)

- CPU: 12 P-cores, 76 GB RAM

- SSD: 900 GB NVMe

- Price: 1.06 EUR/h

Served it via vLLM → KServe → Envoy AI Gateway, TLS + token metering + rate limiting on top ( I already have a running Kubernetes cluster, I attached to trooper GPU node), or you can it serve it directly with compose if you are a single user.

Runned load tests: (10 concurrent requests, 32768 context)

- TTFT: ~0.9s

- Per-stream decode: ~28 tok/s

- Aggregate: 152 tok/s

Full deploy guide if you want to deploy it: https://github.com/redaER7/qwen3.8-27b-self-hosted

Now, looking to deploy the full model FP16 on RTX 6000 Pro

I’ve been testing Qwen 3.8 27B as a long-running coding agent on my RTX 5090.

I used GPT-5.6 Sol to help write a short plan md for FloodLayer, then let Qwen execute the build in DeepSeek Harness.

FloodLayer is a small 3D AEC sandbox where water follows the actual floor slope, moves toward drains, pools, and can escape through thresholds.

What finally worked well for me was 131K context, full GPU offload with ngl 99, Flash Attention, Q8 KV cache, parallel 1, and MTP with draft max 2.

In DSH I set contextWindow to 131072 and maxTokens to 16384.

Loading the model directly instead of using router/preset mode was also much more stable for me.

Without MTP I was getting around 54 tok/s. With MTP I’m seeing roughly 70–100 tok/s depending on context length and draft acceptance.

Auto-compaction is working now too, so it can keep going for much longer without constantly needing manual continue.

Still testing the long-run behavior, but this is the first setup where local Qwen genuinely feels useful as a serious coding agent.

Okay, I have to admit I’m genuinely blown away. I absolutely did not expect

Qwen 3.8 27B Q4_K_M with MTP, capped at 64k context, running on a single RTX 3090 with the OpenCode harness.

A single prompt:

“Build a realistic ocean water scene with lighting, etc. using Three.js or another engine if more appropriate.”

20m 34s of work, and this is the result (with controls added as a bonus).

I don’t know what you guys think, but I find it pretty incredible that my single GPU managed to generate something like this in one shot. Not that long ago, only the best commercial models could reach this kind of quality in such a short amount of time.

I know for a fact Qwen3-Coder and Qwen 3.6 27B weren’t giving me results like this. This is clearly a step above.

Feel free to ask if you have any questions!

My full hardware setup for those wondering:

- CPU: AMD Ryzen 9 9950X
- GPU: RTX 3090 FE 24GB (Gelid thermal pads)
- RAM: 64GB DDR5 (2×32GB) Corsair CL30 @ 6000 MHz
- Motherboard: Gigabyte B650 DS3H
- CPU Cooler: Thermalright Peerless Assassin 120 SE
- SSD: 500GB M.2
- PSU: Corsair RM1000x
- Case: ASUS Prime AP201 Mesh

What does this even mean? Probably nothing.

Disclosure first: I'm one of the developers of Paddock (Truespar). It's a new inference engine, free for individuals and companies with no usage limits, and these are our own published benchmarks, so be skeptical and reproduce them if you can.

Numbers from the public bench (RTX PRO 6000, byte-identical official FP8 checkpoint, spec decoding off everywhere, max batch 32, ctx 8192).

Throughput, tok/s at 1 / 8 / 32 concurrent clients:

• Paddock (FP8): 47.7 / 334.1 / 968.1
• vLLM (FP8): 45.0 / 321.1 / 968.1
• SGLang (FP8): 45.0 / 321.9 / 835.0
• llama.cpp (Q8_0): 33.7 / 64.9 / 239.1

Time to first token at 32 clients:

• Paddock: 697 ms
• SGLang: 1.9 s
• vLLM: 2.5 s
• llama.cpp: 6.9 s

Full methodology: https://truespar.com/paddock/benchmarks/qwen36-27b. Paddock exposes a plain OpenAI endpoint, so you can throw NVIDIA's aiperf at it on your own hardware.

Since the Qwen3.8-27B drop we've been optimizing kernels for the whole 3.x generation, and 3.8 runs today: if you have a supported Blackwell or Ampere card (list in the docs) I'd love to see your 3.8 numbers before we publish our own bench for it.

Two things beyond raw speed, since the built-in Studio is half the product: drop a document on it and it runs an integrated Docling-style pipeline (segmentation, tables, reading order) plus tag-based extraction with Granite 4.1 Vision, tables to JSON/CSV straight from page images, dozens of pages in a single inference with Unlimited-OCR. And images get EXIF/metadata enrichment before the model sees them, so a local Qwen knows when/where a photo was taken without any tool calls. You can also A/B a local model against OpenRouter models side by side.

One dependency-free download (Windows/Linux x64, NVIDIA driver 580+), no Python stack. Download + docs: https://truespar.com/blog/introducing-paddock

I'll be in the comments

Qwen released the 2.4T Max weights and I was curious how well it can re-create COD in one prompt

I ran the model on a rented B200 cluster and used roughly 1.1M output tokens over a 5 hour time span

Realistically barely anyone can run this model locally, but the fact that it's open weights unlocks a lot of opportunities for the local AI community even regarding quantization options that might bring us closer to running frontier level intelligence on consumer-ish hardware 

For consumer hardware there is a 27B model from the same release and it's really capable for its size, many people have posted 3D games made with it on X

I'm from the atomic.chat team, so If you wanna try running the new Qwen locally you can do so inside our app (any feedback is appreciated, since we want to make our product as good as possible for you guys) 

We have our own and other community quants of the 27B model available to download directly inside the app

P.S. I am a noob in shooters so pls no hate my low skill gameplay

What does this mean? Is there something else coming? Maybe 122B? Or no models?

New Row-Bot demo: turning your inbox into an action plan.

​

Row-Bot checks important emails, finds action items, drafts replies, creates calendar events, and schedules reminders, with approvals for sensitive actions.

​

Not just chat. Real workflow automation.

​

[https://github.com/siddsachar/row-bot\](https://github.com/siddsachar/row-bot)

This is the first model after 3.6 which was even able to register the Sharingan MCP which has basically most of the popular cyber security tools and take this is built based on Johnrizzo1 reverser and Kali linux. This was the first time running this on qwen code 3.827B and it was able to use all of the tools and also able to use it efficiently

Hi,

After a long night of optimizations, I believe I have made the fastest inference engine for Qwen3.6-28B on a 3090.

Quick metrics:

- 250w power capped

- Up to 195k context (ships with 150k for safety though)

- 82 tps single request, 417 tps sustained with 64 concurrent

- Between 17% to 149% faster than ninfer depending on the amount of concurrent requests.

Quick how:

- W4A16 quantization -> 16.8gb in vram - cache 66k

- + fp8 KV cache -> 16.8 gb in vram - cache 155k

- + lm_head int8 -> 15.4 gb in VRAM - cache 192k

- + embed_tokens int8 -> 14.2 gb in VRAM - cache 200k

Quantization loss of 0.6% in the lm head and quant embed compared to bf16.

It runs via vLLM and needs a few patches to work perfectly, but should be easier to setup than ninfer.

Also only tested on linux, but should work on windows too.

https://github.com/syv-ai/qwen38-27b-rtx3090