Follow-up to my last thread.

Unfortunately GitHub closed many PRs after cleanup of repo's commit history. Niko's reply here for all PRs including this PR. So now we need to wait for new PR from closed PR's creator. Meanwhile I reminded Niko about this PR. His reply below.

>I will think about it!

So please just reply him there with your message, let him know that this has huge demand from many of us. I really want Strata to support many models(Qwen3.6-35B-A3B, Gemma-4-26B-A4B, Nemotron-3.5-Lightning-30B-A3B, Laguna-XS-2.1, North-Mini-Code-1.0, Kat-Coder-V2.5-Dev, etc.,) as much as possible, even experimental fine for me.

Expecting around 20 replies there. Consider it's a last chance to get it.

EDIT : I see new PR for same. Lets see

https://github.com/Niko1221/Strata/pull/1402

Repo : https://github.com/aflin/Strata

Folks! We're just 50 PRs away from more faster inference. Hopefully by end of year.

Experts!, please chip in there.

List of Open/Ongoing PRs(and also Discussions) related to CPU/RAM/Disk/Hybrid:

1. [Discussion] RFC: MoE expert cache, VRAM caching of hot CPU-resident experts with hybrid hit/miss execution #24528
2. AVX2: Speed up large batch size prompt processing of IQ models #27402
3. llama: add Maple 20B-A1B ternary MoE architecture (CPU)- #27000
4. ggml-cpu: tiled mul_mat for k-quants- #27851
5. ggml-cpu: add AVX-512 and VNNI paths for Q5_K/Q6_K dot products- #27590
6. ggml-cpu: add x86 VNNI Q2_0 dot product -- 3x speed improvement for VNNI-compatible CPUs- #26348
7. llama: add pshard runtime for plan switching and streamed weights- #22692
8. CPU Optimizations - Prefill, Tokenization, and Token Generation- #27032
9. llama : stream MoE routed experts from disk - #25294
10. ggml : speed up batch-1 CPU decode, align large allocations- #27478
11. misc : prevent RAM peaking at model loading stage- #27483
12. recurrent : support equal splits for recurrent-state rollback- #25004
13. ggml-cpu : add AVX2 vec_dot kernel for STQ1_0- #27377
14. --numa mirror: mirror model weights to every Numa node in the system- #16000
15. CPU flash-attn: support quantized K/V in the tiled prefill kernel- #26948
16. ggml-cpu/amx: fix block_q8_K VNNI quantization and enable VNNI path- #27024
17. server : add /slots endpoint action=clone_to (KV clone between slots)- #26204
18. ggml : fuse soft_max sweeps into fewer passes- #26468
19. ggml-cpu : add STQ1_0 ternary quantization with ARM NEON vec_dot kernel- #22836
20. llama-hot-experts: pin hottest MoE experts in RAM via --pin-hot-experts- #26414
21. llama : add --lazy-experts for MoE models larger than RAM- #26003
22. ggml : vectorize rms_norm reduce and fuse the scale write- #26486
23. MoE disk offloading for Metal- #23440
24. ggml-cpu: Added RVV VLEN=1024 vector dot product (vec_dot) kernels for quantized types.- #25397
25. ggml-cpu: detect AVX-VNNI in MSVC native builds- #25346
26. ggml-cpu: replace cyclic chunk distribution with atomic work-stealing- #25048
27. Improve performance of ggml_gemv_q4_K_8x8_q8_K for +12-23% tok/s on AVX-VNNI systems- #23309
28. ggml-cpu: Optimized Arm NEON cpu q1_0 dot (with plain/DP/I8MM)- #23358
29. ggml-cpu: ARM Repack kernels for Q1_0- #23492
30. ggml-cpu: add wasm simd path for iq4_nl_q8_0- #24058
31. ggml-cpu: optimize ggml_gemm_q4_K_8x8_q8_K interleaving/staging for AVX-512 (and AVX2)- #22525
32. ggml/cpu: skip zero-scale blocks in TQ1_0 and TQ2_0 vec_dot kernels- #23439
33. ggml-cpu:Optimized risc-v cpu nvfp4- #23402
34. ggml-cpu : fix riscv xtheadvector builds and add a q1_0 vec dot kernel- #23009
35. Q5_0 - Block Interleaving Implementation for x86 SIMD (AVX512/AVX2)- #22250
36. ggml-cpu: optimize q8 quantization on x86 SIMD- #22331
37. Optimize reduction stage of dot product of q4_L/q5_K to q8_K on AVX2- #22181
38. ggml: introduce GGML_NUMA_MIGRATE to optimize cross NUMA op computation - #14232
39. ggml-cpu: improve --n-cpu-moe TG performance- #20596
40. ggml : add CPU backend reference implementation (wip)- #16004
41. ggml: optimize ggml_vec_dot_mxfp4_q8_0 dot product on ARM SVE- #19171
42. Q6_K - Block Interleaving Implementation for x86 SIMD (AVX512/AVX2)- #19706
43. ggml-cpu: optimize q4_0_q8_0 scales using Zvfhmin- #19196
44. ggml-cpu: add q4_0 repack support for wasm- #18858
45. Improving inference speed for the repack buffer type on NUMA architectures- #18698
46. ggml: optimized runtime for x86 cpu backend and Q4_K quantized weights paired with Q8_K activations - #18495
47. CPU SIMD and pipeline optimizations across vec/mmq/ops/kv-cache/repack - #17113
48. ggml-cpu: optimise rms_norm op- #16650

PRs related to New Quant types:

1. Add ROCmFP4 CPU quantization support- #24185
2. ggml: add support for MXFP8 CPU- #26157
3. ggml: Add initial MXFP6 CPU implementation- #22671
4. ggml : add E4M3 (fp8) CPU quantization type- #25336

(Just had some extra time, so went through almost entire Open PRs of llama.cpp. For Poor GPU Club mainly)

EDIT : My guesstimation : After merge of these these PRs, 2 Channel DDR5 RAM could give ~8GB VRAM's performance possibly.

I'll be updating this thread with adding similar type PRs & also updating status of each PRs time to time.

>Thoughts About Scaling Law

Scaling, but not only of parameters. Every model release now ends with the same question: how many parameters? It isn't a question that can be answered on its own. Parameter count is only meaningful alongside three others — how much data you have, where you intend to spend your compute, and who will run the model, under what conditions.

The field learned this the hard way. Kaplan et al. (2020) fit an exponent that told everyone to grow parameters faster than data — roughly 2.7:1 — and the industry complied: GPT-3, Gopher, MT-NLG. Hoffmann et al. (2022) redid the experiment across four hundred models and found the compute-optimal split is closer to 20 tokens per parameter, and that with sufficient compute the two should grow at the same rate rather than drifting apart. The error in the earlier fit compounded with every order of magnitude of compute, which is why the largest models of that generation were the most misallocated. The trillion-parameter round was, in retrospect, a detour the whole field took together and then reversed.

Chinchilla wasn't the end either. It optimized training compute for models that would be trained once and evaluated. Today a model is called billions of times a day and inference dominates lifetime cost. Put inference into the objective and the optimum moves toward smaller models trained far longer — deliberate over-training, which is what Llama-2-7B and Gemma-2-9B were doing at roughly 290 and 889 tokens per parameter.

Sparsity moved the target again. In a MoE model two quantities have to be kept apart: total parameters govern roughly how much the model can hold — knowledge, facts, the long tail — while activated parameters and effective depth govern roughly how far it can think, how many steps of a causal chain it can carry before it comes apart. A dense 20:1 ratio does not transfer. And the ratio isn't a single number at all: Roberts et al. (2025) find the optimal tokens-per-parameter is task-dependent, with memorization favoring more parameters and reasoning favoring more data. Follow-up work on MoE observes that at fixed TPP, pushing total parameters higher actually degrades reasoning, while activating more experts reliably helps it.

This matters for what we are building toward. Finding a vulnerability is not a retrieval problem. It doesn't come from having memorized more CVEs; it comes from carrying a twenty-step chain of inference to the end without losing the thread. That capability does not live in total parameter count.

Which brings us to this release. Total parameters appear to matter up to a threshold — enough to hold the world — after which additional capability comes from scaling elsewhere: effective depth per forward pass, and above all post-training. GLM-5.3 is our controlled experiment on that claim. Same base, same architecture, same total and activated parameters as GLM-5.2. One month of scaling long-horizon environments and RL. The gains are not marginal. Well, scaling has more than one dial. We turned the post-training one this time because it had the most slack left in it — not because the others are finished. Base model size, pretraining data, compute spent per forward pass: all of them are still on the table, and we will come back to each. What this experiment taught us is that the dials do not have to be turned together, and that the one worth turning next is rarely the one that was worth turning last. We are not done scaling. Next time, maybe mid-training, pre-training, and even more.

Tweet : https://xcancel.com/jietang/status/2089941544581403107#m

EDIT : Found Retweet with more stuff - https://xcancel.com/auto_grad_/status/2089970913408380932#m

>We introduce Mobius-v0, an architecture that comprises a globally shared Memory (FFN) that stores knowledge vectors and multiple Reasoners (Self-Attn) that iteratively achieve compositional reasoning. Using hidden states as cache and carrier, reasoners repeatedly query memory for required knowledge-vectors, while the knowledge is transmitted back to reasoning operators. Through this knowledge-reasoning-separation architecture, Mobius achieves better knowledge compression and reasoning efficiency. Built upon Mobius-v0 architecture: 1) Our 7B model trained-from-scratch achieves similar downstream score as a 7B Transformer baseline with 62.6% of baseline's training data. 2) Our Intern-S2-Mobius, continually-pretrained from Qwen3.5-35B, achieves similar downstream score while delivering nearly 4x end-to-end inference speedup.

• arXiv : https://arxiv.org/abs/2608.14290
• PDF : https://arxiv.org/pdf/2608.14290
• GitHub : https://github.com/InternLM/Intern-S2-Mobius
• HuggingFace : https://huggingface.co/internlm/Intern-S2-Mobius

arXiv : https://arxiv.org/abs/2608.00146

Full Paper : https://arxiv.org/pdf/2608.00146

Tweet : https://xcancel.com/googlegemma/status/2086849199052845451#m

FYI both (llama.cpp) PRs ( 24423 & 24427 ) went to Draft mode. I'm still waiting for this one as I could get faster t/s on my 8GB VRAM.

📝 Introduction

We present Wan-Animate-2, a novel end-to-end character animation framework that directly consumes driving videos in a redesigned Diffusion Transformer, which achieves high-fidelity motion generation and strong identity preservation by eliminating intermediate motion extractors. We further add text-driven viewpoint control to decouple the output camera perspective from the driving video. In addition, we develop Wan-Animate-2-Lite, an efficient variant that reduces inference latency to real-time thresholds for streaming character animation.

Release Notes:

• August 07, 2026: 🎉 We release the Wan-Animate-2 inference scripts.
• August 07, 2026: 🎉 We release the Wan-Animate-2 Base model weights.
• August 07, 2026: 🎉 We release the Wan-Animate-2 Distillation model weights.

Models :

• https://huggingface.co/Wan-AI/Wan2.2-Animate-2-14B
• https://huggingface.co/Wan-AI/Wan2.2-Animate-2-14B-Diffusers
• https://huggingface.co/Wan-AI/Wan2.2-Animate-2-14B-Distilled-Diffusers

GitHub : https://github.com/Wan-Video/Wan-Animate-2

Questions & Responses(in BOLD) below.

Favorite question(s) moved to end of the thread with combined responses(removed duplicates).

Be optimistic folks. I'm sure we're getting other models too apart from 27B. And 27B gonna make massive noise on release. (Based on their responses)

Tweet thread : https://xcancel.com/QwenDevs/status/2084102417885585597#m

you guys skipped 27b and 122b last time, can we expect those this time around? Also i can't seem to find crit pit score in the cards.

For sure! We’re actually releasing a 27B model very soon. Stay tuned. As for the Crit Pit score, please wait for the official Artificial Intelligence score.

Is the 27B just a retrained 3.6 27B? Or is it based off 3.8 bigger brother ?

We promise this 27B comes with a whole new level of capability!

Is the 100hrs of video understanding an agent swarm that parses sections of the video in parallel and orchestrates some sort of semantic representation graph?

Broadly speaking, yes, but not entirely. It is closer to a hierarchical video memory system rather than a traditional agent swarm. Video segments are encoded into a structured textual graph containing scenes, entities, events, and their temporal relationships, enabling retrieval and reasoning across more than 100 hours of content.

hey! is there anything special about the pretraining distribution compared to other labs' models?

We hope our data is built on a more solid foundation!

how long do you think it would take to surpass anthropic level architecture?

well, we’re working hard on it, we promise😇

will u release a harness especially for qwen code ???

Any plans for a codex-like app?

More updates on Qoder and QwenWork are coming soon.

qwen 3.8 active params?

2.4T parameters (95B active)

how much RL was done in post training compared to previous models?

A truly unreasonable amount of compute.

Did they intentionally skip the previous Qwen3.7 27B and 35B A3B?
Does the revival of Qwen3.8 27B reflect the voice of the community? Or was it planned?

Of course! This is the result of taking the voices of the community seriously.

since its a pretty significant release will we get a technical report with full details?

No technical report for this one yet. We’re trying to keep up our near-monthly release cadence, though, and more powerful models are already in the works. Keep an eye out!

why does the model think so much mr qwen, my ai brain wonders.
wheres the token efficiency at
great model though

We support different levels of reasoning effort.

You showed SAE-guided fine tuning fixing code switching with qwen-scope. Is that kind of interpretability driven intervention part of the post training process now or is it still a research only technique?

It’s still primarily a research-oriented technique for now, though some of the insights may help inform future training and post-training improvements.

Attention? Hybrid?

The model architecture is similar to 3.5, but it’s a much larger-scale model!

When are we getting a CLI coding interface?

You may want to take a look at @qoder_ai_ide .

do you guys use qwen as your main interal tool? does this model show the same signs of intellegence as some openai models ("gpt 5.5 helped create 5.6")?

Sure!

How close is Qwen3.8-27B to GPT 5.4? 🤔

Well, you’ll be able to see for yourself soon.

what harness works best with Qwen?

Qwen is committed to delivering the best possible experience across all harnesses.

What made you guys wanna opensource the max weights ?

We heard what the community has been asking for

I wonder when I can surpass fable5

Trying hard

Great work guys🥂

1. What is something that you would like to see being built with the new model and its capabilities!?

2. I really want to explore the swarm of agents technique for building applications, any best practices or tips for the new model!?

1. We hope it can bring practical productivity value to people across different industries.
2. We recommend using it for tasks that involve more parallelized workflows or parallel execution needs.

I wanna know what rubric metrics you guys are using for FE

We use both absolute metrics for functionality and aesthetics, as well as relative metrics based on win/tie/loss comparisons.

Would be great to hear where you think Qwen is strongest for agentic workloads specifically: long-context planning, tool use reliability, coding, or cost at scale?

All of the above combined — ultimately delivering the most practical and reliable outputs for users.

How much is Qwen helping with Qwen research ?

It has already become a significant part of the model iteration process, with the model involved in nearly every stage.

Most Frontier labs have created a code-specific model (eg. Qwen3-Coder and GPT-5.3-Codex), but never followed up on them.

Did specialized models have problems? Or did general models end up being efficient enough to not bother creating a separate model?

We hope to build an all-in-one model.

will Qwen 3.8 have a stable, documented tool-calling and structured-output contract so local agent harnesses can swap models without prompt-specific tuning?

We provide native support interfaces for various protocols. You can check the Qwen blog for more details.

1: When quantizing Qwen 27B down for local deployment (e.g., 4-bit GGUF, NVFP4, or MXFP4), which transformer layers or vision attention blocks are most sensitive to degradation? Are there specific strategies you recommend to maintain both visual reasoning and high SWE-bench pass rates?

2: Qwen3.6-27B outperforms much larger MoE predecessors (like Qwen3.5-397B) on agentic coding benchmarks like SWE-bench and Terminal-Bench. Beyond raw data volume, what was the single highest-leverage factor in achieving this dense efficiency?

And thank you for the amazing work. Qwen3.6-27B has beed my main coding assistant for months.

1. Use QAT, or quantize only the FFN to 4-bit while keeping the attention layers’ QKV linear projections and output projection in 16-bit.
2. Higher-quality data engineering

Guys , when can we get a deepseek like small and cheap model with best performance . The deepseek v4 flash seems to be a great deal .
I think we need to slow down scaling and start improving the existing model efficiency

Scaling and cost-efficiency are not mutually exclusive — we’ll continue to pursue both.

Is Qwen3.8-27B dense? And roughly how much smarter than 3.6-27B?

A pretty huge jump!

Good. The useful questions are not just how capable Qwen is.

I want to know where it still fails, how the team evaluates those failures, and what "open" means in practice for weights, tooling, and reproducibility. Open models matter most when people can inspect the limits and build on the work without asking permission

There is still some gap between our automated and human evaluation systems and real user experience. That’s also why we are committed to releasing preview versions first — so we can iterate and ultimately deliver the best possible experience to users.

how does the new 27b model compare to the previous one ?

A pretty huge jump!

what do you think about looped transformers?

interesting research idea

Why Qwen, what made you create Qwen and specifically such light and fast models. Why focus efficiency when others just went for brute power? Also, do you think inference engines reached their limit in optimization or can they still improve?

Scaling and cost-efficiency are not mutually exclusive — we’ll continue to pursue both.

We have noticed that in thinking mode the model usually consumes the entire reasoning budget without stopping, which increases latency. Is this a known issue, and are there any improvements planned for Qwen3.8?

You can try 3.8! And 3.8 supports different thinking efforts!

...................................................................................................................

Are 70b models gone for good?

Is it possible to get a 40-50B model (something which fits around 30-32Gb) to improve performance while still useable on a lot of computers ?

Thank you for your promise to provide qwen3.8 27b weight! I want to know if there will be qwen3.8 35b a3b. Many people also want this.

Can we expect the ~122B model this time? The 120B segment is dated and lackluster atm and would greatly benefit from a competent release!

First of all, congratulations on the release of Qwen 3.8!
As for the question, are you going to release a 35B a3b version of Qwen 3.8 aswell?

Plans for 35b Moe model? (3.8)

Any plans for the omni family? You told everyone the weight sizes of 3.5, then never released them and haven’t done anything new with it. 3.6/7/8 variants would have also been nice. It could be your most popular family if you gave it attention and kept the weights small.

Are there no plans to release any models other than the 27b?
I'd love to hear about the successors to amazing models like the Qwen3 8b and Qwen VL 8b....

Are there any plans for updates for 0.6b or 8b weights?
These have become important positions in the open weight of image and video generation. I look forward to seeing that part evolve.

This is such a huge release, I am really happy to see that a 27B model is shipping too! Though, can't help but wonder, will we ever happen to see again any new small dense Qwen models 9B, 4B any time in the future, similarly to 3.5?

Will you release smaller models like the qwen 3.5 family ?

Thank you for your promise to provide qwen3.8 27b weight! I want to know if there will be qwen3.8 35b a3b. Many people also want this.

we hear you! collecting everyone’s requests and taking them into account as we plan future iterations.

We will gather your requests as a reference when considering future updates.

We hear you. Stay tuned.

We’ll collect everyone’s requests and take them into account as we plan future iterations.

Noted, collecting the requests and see what we can work into future iterations.

Keep the requests coming. We’re listening, and we’ll use them to help prioritize future updates.

Tweet : https://xcancel.com/Mononofu/status/2080937562739531837#m

>Model quantization has become essential for efficient large language model deployment, yet existing approaches involve clear trade-offs: methods such as GPTQ and AWQ achieve practical compression but are lossy, while lossless techniques preserve fidelity but typically do not accelerate inference. This paper explores the middle ground of statistically-lossless compression through three complementary notions of losslessness for quantized LLMs. First, task-lossless compression preserves zero-shot benchmark accuracy within natural sampling variance and remains achievable at aggressive bitwidths. Second, we formalize the stricter notion of distribution-lossless compression, requiring the quantized model's next-token distribution to be practically indistinguishable from the original, and propose the Expected Acceptance Rate (EAR), the maximum token-agreement probability under optimal coupling, as a directly interpretable fidelity metric (for example, EAR >= 0.99 indicates 99% agreement). Third, we prove a gamma-squared variance law showing that symmetric quantization inflates noise variance by gamma squared relative to asymmetric quantization, making asymmetry necessary for distribution-lossless fidelity but not for task-level preservation. Using SLQ, a layer-wise non-uniform method with asymmetric quantization and wide bitwidth search, we achieve task-lossless compression at well below 4 bits per parameter (as low as 3.3 bits depending on the model), distribution-lossless compression at 5 to 6 bits per parameter on average, and inference speedups of 1.7 to 3.6x relative to FP16 with optimized kernels.

• arXiv : https://arxiv.org/abs/2605.02404
• Full Paper : https://arxiv.org/pdf/2605.02404
• GitHub : https://github.com/IST-DASLab/SLQ (Code coming soon)

Note : This is 2 Months old Paper & Repo. Sharing this as RedHat AI tweeted this sometime back.

In Full Paper, I found llama.cpp & GG few times.

>From the accuracy/compression perspective, existing approaches can be clustered into two categories. The first is represented by lossy compression techniques, such as Roundto-Nearest (RTN) quantization (Dettmers et al., 2022), llama.cpp (Gerganov & llama.cpp contributors, 2023), GPTQ (Frantar et al., 2023), or AWQ (Lin et al., 2024) which seek to map existing models to popular hardware-supported formats, such as 4-bit grouped weight quantization.

>We focus on obtaining near-lossless quantized models via layer-wise non-uniform scalar quantization, chosen for its broad support across GPUs (Frantar et al., 2024; 2023; Lin et al., 2024) and CPUs (Gerganov & llama.cpp contributors, 2023; Pegolotti et al., 2023; Ma et al., 2024);

>Introducing FLUX 3.
One multi-modal model for Image, Video, Audio and Action-Prediction. Creations are truer to life in every kind of style.

Blog Post : https://bfl.ai/blog/flux-3

>Full-parameter post-training of trillion-parameter-scale MoE models introduces substantial system-level challenges for large-scale distributed training, including severe memory pressure, non-overlapped communication overhead, and inefficient kernel execution. While most large-scale LLM training systems are built around GPU-based clusters, this report presents an end-to-end optimization practice on the Ascend NPU SuperPOD. Using the DeepSeek-V4 model family as the target workload, we develop a hierarchical optimization framework spanning model-level parallelism, computation-communication orchestration, and low-level kernel execution. The resulting system achieves 34.22% Model FLOPs Utilization (MFU) with a 2.93x improvement over the open-source baseline recipe while maintaining training stability. Building on this optimized infrastructure, we further establish a CPT and SFT workflow for complex Operations Research (OR) tasks. We refer to the integrated framework as SLAI T-Rex. Using DeepSeek-V4-Flash, we develop OR-oriented CPT and SFT data pipelines that combine collected domain resources with solver-verified synthetic optimization documents. The resulting dataset contains 10K high-quality SFT samples spanning four task categories and three problem representations. The specialized model achieves the highest average zero-shot Pass@1 score among the evaluated models, reaching 71.81% and outperforming GPT-5.4-Mini and the base DeepSeek-V4-Flash model by 3.98 and 11.27 percentage points, respectively. Overall, this work demonstrates a full-stack pathway from efficient trillion-parameter model post-training on Ascend infra to domain-specialized Flash models for solver-grounded mathematical modeling, advancing frontier-model systems for complex reasoning.

• arXiv : https://arxiv.org/abs/2607.20145
• Full Paper : https://arxiv.org/pdf/2607.20145
• GitHub : https://github.com/SLAI-AITP/SLAI-T-Rex
• Modelscope : https://www.modelscope.cn/models/SLAIAITP/DeepSeek-V4-Flash-OR (I couldn't find this one on HuggingFace)