u/dampflokfreund

With all these decomp projects and the modding scene around the N3DS, this really breathes new life into the system. I was a bit disappointed that back in the day, the New 3DS only received a couple of enhanced games and very few exclusives, but now we are looking forward to insane projects like gamecube and Wii games on that thing and games from other platforms like Halo CE. Or take the Twilight Princess 3DS project by Game Tobi as an example, when I was younger I was dreaming of having TP on the go and in 3D nonetheless, and now it becomes reality. The extra processing power the N3DS provides will actually being put to use and I can't wait! And sure, I could just run all these games on my phone without breaking a sweat, but there's just something special about a Nintendo clamshell handheld with such a unique feature that is 3D.

And not only that, I was also able to mod my favorite games. For example, the Majoras Mask restore project and the OoT3D mod which fixed my only gripes with the remake (brings back the blood, the scary dead Hand and also allows me to use the C-Stick as a camera)!

That is just the beginning. Anyone else excited for what's to come?

Tweet by u/hackerllama

Could be copium, but I would love to see Gemma 4.1 there with unified audio input for all model sizes perhaps even up to 120B, much improved tool calling (even with the latest template there are still bugs), higher precision QAT from the start and improved general performance without hurting the things Gemma 4 is good at like creative writing.

Gemma 4 is good already but training an upgrade to 4.1 that does all of the above would be huge for the community. They already did a lot of course and I'm very thankful but Gemma is just an inch away from perfection. Is anyone hyped for this event or do you think they won't release any new models there?

Hello,

For the past few days I have been benchmarking Gemma 4 26b QAT UD Q4_K_XL extensively versus Bartowski's Q4_K_L.

While QAT is certainly very effective and reducing memory consumption versus the highest q4 quant from him, I also have noticed some regressions in my own internal benchmarks I cannot share because I don't want model providers to train on them. These benchmarks also include real world use cases in code and creative writing that need the model to think outside the box and also reference past information way past in the context, so a high precision is very important for the model to accomplish what I ask of it. Others test knowledge which also require precision.

Bascically, the QAT model was smarter in some areas but I cannot say it is an all-around improvement in fidelity. In some areas, the Q4_K_L was better in a way that was statistically significant and I have a pretty solid theory why I have been seeing these results. Let's look at how the tensors are quanted with the non-QAT model.

token_embd.weight	[2 816, 262 144]	Q8_0
blk(30)
blk.0(19)
blk.0.attn_k.weight	[2 816, 2 048]	Q8_0
blk.0.attn_k_norm.weight	[256]	F32
blk.0.attn_norm.weight	[2 816]	F32
blk.0.attn_q.weight	[2 816, 4 096]	Q6_K
blk.0.attn_q_norm.weight	[256]	F32
blk.0.attn_v.weight	[2 816, 2 048]	Q8_0
blk.0.ffn_down.weight	[2 112, 2 816]	Q8_0
blk.0.ffn_down_exps(2)
blk.0.ffn_down_exps.scale	[128]	F32
blk.0.ffn_down_exps.weight	[704, 2 816, 128]	Q8_0
blk.0.ffn_gate.weight	[2 816, 2 112]	Q4_K
blk.0.ffn_gate_inp(2)
blk.0.ffn_gate_inp.scale	[2 816]	F32
blk.0.ffn_gate_inp.weight	[2 816, 128]	F32
blk.0.ffn_gate_up_exps.weight	[2 816, 1 408, 128]	Q4_K
blk.0.ffn_norm.weight	[2 816]	F32
blk.0.ffn_up.weight	[2 816, 2 112]	Q4_K
blk.0.post_attention_norm.weight	[2 816]	F32
blk.0.post_ffw_norm.weight	[2 816]	F32
blk.0.post_ffw_norm_(2)
blk.0.post_ffw_norm_1.weight	[2 816]	F32
blk.0.post_ffw_norm_2.weight	[2 816]	F32
blk.0.pre_ffw_norm_2.weight	[2 816]	F32
blk.0.attn_output.weight	[4 096, 2 816]	Q5_K
blk.0.layer_output_scale.weight	[1]	F32

As you can see here, crucial layers like token embeddings and the attention layers are quanted to q8_0. So the precision is extremly high in a modern q4_k quant like UD Q4_K_XL or Q4_K_L. Higher precision means less chances for errors accumulating especially at high context sizes. Now, how does the QAT version look? (UD_q4_K_XL)

token_embd.weight	[2 816, 262 144]	Q4_0
blk(30)
blk.0(19)
blk.0.attn_k.weight	[2 816, 2 048]	Q4_0
blk.0.attn_k_norm.weight	[256]	F32
blk.0.attn_norm.weight	[2 816]	F32
blk.0.attn_q.weight	[2 816, 4 096]	Q4_0
blk.0.attn_q_norm.weight	[256]	F32
blk.0.attn_v.weight	[2 816, 2 048]	Q4_0
blk.0.ffn_down.weight	[2 112, 2 816]	Q4_0
blk.0.ffn_down_exps(2)
blk.0.ffn_down_exps.scale	[128]	F32
blk.0.ffn_down_exps.weight	[704, 2 816, 128]	Q4_0
blk.0.ffn_gate.weight	[2 816, 2 112]	Q4_0
blk.0.ffn_gate_inp(2)
blk.0.ffn_gate_inp.scale	[2 816]	F32
blk.0.ffn_gate_inp.weight	[2 816, 128]	F32
blk.0.ffn_gate_up_exps.weight	[2 816, 1 408, 128]	Q4_0
blk.0.ffn_norm.weight	[2 816]	F32
blk.0.ffn_up.weight	[2 816, 2 112]	Q4_0
blk.0.post_attention_norm.weight	[2 816]	F32
blk.0.post_ffw_norm.weight	[2 816]	F32
blk.0.post_ffw_norm_(2)
blk.0.post_ffw_norm_1.weight	[2 816]	F32
blk.0.post_ffw_norm_2.weight	[2 816]	F32
blk.0.pre_ffw_norm_2.weight	[2 816]	F32
blk.0.attn_output.weight	[4 096, 2 816]	Q4_0
blk.0.layer_output_scale.weight	[1]	F32

Now we can see that there are much more layers at q4_0 precision which makes sense as Google trained the model to be quantization aware for q4_0. This means that Google trained the model to be aware of the noise that occurs during quantization so it is able to handle that quite well, even though many of the crucial layers are at q4_0.

If you were to quant the regular non-QAT model with this receipe, the result would likely be a disaster. With QAT however, quality is retained rather well which allows it to push far above its weight.

However, QAT is not magic. The information loss between q8_0 and q4_0 is gigantic and especially at longer context, the errors do accumulate. In my benchmarks I can definately see a loss in fidelity at some point, at lower or higher context. But that is heavily dependend on the use case, so your mileage may vary.

What I propose for Google's next QAT models is that they align the QAT model to be aware of modern q4_k formats like the one I have linked above, where many of the crucial layers are quanted to q6k and q8_0 instead of plain q4_0. The downside would be that memory consumption would be identical to q4_K_L and UD_Q4_XL, so a few hundred megabytes more than current QAT models, but honestly both models fit in my RAM and have similar speed, so I would rather have higher quality.

This would result in a quantized model that indeed is much better than the q4k quants we actually use right now and in every single way, not just in certain domaines.

Now you might be saying "Google has published the unquantized QAT model, just make your own!". Sadly it is not that simple. I have asked Daniel from Unsloth and he said that quanting the layers with the regular q4_k_xl receipe actually decreases performance. So Google likely trained the model to be quantized aware for that exact receipe they have chosen, with the embeddings and attention layers quanted heavily to q4_0.

So Google has to train a QAT model to be aligned with modern q4_k formats for best quality and I really hope they will consider doing that in the future.

What are your experiences with Gemma 4's QAT versions compared to their regular ones? So far I have mostly heard about regressions, but if you have a different experience or even benchmarks that are in favor of QAT, this is the thread to share them.

Please share your positive or negative experiences. I think we really need more data to verify if one is truly better than the other. Google itself has not released any data. Please also share whether you have used Unsloth's or Google's QAT.

I really love this model, I have been using the q4_k_l by Bartowski (I have heard QAT is quite the downgrade in some aspects) and it handles every task I throw at it easily. Agentic and coding performance is not as good as Qwen of course but good enough and I have found myself to be constantly surprised by it considering the size and speed.

I really like its personality. It's a great writer and soulful especially with lower soft logit capping values. Of course it also has native multimodality which is a big plus.

But the real star of the show is its language capabilities and knowledge. It is excellent in German so it feels like a big cloud model in that regard and it knows quite a lot for such a speedy local model, too. Definately more world knowledge than Qwen.

And all of that runs at around 10-23 token/s on my aging laptop with 600 token/s prefill performance.

What is your experience with this model? If you haven't been using it for a while, consider giving it a go again with the new chat template.

For example, Subnautica 2 uses DLSS SDK 310.6.0. The new SDKs, I think starting with 310.5, force Preset M automatically at performance mode, so if you have an older card like me you might find that DLSS Performance actually reduces Performance compared to DLSS Balanced (which uses Preset K in that configuration) since Preset M is heavier than K.

This is confusing for the average consumer, so spread the word to help people out.