▲ 65 r/oMLX
OMLX Tuned quants for Qwen 3.8 27b (17gb in size) and Deepseek v4 0731 (93gb MTP Off, 103gb MTP on)
Spent the last few days tuning some quants for these two models. The 5 BPW Qwen model was tuned with ~30% of the data containing multimodal tasks so it should in theory be slightly better at vision tasks but I don't really have an accurate way of checking as neither model showed any degradation compared to the base model.
The Deepseek model showed degradation mainly in MMLU - BUT these were topics these were in areas like Human biology, Foreign relations, Chemistry ect. This loss off knowledge can easily be reinforced with RAG, or by having the model search the internet for the correct data.
https://huggingface.co/True2456/Qwen3.8-27B-AWQ-5.0bpw
u/MatiAI — 5 days ago
M5 Max Qwen 3.8 27b Benchmarks - Bf16 MTP vs AWQ 4.85BPW MTP - Omlx
Some comparisons of the base bf16 with MTP vs a mixed 4 bit AWQ quant using MTP and a few benchmarks (GSM8K, MMLU - 200 questions, Humaneval - 164 questions) (Sorry for the slop looking charts)
u/MatiAI — 6 days ago
45-50 Tok/s on an m5 max 128gb of ram using DeepSeek v4 0731 and MLX
EDIT - Up now https://huggingface.co/True2456/DeepSeek-V4-Flash-0731-AWQ
Been working on a AWQ Quantisation of Deepseek v4 flash the past few days. Experimented with quantising the MTP heads today and managed to get it right up to 45-50 tok/s. I plan on further doing a DWQ distill so it should recover even more of the behaviour from the model, but currently it is able to run 50+ minutes no problem without any looping.
# Context: Code (Python)
# Single request results
Test TTFT(ms) TPOT(ms) ppTPS tgTPS E2E(s) Throughput PeakMem
pp 1024 / tg 128 1650.8 19.9 620.3 50.8 4.2 275.7 102.4 GB
pp 4096 / tg 128 5927.1 20.3 691.1 49.7 8.5 496.5 103.4 GB
pp 8192 / tg 128 12823.2 21.7 638.8 46.4 15.6 533.9 104.5 GB
pp 16384 / tg 128 30535.1 21.4 536.6 47.0 33.3 496.4 106.7 GB
# Batch results
Batch tgTPS ppTPS avgTTFT(ms) E2E(s) Speedup
1x baseline 50.8 620.3 1650.8 4.2 1.00x
2x 36.9 466.5 4389.7 11.3 0.73x
4x 54.6 467.4 8620.2 18.1 1.07x
8x 75.2 470.5 16935.1 31.0 1.48x
u/MatiAI — 8 days ago
▲ 1 r/GoogleAIStudio
Terrible app that overcharges you randomly - What's the point of it vs Antigravity?
What's the point of AI studio? I've been subbed on the Google AI pro plan since the Pixel 8 release. I've happily using antigravity for months and got a bunch of emails saying I have cloud credits and to try out AI studio free. I tried to make a project and got prompted to add $ $25 prepaid credit. I did that used it for about 2h and found it to be way worse than antigravity so I stopped using it. Logged in today to see that im somehow -$40 in credit. Honestly made me feel like cancelling all my Google services. Try to contact customer support and you literally get stuck talking to a crappy gemini 2.5 bot for hours. No phone support for AI studio unless you have an organisation? Honestly why try and mix these things into your consumer subscriptions if you aren't going to add proper support.
u/MatiAI — 17 days ago
▲ 4 r/AppleMLX
15K Frontier Qlora Distill for Gemma 4 12b
Finished a distill today of ~15k Fable, Kimi k3 and GPT 5.6 Sol sequences today - link is up on Hugging face if anyone wants to check it out! https://huggingface.co/True2456/gemma-4-12b-it-qat-4bit-frontierdistill - Tool calling got quite a big increase!
u/MatiAI — 26 days ago
▲ 3 r/MiniMax_AI
Testing MiniMax M2.7 Q3 on a 128GB M-Series Mac: Self-Distillation Python Harness (Long Agentic Loops)
Hey everyone
I’ve been putting the new MiniMax M2.7 (running an unsloth dynamic Q3 quant) through its paces on my 128GB MacBook Pro using a custom harness I’ve been building.
For the test, I tasked it with writing a complete self-distillation program leveraging LM Studio and the MiniMax API. Given that M2.7 specifically targeted "self-evolution" during its training—meaning it was trained to build its own harnesses and optimize its own loops—I wanted to see how the quantized local version handled it.
Check out the video here!
u/MatiAI — 1 month ago