u/Opposite_Leave_8338 — reddlx

Swift/ThinkingCap fp16 quantizations

Edit :

That’s why I appreciate that man, not a one day passed and he posted both with all the quants for both bf16 and fp16 , ScottLowry is my hero, here is the collections :

https://huggingface.co/collections/scottlowry/thinkingcap-qwen38-27b-oqe-mtp

https://huggingface.co/collections/scottlowry/swift-15-qwen38-27b-oqe-mtp

Thanks again man we all appreciate your work

Hello guys,

Why there is no good Swift 1.5/ThinkingCap Quantizations with fp16 for M1/m2 macs yet, where is the good guys we used to thank one day after a model release, if it’s an appreciation issue so I’m here to thank all the good uploaders I used to download fp16 models from.

Thanks to :

Scottlowry ( my favorite , most of my models are quantized by you )

Jundot of-course ( thank you for all of that oMLX great updates, without you we cannot achieve any of that )

Yanun ( I’m using your swift 3.8 oq4e already and I’m flying with it but what about 1.5 and ThinkingCap )

And lots and lots of other legends , you guys pushing us forward ( all uploaders are appreciated I only mentioned the names I use daily but you guys all helping us and we wish you all good things in life , thank you anyone uploaded something for the community)

Please upload all variants and quantizations from these 3 good models ( Swift 1.0 / Swift 1.5 / the new ThinkingCap 3.8 ) for fp16.

Thank you all so much and everyone who is helping too.

u/Opposite_Leave_8338 — 9 days ago

▲ 18 r/oMLX+1 crossposts

Is Qwen 3.8 Flash Next usable on M1 Ultra 64GB ?

Hi, I have an m1 ultra with 64GB and I really a little bit struggling to get the best behavior out of it, It’s a good device with good memory bandwidth but I feel that I stuck in between of the small models and the very big models , here is what I tried :

Ornith ( 1.0 or 1.5 ) both are really great with oQ8e, Getting seriously good numbers out of both, 35-60 TPS ( 70 with mtp ) and 500-1100 Prefill , those 2 are my best setup but they are a little bit challenged in serious coding tasks considering that they are only a3b.
Qwen 3.8 27B, the forbidden fruit, on paper I should conquer this model in any quant I want but in reality since it’s a dense model no matter how much I try , it is so slow, very helpful and very powerful but will still be painfully slow, the oQ8e gives me a 28 TPS Start but get down to 10-15 right away and the prefill is always 100-190, it takes too much time so always the harnesses cancel the request, I tried oq6e but it’s some how it’s even slower, oq4e is quiet the same as oQ8e it starts with 30 TPS but drops to 10-15 also , and the prefill is really low on all of them ( All Native mtp enabled , if not I get a non usable results ) i also tried mtplx , got a better tps , but prefill is still low and the model is really dump even when I choose the optimized quality variant.

I’m using oMLX with ANE Closed , Native MTP enabled, Context window 131K , all Models are fp16 , no kv cache quantization

When I try enabling ANE it gets slower , Same If I try to use Spec Prefill, it gets slower and it dumb the model down although I run the drafter model on bf16

I also tried Dflash2 and it’s getting slower and dumber also

No matter what I do I cannot get 27B to be effectively usable on my machine although it does have what it takes to get the job done, once the harness decides to open a sub agent or even compress the context or when the model is writing a slightly long file I get crash and I start from the beginning.

So I want to upgrade for a bigger MOE Model but sadly we have nothing in the 50-80B area that will work perfectly on my device, the next very good big thing is the Qwen 3.8 Flash Next.

I read a lot on how people ran it on M1 max with 64 and some times even 48GB but didn’t see a direct opinion is it really usable on that case or not.

Sadly my internet speed is not so great and I also have an internet quota not unlimited internet and it’s not at all cheap ,and all these models is very very big, will take days to download and test and also will force me to renew my internet quota multiple times so I don’t have the possibility to download them all and test each one and each way, it would be better if someone tested it really answer me this :

Is qwen 3.8 flash next really usable on 64 Macs ? Specially fp16 models ( m1/m2 ultra/max ) , but really usable for real work not tests and benchmarks
If Yes, Which way I should use with which model ( forked lamacpp or whallm or oMLX or AtomicChat or a reap model and which one I found too many, what is the best efficient way ?)
If not , is there is a way to make 27B really usable ( it works great but not for a long run, speed drains and most of the times fail while closer to compaction )

Thank you so much all will appreciate your help.

reddit.com

u/Opposite_Leave_8338 — 28 days ago

▲ 67 r/oMLX+1 crossposts

Is it worth it to use big models at 1 or 2 bit quants?

Hi, I recently had a Mac studio m1 Ultra with 64GB and I’m using it as a stationary ai server for my home, I’m using oMLX with otnith 35B at 8 bit and everything is running very well, but the 35B a3b base is having some difficulties to do my work I’m working on big coding projects. So i decided to use the 27B of qwen oQ8 with mtp, but no matter what I do it’s slow, and it’s causing most of the coding agents to timeout, and once the harness call a subagent everything drops in seconds, so I think I need a bigger MOE Model to work better on my device and still be a lottle faster, but a bigger than the 35B , I tried to look for the laguna s2.1 , I can run the oQ2e version comfortably but it’s a 2bjt quant, I know those are bad !! So here is my questions :

Are bigger models on less quants worth it ??? , will they perform better than small models with big quants ?
Is there is anyway to make 27B really useful like ornith ?

Note : I’m using zcode as my harness, it a codex clone and it really extremely does the job

Thanks in advance

u/Opposite_Leave_8338 — 2 months ago