I run small models awen3.5-9b and gemma-4-12b for some tasks. Recently tried GGUF format and looked pretty much okay. Getting 15-20 tok/sec and 100-150 on prefil with MTP.

Decide to try again oMLx. I cannot get better performance than ggufs. Maybe I am missing small models MTP versions. Or faster models. Or my settings bad?
So question: what models you guys using on similar hardware how much you are getting ?

Use cases: parsing, email drafting and general assistant.

My Device: mac m2 24gb ram

TL;DR: Upgraded from Q4 to Q6 quantization. Local model quality surprised me. Both one-shotted. Sonnet is faster, but local works fine for simple tasks.

📋 What I Tested

Comparison	Model	Quantization
Local	QWEN3.5-9b-Q6_K	Q6_K
API	Sonnet 4.6	Low thinking

🎯 The Task (Given to Both Models)

Goal: Create a simple HTML converter that takes JSON input and outputs CSV rows for copy-paste.

Input JSON:

json

[  {    "notes": "Inspection: Passed",    "grass": "wet",    "cut": "not possible",    "weather": "sunny",    "mood": "tired",    "coffee": "needed",    "deadline": "tomorrow"  }]

Prompt:

>

Expected Output:

notes,grass,cut,weather,mood,coffee,deadlineInspection: Passed,wet,not possible,sunny,tired,needed,tomorrow

📊 Results

QWEN3.5-9b-Q6_K (Local)

1st attempt: ✅ One-shotted perfectly ![Screenshot 1]

https://preview.redd.it/psz83nbh1o7h1.png?width=939&format=png&auto=webp&s=4f9afa8878d12d9e8135d548fb83591fb5e6348e

After that asked: "make it more attractive": ✅ Also handled it ![Screenshot 2]

https://preview.redd.it/xblrdcvk1o7h1.png?width=1008&format=png&auto=webp&s=4440acc2e51a0ee01355e820e34d3c16830e9d44

Generation time: ~2 minutes

Sonnet 4.6 on Low (API)

Result: ✅ One-shotted ![Screenshot 3]

https://preview.redd.it/9cosixat1o7h1.png?width=1913&format=png&auto=webp&s=b3233dccbb654e5750ab571c3a9bf9e21730736d

Generation time: ~1 second

💡 My Take

>

• ✅ Task was simple, but idea is that even local models can be used in various tasks
• ✅ Both models one-shotted - no iterations needed
• ✅ Quality is surprisingly good for local LLM (Q6_K makes a real difference vs Q4)
• ⚠️ Latency is the main difference (2 min vs 1 sec)

🤔 So, is local LLM worth it? (General Use)

Yes, consider local LLMs if:

• 🏠 Privacy matters - data never leaves your machine
• 💰 No API costs - save money on high-volume usage
• 🔌 Offline capability - works without internet
• 🎯 Complex privacy-sensitive tasks - legal, medical, personal data
• 🧠 Creative work - chatting, ideation, content generation
• 🤖 Agentic workflows - local agents don't need external calls

No, prefer API models if:

• ⚡ Speed is critical - production SLAs, instant responses
• 🔥 High accuracy needed - complex reasoning, edge cases
• 💵 Budget matters - cheaper than running hardware
• 🏢 Team collaboration - centralized, managed service

⚠️ Disclaimer

JSON data is RANDOM – This was a demo to show actual performance and quality differences.

EDIT: TL;DR again - both models worked great. Local took longer but quality is good. Sonnet is faster but costs more and needs internet. Is local LLM worth it for general use? Depends on your privacy needs and speed tolerance.

EDIT: yes post was written by the same Qwen3.5-9b with a human review.

Hey guys! I tested various of models including Gemma-4-12b or Qwen3.5-9b + MTP.

Setup:
- macbook pro m2 24gb ram
- llama.cpp
- context from 4096 to 70k depending on task (just chatting vs research vs agentic harness

Questions based on my hardware:

1. Is it possible that MTP models doesn’t make any good impact or even make it slower?
2. If Unsloth Studio supports mlx models which ones actually better in performance gguf or mlx?
3. Any suggestions for other models for agentic tasks? My expierence: gemma-4-12b is super slow. Q4. Qwen3.5-9b also very slow and not smart enough for my tasks. Seems its ruining what it builds. Tried qwen3.5-9b-q6 maybe a bit better, performance is the same as Q4.

For both < 10 toks/sec and 85-100 prompt processing. For agentic harness even slower.

Thank you!!!

Hi!! I am planning to upgrade from my Macbook m2 pro to something more powerfull.

Did not decided go for unified memory way (basically macs) or GPU like rtx 3090, 4090, 5090 etc.

Budget is \~3k. Can spend less then upgrade (if GPU).
Use case: agentic tasks, Hermes Agent, chats etc. 1-3 models smaller <10b, mid 10-35b.
Upgradability from models perspective is important for the future but not till end of this year I think no so important.

Till now familiar with many tools, ggufs and mlx. No preference on equipment.

What would be your go to and why?

Thanks for the honest answers!

I have been using OpenWebUI for a while with local models but seems like I am missing something. There are so many features but nome of them feels native for me. Small thing but statistics during and after generation would be useful for me.

Update model params? Not comfortable at all..

Setup:

- Models on LM Studio pointed to OWUI. Tried with llama.cpp as well.

- Mac m2 24gb unified memory.

- Since its work laptop so I run smaller models 4-20b depends on the task.

Skills? Installed several - not working.

System prompts? Not using since my LLMs have different use cases so need to swith fast. Here not really possible.

What

- works: scheduled tasks

- websearch

I know community like this tool. I will try testing for a week more to give it a second chance.

What are your use case’es? Help me like it as a daily driver.