u/halfpennymac

When Qwen3.8-27B dropped on Thursday the obvious question came up for us : does a 27B dense model actually fit on the 24GB Mac Mini machines? Ran it properly over the weekend on an M4 Pro (24GB unified, macOS 26.6.1, llama.cpp b10488, bartowski GGUFs). Numbers and gotchas below.

Speed (llama-bench, 3 runs each, flash attention on, full GPU offload):

Quant	File size	pp2048	tg128
Q4_K_M	17.77GB	96.8 tok/s	11.4 tok/s
IQ4_XS	15.57GB	95.4 tok/s	10.9 tok/s

Basically identical speed, so take IQ4_XS: the 2GB you save goes straight into context headroom. Q5_K_M and up, forget it on 24GB.

The three settings that matter:

1. Raise the GPU wired limit. macOS won't wire 17.8GB to the GPU by default on a 24GB machine. sudo sysctl iogpu.wired_limit_mb=20480 fixes it (resets on reboot). After this, Metal reports a 21.5GB working set and the whole model loads clean.
2. Quantise the KV cache for context. With IQ4_XS + -fa 1 -ctk q8_0 -ctv q8_0, a full 32k context loads and answers fine: ~16.6GB resident, 20% of system memory still free.
3. Turn thinking off for interactive use. This is a reasoning model and at 11 tok/s that hurts: our first coding prompt produced 6,500 characters of chain-of-thought and hit a 1,600-token cap without ever starting the answer, 142 seconds of deliberating. With "chat_template_kwargs": {"enable_thinking": false} in the llama-server request, the same prompt returned a complete working Python tool in 28 seconds. Keep thinking on for batch/overnight jobs where the quality gain is worth the wait.

One more thing: llama-cli with a raw -p prompt ran away on us and dumped gigabytes of output. Use llama-server, it handles the chat template properly and you get the web UI for free.

Verdict: genuinely usable at 4-bit, ~11 tok/s is slow but okay for off line tasks and just about bearable reading speed. Q4 output quality is strong. But it's the ceiling for 24GB: no room for Q8 (28.6GB), no room for the vision encoder next to a big context, no room for the rest of your stack. The full writeup with methodology is here: https://halfpennymac.com/qwen38-27b-m4-mac-mini-benchmark

Happy to answer questions or rerun anything with different settings, the test machine is sitting next to me.

We run a Mac mini cloud hosting service and wanted to publish honest benchmarks of what these machines can do for local inference. We installed Ollama on one of our M4 Mac minis (16GB unified memory) and ran three open-source models against GPT-4o on identical coding tasks.

Test 1 — Binary search (explain + implement in Python):

Model	Generation	Tokens	Time
Qwen 3 8B	19.5 tok/s	1,509	1m 19s
Qwen 2.5 7B	22.3 tok/s	788	35s
Llama 3.1 8B	21.0 tok/s	365	38s
GPT-4o (API)	136.6 tok/s	498	5.2s

Test 2 — Full Express.js REST API with Zod validation, error handling, TypeScript types:

Model	Generation	Tokens	Time
Qwen 3 8B	19.4 tok/s	2,048	1m 46s
Qwen 2.5 7B	22.3 tok/s	1,399	1m 3s
Llama 3.1 8B	20.8 tok/s	1,335	1m 5s
GPT-4o (API)	148.4 tok/s	1,392	10.6s

The honest take: GPT-4o is roughly 7× faster and produces better output on complex tasks. No sugarcoating that.

But the local models all produced valid, runnable code. Qwen 3 8B uses a built-in reasoning mode (think trace before answering) which inflates token count but gave the most thorough output. For everyday coding — autocomplete, boilerplate, docs — they handle it fine at 19–22 tok/s.

The economics are where it gets interesting. Running AI agents 24/7 through cloud APIs costs $100–300+/mo in tokens. Locally it's $0 after hardware, and your code never leaves the machine.

Full write-up with methodology and prompt ingestion speeds: https://halfpennymac.com/qwen3-llama3-m4-mac-mini-benchmark

We've got these machines running 24/7. We are happy to run any other models or prompts people want to see benchmarked. Just drop a comment and we will run