[Benchmark] Kimi K2.7 Code Q3 on Mac Studio M3 Ultra + RTX PRO 6000 over llama.cpp RPC: prefill improves, no changes in token generation/decode
▲ 26 r/LocalAIServers+3 crossposts

[Benchmark] Kimi K2.7 Code Q3 on Mac Studio M3 Ultra + RTX PRO 6000 over llama.cpp RPC: prefill improves, no changes in token generation/decode

I came across this interesting article https://blog.exolabs.net/nvidia-dgx-spark/ while I don't have the DGX spark but it made me curious will this kind of arch speed up my setup for LLMs?

Mac can host large models but the prefill speed sucks, so I tested in it on my setup for Kimi 2.7.

Short answer: it helps prefill, but it does not meaningfully help decode on this setup. RPC is still mostly a capacity tool unless the network/interconnect and split mode are much better.

Setup

  • Host: Mac Studio M3 Ultra, 512GB unified memory, Metal
  • Worker: Linux box with NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 96GB VRAM, CUDA
  • Network: direct Ethernet between Mac and Linux box, but only 1GbE in practice
  • Measured RPC transfer rate: about 112-113 MiB/s
  • Model: unsloth/Kimi-K2.7-Code-GGUF, UD-Q3_K_XL
  • Model size on disk: about 432GB across 11 GGUF shards
  • Runtime: llama.cpp server version 9827 (4c6e0ff3a), Unsloth build

Controlled test

Same synthetic prompt for both runs:

  • Prompt tokens: 7120
  • Generated tokens: 64
  • temperature: 0
  • ignore_eos: true
  • Prompt cache disabled
  • Prefill gain: about 14.8%
  • Decode gain: about 4.2%
  • Total request time improvement: about 12.3%

Split trend

The generation columns are - where I only ran prefill. The controlled generation rows used the exact same 7120-token synthetic prompt; the earlier split-sweep rows were around 7.1K prompt tokens but not always the exact same prompt.

Run RTX share Split Prompt sec Prefill tok/s Decode Total RTX VRAM
Mac 0% - 53.58 132.88 17.55 tok/s 57.23s none
Mac + RTX 15% 15,85 51.48 138.3 - - 69.4GB
Mac + RTX 19% 19,81 50.22 141.77 - - 84.1GB
Mac + RTX 20% 20,80 49.54 143.72 - - 93.2GB
Mac + RTX 20% 20,80 46.69 152.49 18.28 tok/s 50.19s 93.3GB
Mac + RTX 21% 21,79 - failed - - failed

20,80 was the practical max on this card with 128K context.

21,79 failed even at 8K context:

RPC/network trace

For the 7120-token prefill-only 20,80 run:

  • Mac -> RTX: 251.59 MiB, 2.03s
  • RTX -> Mac: 194.69 MiB, 1.49s
  • Total RPC traffic: 446.28 MiB, 3.52s
  • RTX graph compute: 1.34s

The RPC traffic is mostly hidden activations, not text tokens. For prefill it is chunked/batched, so the network cost is noticeable but not fatal. For decode, the boundary is crossed every generated token, which is why I expected decode to suffer more. In this test decode was roughly the same as Mac-only: 18.28 tok/s vs 17.55 tok/s.

Learnings

  • I can knock off few more seconds by using a better cable, but not sure it's worth it
  • It is useful for fitting models/splits that otherwise do not fit one device.

Question: As I was increase the shards, the prefill speed was decreasing, but will this trend continue if I add one more GPU? People with multi GPU setup what's you take on this?

u/No_Run8812 — 4 days ago

Should I keep or sell my M3 Ultra Mac Studio with 512GB RAM?

I bought an M3 Ultra Mac Studio with 512GB RAM for around $9K. At the time, I was worried that once the M5 Ultra comes out, the resale value might drop hard, maybe to the $3K-$4K range.

But with Apple’s recent pricing increases, I’m wondering if that fear is overblown. Is it reasonable to think that even after an M5 Ultra launches, a 512GB RAM M3 Ultra Studio could still hold value reasonably well?

The other side of this is that I bought it thinking I’d do serious local AI / ML / creative work on it, but honestly it’s mostly just sitting around. Part of me thinks I should sell it now while resale demand is still strong. Another part of me wants to keep it, use it properly, and maybe upgrade to an M5 Ultra 512GB model whenever that exists.

For people who have owned high-end Mac Studios, what would you do in this situation?

Would you:

  1. Keep it and actually build a workflow around it

  2. Sell now before the M5 generation arrives

  3. Wait and see how M5 Ultra pricing affects resale

I’m mainly trying to avoid holding expensive hardware just because I like the idea of using it.

reddit.com
u/No_Run8812 — 7 days ago

The gap between knowing something and actually understanding it — AI accelerated my learning curve

I've been experimenting with setting up local LLMs lately, and here's what hit me hard:

Just because it's cheap to build something doesn't mean you should. If a compatible tool already exists for your use case, use it first. Only roll your own once you've confirmed the existing option falls short.

knew this before — but knowing something in theory and truly understanding it through experience? Completely different.

This is especially important for people who love building things or are early in their careers. AI makes it look like anyone can build anything nowadays, which is both inspiring and misleading. The barrier to start looks low, sure — but the path to actually getting it right still takes time and patience.

Trust me, you'll save yourself a lot of frustration if you internalize this sooner rather than later.

To the experienced folks here: what's one piece of advice you'd give to newbies to help them avoid common mistakes?

This post is refined by minimax2.7 local in openweb UI

reddit.com
u/No_Run8812 — 2 months ago

True story,

I got interested in AI after seeing it at work and wanted to run models locally. I started with an M3 Ultra 96GB, quickly learned it was not enough for what I wanted, and kept upgrading hardware (including refurbished Mac Studios at 256GB/512GB and now an RTX Pro 6000 that arrived today). I tested many model families (Qwen, DeepSeek, Gemma, Minimax, etc.). My current favorite is MiniMax M2.7 230B/A10B. I’m also waiting for LM Studio support for DeepSeek v4 Flash.

I have mixed feelings: excitement about local speed/bandwidth and sadness about how much money I spent learning this stack. Also funny point: my 16GB MacBook Pro has been more stable than my 512GB setup, which crashed multiple times.

Still, I’m convinced local LLMs are the future, and this community helped me learn a lot. Thank you to everyone here.

Question for the group: For people running high-end local setups, what gave you the biggest real-world stability + speed gains (not just benchmark wins)?

If you want, I can also give you a more technical version focused on benchmarks/specs.

reddit.com
u/No_Run8812 — 2 months ago