GitHub: https://github.com/mikechambers84/ik_llama.cpp/tree/numa-mirror

Be sure to checkout the numa-mirror branch.

Sharing this for anyone else who's trying to use their multi-socket CPU systems for inference. I've been wanting a NUMA mirror mode for a long time, so I finally forked ik_llama.cpp and added it.

ik_llama.cpp is a llama.cpp fork that adds major performance improvements for CPU inference, so it made sense to fork that here rather than baseline llama.cpp.

For anyone who isn't aware of the problem this is meant to solve, it's that multi-socket machines have memory that's local to each socket. When a CPU accesses its own local memory, it's very fast. If a CPU has to remotely access memory that's non-local through a different socket, there's a huge performance penalty because it has to transfer the data through a bridge that's far, far slower than local memory.

For most workloads, it matters very little and you probably won't notice. But since LLM inference performance is heavily bound to memory bandwidth, performance completely tanks if you try using multiple CPUs and they have to read large amounts of remote memory for each token.

The usual answer for this just to use --numa isolate in llama.cpp, which pins model/context data to a single socket's CPU and memory, eliminating remote memory accesses but having multiple CPUs is no benefit here, all but one just sit idle.

This fork adds --numa mirror which makes full duplicate copies of model weights and KV cache so that every CPU socket has a node-local copy. This allows you to actually use all of your CPU cores across all sockets to actually speed up inference instead of making it slower.

The trade-off is obviously that you need more memory. If you have two CPU sockets, it needs to use twice the RAM.

I'm hoping ikawrakow will accept it in a pull request. I'll try to submit one soon, but I'm hoping to have more people test in various hardware configurations beyond mine first.

My benchmarks are showing significant gains! My hardware is somewhat outdated, I'd be interested to know how it runs on newer stuff.

Test setup

• Operating System:
  • Debian 13 "Trixie" with numa_balancing disabled during benchmarking
• Hardware:
  • Model: Dell PowerEdge R740
  • CPU: 2× Intel Xeon Gold 6248R (Cascade Lake), 2 NUMA nodes (24 cores / 48 threads each)
  • RAM: 768 GB RAM (384 GB per node) ECC DDR4 2400 MHz, all 12 memory channels populated
• Build: CPU backend, Release, -DGGML_NATIVE=ON -DGGML_AVX512=ON -DGGML_AVX512_VNNI=ON. (VBMI/BF16 are not enabled — Cascade Lake does not implement avx512_vbmi / avx512_bf16.)
• Tool: llama-bench, 3 repetitions per result (-r 3).
• Per-run flags: -rtr 1 -b 16 -ub 16 -p 512 -n 128 (run-time repacking on; batch and micro-batch 16; pp512 = prompt processing of 512 tokens, tg128 = generation of 128).
• Modes compared (threads set equal for -t/-tb):
  • isolate — --numa isolate -t 24 -tb 24 (one socket / 24 cores) — single-socket baseline
  • mirror — --numa mirror -t 48 -tb 48 (both sockets, weights + KV duplicated per node)

All throughput numbers are tokens/second (higher is better).

Token generation (tg128)

Model	isolate (1 socket, 24t)	mirror (2 sockets, 48t)	mirror vs isolate
gemma-4-E2B (dense, Q5_K_M)	47.20	62.00	1.31×
gemma-4-E4B (dense, Q5_K_M)	23.77	33.62	1.41×
gemma-4-26B-A4B (MoE, UD-Q4_K_M)	23.59	34.76	1.47×
Qwen3.6-27B (dense, Q4_K_M)	5.27	8.32	1.58×
Qwen3.6-35B-A3B (MoE, UD-Q5_K_M)	24.70	31.56	1.28×
Qwen3.5-122B-A10B (MoE, UD-Q3_K_XL)	10.00	14.46	1.45×

Prompt processing (pp512)

Model	isolate (1 socket, 24t)	mirror (2 sockets, 48t)	mirror vs isolate
gemma-4-E2B (dense,Q5_K_M)	259.90	256.69	0.99×
gemma-4-E4B (dense, Q5_K_M)	141.88	184.06	1.30×
gemma-4-26B-A4B (MoE, UD-Q4_K_M)	143.41	201.69	1.41×
Qwen3.6-27B (dense, Q4_K_M)	33.04	54.22	1.64×
Qwen3.6-35B-A3B (MoE, UD-Q5_K_M)	153.68	193.21	1.26×
Qwen3.5-122B-A10B (MoE, UD-Q3_K_XL)	57.17	83.01	1.45×

I've been using Qwen3.5 122B A10B (Q3_K_XL) a lot lately for coding, and it's been pretty incredible overall like it feels not far off from frontier-level for most tasks -- but I've been noticing that usually once I hit around 75-80k context use, it starts to get dumb all of a sudden.

It just hits a brick wall and quality deteriorates rapidly and drastically. It'll begin hallucinating, forgetting things, or think something it said/suggested was actually something that I said.

I found I have to compact before I get to that point, and then it keeps going on just fine.

Is this because I'm running Q3? Unfortunately Q4 is just outside of the capability of my system specs unless I want to start disk swapping.

So is it just an issue with this particular model? Or because it's Q3? Are there llama.cpp settings that can help?

I'm already using BF16 KV cache.

EDIT to add the snippet of my model config file for this one:

[*]
flash-attn = on
n = 8192
t = 8
tb = 8
cpu-range = 0-7
cpu-strict = 1
cpu-range-batch = 0-15
cpu-strict-batch = 1
jinja = on
reasoning-budget = 4096
reasoning-budget-message = " -- Reasoning budget exceeded, proceed to final answer."

[Qwen3.5-122B-A10B-UD-Q3_K_XL]
model = G:\models\Qwen3.6-122B-A10B\UD-Q3_K_XL\Qwen3.5-122B-A10B-UD-Q3_K_XL-00001-of-00003.gguf
ctx-size = 131072
cache-type-k = bf16
cache-type-v = bf16
presence-penalty = 1.1
repeat-penalty = 1.05
repeat-last-n = 512
temp = 0.1
top-p = 0.95
top-k = 20
min-p = 0.00

I am kind of thinking of maybe a 12th gen i7/i9 Optiplex and slapping a quad port Intel NIC in it.

Does this sound good? Will ESXi complain about anything here? I tried it once on a 10th gen i7 Optiplex and it seemed happy, but I need something a little beefier than that. The one thing that concerns me is the newer Intel desktop chips and e-cores. I'm not sure how well ESXi will handle these, or if it knows how to properly handle scheduling with them.

Any better suggestions? I also wonder how much I can drop the power load by just removing the second CPU, but I suspect it will still use considerably more than something like an Optiplex.

I'll miss having 512 GB RAM, a boat load of cores, and redundant everything, but I need to be more practical. The rack server is just so expensive to run.

EDIT for specs -- It's an R740 with:

2x Xeon Gold 6148 @ 2.4 GHz (so 40 cores total)

512 GB RAM

8x 4 TB SATA SSDs in RAID10

And the wind is alternating between gusty and dead still.

My bird flew on my shoulder 10 minutes ago and he's just been sitting there totally silent, which is weird. He usually does not shut up.