u/gerhardmpl

Hi everyone, I am considering buying either the EVO-X3 or the MS-S1 MAX for local AI usage (e.g. Qwen3.8-27B or MoE models like Gemma 4 26B A4B or Qwen3.8-Flash-Next). I am currently running a Dell R720 with 128GB of RAM and two NVIDIA P40s on Linux (Debian), but this platform is getting more and more difficult to manage and maintain.

My budget is around €4,000 and I am aware of the fact that the Strix Halo platform is limited, mainly by its memory bandwidth. I could buy a lot of cloud AI credits but prefer to run LLMs locally.

The trade-offs in the EU/Germany offers I'm considering are:
GMKtec EVO-X3 (~€3,700): 4TB SSD, native OCuLink and two PCIe 4.0 x4 M.2 slots. Downsides: only 2.5GbE, one USB4 port and less convenient chassis access.
MINISFORUM MS-S1 MAX (~€4,000): 2TB SSD, dual 10GbE, USB4 80Gbps, accessible chassis and internal PCIe expansion. Downsides: higher price and second M.2 limited to x1. The expansion slot is electrically Gen4 x4, so OCuLink might be added later on.

What are important considerations when comparing the EVO-X3 and the MS-S1 MAX? How are stability, sustained performance, fan noise and support? Would you buy the same machine again?

I've been experimenting with Multi-Token Prediction (MTP) using llama.cpp and unsloth/Qwen3.6-27B-MTP-GGUF at Q4_K_M on a dual-P40 setup.

I'm seeing a measurable but modest improvement and wanted to share my setup for feedback - is this to be expected with this older platform, or can I improve performance?

TL;DR: Without MTP: ~12 tok/s. With MTP: ~15 tok/s. Expected more. Am I hitting a hardware ceiling or misconfigured?

Configuration:

• 2× NVIDIA P40 (24GB VRAM each) in an XCP-ng VM
• Debian 12, Nvidia Driver Version: 580.126.20, CUDA Version: 13.0
• 8 vCPU, 32GB RAM
• Qwen3.6-27B-Q4_K_M.gguf (unsloth/Qwen3.6-27B-MTP-GGUF at Q4_K_M renamed for clarity.)

Results:

• Without MTP: ~12 tok/s
• With MTP enabled: ~15 tok/s

That's roughly a 25% improvement, which is nice but I was hoping for more given the dual-GPU setup and other MTP results with this model. Here's my docker-compose.yml:

services:
  llama-server:
    image: ghcr.io/ggml-org/llama.cpp:server-cuda
    container_name: llama-server
    restart: unless-stopped
    ports:
      - 3000:8080
    volumes:
      - /opt/docker/llama-cpp/models:/models
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    command: >
      --alias qwen3.6-27b-q4
      --model /models/qwen/Qwen3.6-27B-MTP-Q4_K_M.gguf
      --no-mmproj
      --no-mmap
      --host 0.0.0.0
      --port 8080
      --metrics
      --gpu-layers 999
      --tensor-split 1,1
      --parallel 1
      --cache-type-k q8_0
      --cache-type-v q8_0
      --flash-attn auto
      --ctx-size 131072
      --reasoning off
      --spec-type draft-mtp
      --spec-draft-n-max 1

networks:
  default:
    name: llama-server
    driver: bridge

What I've tried so far is MTP disabled/enabled with the same model and --spec-draft-n-max set to 1, 2, 3 and 5, but I do not see any significant increase in token/s. So I wonder

1. Is ~25% speedup from MTP typical on P40s, or am I seeing more of a hardware limitation (P40s are Tesla-gen, not Ampere+)?
2. Any other llama.cpp parameters that could squeeze more out of this dual-GPU setup?
3. For anyone else running MTP — what speedups are you seeing on comparable hardware?

Happy to share more benchmark details if helpful. Thanks for reading! P.S. Drafted with the very model I'm benchmarking - ironic, I know.

Hermes allows for per-task auxiliary routing to secondary models e.g. for vision processing, web extraction, session search, or text compression (configured in config.yaml). You can use lightweight models for side tasks and save resources (money, latency, runtime) compared to routing everything through the primary model.

I am testing local models (granite4:3b, gemma4:e2b, gemma4:e4b, qwen3.5:9b) because I like to tinker and to save resources but with mixed results. While some handled specific tasks well, others struggled with accuracy or speed depending on the task type (e.g., vision vs. compression vs. session search).

Are you actively using auxiliary routing? If so, which local (or cloud) models have you found work best for specific auxiliary tasks? I’m especially interested in recommendations for web and session search.