u/Confident-Pass6353

I am wondering if I could get some tips on my current inference setup, and if there is anything I am missing that could make it faster.

Machine: HP Z6 G4
CPU: Xeon Gold 6240
Ram: 6 Channel DDR4 RDIMM 2666 MHZ 192GB
GPU: 2x Radeon Pro V620 32gb (6800xt datacenter equivalent)

Using the latest llama.cpp with the UD_Q8_K_XL Lossless Unsloth Quant of Deepseek-V4-Flash-0731, I am getting average 90-130 tps Prefill and 6-7.5 tps Decode. Here is my server config:

services:

llama-server:

image: ghcr.io/ggml-org/llama.cpp:server-rocm

container_name: deepseek_v4_flash_moe_hybrid

restart: unless-stopped

devices:

- /dev/kfd:/dev/kfd

- /dev/dri:/dev/dri

group_add:

- video

- render

security_opt:

- seccomp:unconfined

ports:

- 8081:8080

volumes:

- /home/hpai/Downloads/models:/models:ro

- /home/hpai/llama-cache/slots:/cache/llama-slots

command: >

-m

/models/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf

--host 0.0.0.0 --port 8080 --alias DeepSeek-V4-Flash -c 1048576 -ngl

99 --cpu-moe --threads 16 --parallel 1 --flash-attn

on --kv-unified -b 8192 -ub 4096 --slot-save-path /cache/llama-slots

--spec-type draft-dspark --spec-draft-n-max 2

networks: {}

Seems like I have no difference with dspark enabled. I didn't see a big difference between no-mmap and mmap either. Am I missing anything? Using this for my opencode setup, hoping to squeeze some more performance out of my little machine. Any ideas?

is this a steal?

ultimateslower

I’ve built 7 of these chairs. One in white oak and 6 in walnut for my dining room table. Here’s a few other pieces I’ve done.