Performance issue: Low token generation (~20 tok/s vs 50 tok/s) on Radeon AI PRO R9700 (gfx1201) with vLLM ROCm & Gemma 4-26B
I’m testing cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit on a single AMD Radeon AI PRO R9700 (32 GB, gfx1201) using vLLM ROCm.
Current result: about 19–20 generated tok/s after warmup, with a short single-user decode benchmark. I expected something closer to 50 tok/s based on a published R9700 result for this model.
My current vLLM setup:
- vLLM: 0.22.1rc1.dev499+g470229c37.d20260613
- model: cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit
- TP: 1
- max-num-seqs: 1
- max-model-len: 8192
- gpu-memory-utilization: 0.90
- kv-cache-dtype: fp8
- attention backend: TRITON_ATTN
- --enforce-eager
- --language-model-only
I see this warning on startup:
> Using default MoE config. Performance might be sub-optimal!
> Config file not found: E=128,N=704,device_name=AMD-gfx1201,dtype=int4_w4a16.json
CPU is a Ryzen 5 3600 with 32 GB DDR4-2666 (Upgrading to 900 series soon), but GPU memory is not full and RAM has ~22 GB available.
Is there an existing or recommended tuned MoE config for gfx1201, E=128, N=704, and int4_w4a16?
Is this model known to be slow on R9700 with Triton W4A16 / current ROCm vLLM?
What exact vLLM/ROCm image, flags, and benchmark method produced ~50 tok/s?
Is --enforce-eager costing meaningful decode performance on gfx1201, or is the missing MoE config the main issue?
I also tried Lemonade’s portable gfx120X runtime, but it ships gfx1200 kernel packs and fails on this gfx1201 card with hipErrorInvalidImage.