You have requested an option to log the telemetry data (GPU/VRAM usage, VRAM bandwidth, temperature, power, fan speed, PCIe bandwidth etc.) from hw-smi to a file. Today I have implemented exactly that. Have fun montoring your hardware and applications, be it Intel Arc (Pro) or any other Nvidia/AMD GPUs on Windows and Linux! 🖖
Good evening all, I am a recent convert, coming over from the stacked RTX GPU club (5070Ti & 5060Ti). Sunday I installed a pair of B70 Pros after my 5070 laid over on me. SYCL running with Qwen3.6:27b Q8. Two days of pretty steady work, overnight spine runs 5-7 hours depending on daily activity.
Today, I was running a catchup process during work from downtime Saturday and mid day Sunday. The process completed without issue, GPUs went silent, and 7 minutes later the system crashed with dgxkrnl.sys crash. Reboot and health check passed, but not sure why it crashed while idle?
For reference, the GPU that crashed was in a different slot than the original 5070 was, so I don't think it is a robot issue. All drivers up to date, firmware up to date. If anyone has any insight or a similar experience and could offer a bit of advice I would be appreciative. I was just gearing up to smoke test 3.8:27b Q8, but that is on hold until I get back to normal.
Arc Power 1.0.2 - is now officially released, and with that comes the official Download and Release on Github.
This Version includes:
- Hotfix for the Core and Voltage Offset not applying in the initial Release.
- Stock Overlay Theme Classic & Arc - Arc is a new one having an Intel Arc inspired look, not vastly different from Classic.
- Advanced Overlay: Advanced Overlay is an Overlay inspired by AMDs Adrenaline type Overclock / Settings Overlay, to change OC Settings on the fly. Current hotkey = "CTRL + P" -> Changeable via Settings.
- Overlays now won't pop-up on Boot/Startup anymore, only via Shortcuts.
Website: https://yamsse.github.io/Arc-Power/
Download: https://github.com/YamsSE/Arc-Power/releases/latest
Hey everyone,
There’s a massive misconception that if you aren't dropping $2,000 on an NVIDIA GPU, you can't run serious Local AI workflows. I wanted to see how far I could push a consumer AMD card, and therefore bought a rx7800xt 16b VRAM.
Right now, my workstation node is running llama-server hosting a DENSE 27B model -> Qwopus3.6-27B-v2-Q3_K_S.gguf (12 GB) and Qwen3.6-35B-A3B-UD-IQ3_XXS.gguf (13 GB Mixture of Experts, 3B active parameters per token) continuously. I am regularly feeding it contexts that reach 91k to 128k tokens in my daily workflows.
Here is the exact setup, compiler parameters and optimization flags.
THE COMPILER BUILD
To get flash attention and RDNA3 optimizations working correctly on ROCm 6.4.4, I built llama.cpp from source using these specific cmake flags:
cmake -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1101 -DrocWMMA_FATTN=ON
cmake --build build --config Release
This targets the gfx1101 architecture of the RX 7800 XT directly and compiles support for hardware-accelerated Flash Attention kernels.
THE EXACT RUNTIME FLAGS
My systemd service runs the server with this exact command line:
llama-server --host localhost --port 8080 --api-key xxxx --parallel 1 --n-gpu-layers 99 --batch-size 512 --ubatch-size 128 --flash-attn on --cache-type-k q8_0 --cache-type-v q4_0 --ctx-size 131072 --reasoning off --sleep-idle-seconds 300 --cache-prompt --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0 --presence-penalty 1.5 --repeat-penalty 1
HOW I CRUSHED THE VRAM LIMIT: KV CACHE QUANTIZATION
A model like Qwopus 27B or Qwen 35B MoE fits in 16GB VRAM at a small context size. But at 128K context, the raw FP16 Key-Value (KV) cache alone would consume upwards of 32 GB of VRAM, making it impossible to run on consumer hardware.
To solve this, we split and quantize the cache:
- Key cache is quantized to 8-bit (q8_0) using --cache-type-k q8_0
- Value cache is quantized to 4-bit (q4_0) using --cache-type-v q4_0
This compresses the memory footprint of the KV cache by roughly 5.6x. Thanks to this optimization, the entire active model weights plus the 128K token KV cache sit comfortably in VRAM, utilizing exactly 96% of the 7800 XT's memory. No layers spill into slow system RAM, avoiding the PCIe transfer bottleneck entirely.
THE MATH BEHIND 128K CONTEXT: YaRN ROPE SCALING
Qwopus and Qwen architectures use Rotary Position Embeddings (RoPE). Because these models have a base context window smaller than 128K, running at 131,072 tokens requires positional frequency scaling.
Instead of basic linear scaling (which stretches all frequencies equally and destroys the model's short-range spelling and grammatical coherence), llama.cpp utilizes YaRN (Yet another RoPE extensioN).
YaRN divides the embedding dimensions into three frequency bands:
This uneven scaling prevents the attention entropy and perplexity from exploding. In practice, the model remains highly coherent and retains logical consistency even at 91k+ tokens.
REAL-WORLD TELEMETRY AND SPEED
During heavy prompt processing, the card maintains solid throughput:
- Prefill speed: ~210 tokens/second (utilizing flash attention)
- Decode speed: 11-17 tokens/second
- GPU Power: Draws ~188W (with a systemd power cap set at 190W via rocm-smi)
- GPU Temps: Stable between 52 C and 70 C across edge, junction, and memory sensors.
If you are running consumer AMD hardware, do not settle for small context sizes. Build with ROCm, turn on Flash Attention, quantize your Key/Value cache separately, and let YaRN handle the frequency scaling.
I wrote up a detailed guide comparing these measurements, native Windows vs Linux ROCm paths, and power sweeps on my blog here: https://sergiiob.dev/posts/rx7800-xt-llama-cpp-benchmarks-moe-context
I share my daily telemetry runs, local model benchmarks, and hardware configurations on X. If you want to see live updates and benchmarks, follow along here: https://x.com/SergiiioBS