Honestly 99% of my problems across all models cloud or local are around quality control of remembered data how it relates to produced works that could probably be solved with some additional layers of consideration. Using xhigh setting in yaml helps.
Always learning.
Qwen 3.6 27b
If you own an AMD 7900 XTX and you are sick of the official ROCm releases ignoring consumer silicon; stop waiting for AMD to fix it. I spent the last few days manually auditing the build process; and the result is a massive performance jump.
The standard drivers and prebuilt ROCm stacks are optimized for server grade cards (like the MI300); often pushing consumer GPUs into inefficient code paths. By building from the source repository (rocm/therock) and pruning the distribution bloat; you get a runtime tailored to your specific silicon.
I am now pushing 155k context windows with 23GB VRAM allocation using this deployment strategy; and the latency is significantly lower than the standard stack.
This script initializes a volatile ramdisk for high speed KV caching; redirects the runtime to your custom build; and tunes the engine threading to prevent CPU bottlenecking.
#!/bin/bash
# Configuration
# Replace these placeholders with your actual directory paths
SERVER_PATH="/path/to/llama-server"
RAMDISK_DIR="/mnt/ramdisk/slots"
LOG_DIR="/mnt/ramdisk/telemetry"
INSTALL_ROOT="/path/to/TheRock/install"
MIN_REQUIRED_MB=10240
# 1. Setup Volatile Ramdisk (16GB Matrix)
sudo mkdir -p /mnt/ramdisk
sudo mount -t tmpfs -o size=16G tmpfs /mnt/ramdisk
sudo mkdir -p "$RAMDISK_DIR"
sudo mkdir -p "$LOG_DIR"
sudo chown $USER:$USER "$RAMDISK_DIR"
sudo chown $USER:$USER "$LOG_DIR"
# 2. Launch Background Telemetry Server
echo "[SYSTEM] Initializing live debug telemetry on port 8082..."
python3 -m http.server 8082 --directory "$LOG_DIR" > /dev/null 2>&1 &
TELEMETRY_PID=$!
# 3. Cleanup Trap
cleanup() {
echo "[SYSTEM] Purging volatile cache; killing telemetry; and unmounting..."
kill $TELEMETRY_PID 2>/dev/null
sudo umount /mnt/ramdisk
exit
}
trap cleanup EXIT SIGINT SIGTERM
echo "[SYSTEM] Deploying architecture (Context: 155648; VRAM Target: 23GB)..."
# Environment Overrides for Custom ROCm Build
export LD_LIBRARY_PATH="$INSTALL_ROOT/lib:$INSTALL_ROOT/lib64:$LD_LIBRARY_PATH"
export ROCM_PATH="$INSTALL_ROOT"
export HIP_PATH="$INSTALL_ROOT"
export HIP_VISIBLE_DEVICES=0
export HSA_FORCE_FINE_GRAIN_AMDGPU=1
export HSA_XNACK=1
# Execution
exec "$SERVER_PATH" \
-m /path/to/models/model.gguf \
-c 155648 \
--n-gpu-layers 60 \
--numa distribute \
--flash-attn on \
--no-warmup \
--port 8081 \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
-np 1 \
--cache-ram 16384 \
--slot-save-path "$RAMDISK_DIR" \
--log-file "$LOG_DIR/engine_debug.log" \
--batch-size 512 \
--threads 12 \
--mmproj /path/to/models/mmproj.gguf \
--image-min-tokens 1024 \
--reasoning on \
--swa-checkpoints 0 \
--ctx-checkpoints 69 \
--cont-batching \
--no-kv-unified \
--spec-type none \
--cache-idle-slots \
--host 0.0.0.0
Whatever has changed this week blows, revert or I'm out.