Reliable local benchmark to find the best LLM and configuration for my PC?
Is there a reputable benchmarking program that runs on your own PC and helps determine which local LLMs it can realistically handle?
My setup:
Intel Core i7-9700K
NVIDIA RTX 4070
32 GB DDR4 RAM
Windows 10 with Ubuntu through WSL2
Currently using Ollama
My main use cases are coding assistance, debugging, reviewing code, and drafting pull requests.
Ideally, I’d like something that tests actual performance and considers:
Different models and quantization levels
Alternative inference runtimes and third-party optimization tools
Context length, GPU/CPU offloading, and KV-cache settings
Tokens per second, time to first token, RAM/VRAM usage, and stability
Output quality on coding tasks, alongside speed
I’m especially interested in comparing Qwen and Gemma models and finding a practical balance between quality, speed, and memory usage.
Does a reliable tool already cover most of this, or would you recommend a combination of benchmarking tools and a repeatable testing workflow? Open-source options would be ideal. Links to projects and results from similar hardware would be appreciated