r/llamacpp
Mini pc set up
I bought a bosgame p3 lite (32gb ram with a Radeon 680M igpu and Ryzen 7 6800H)
Just created a fresh cachyOS boot.
My goal is to run llms as fast as possible. Heard good things of llama.cpp.
Been looking a few guides and asked a few llms but they've been giving me some pretty weird instructions, from tampering with the BIOS to installing a few bits and bobs.
Can anyone give me a hand or some resources? Really want to avoid doing anything too stupid.
When I am not using it for inference I also want to use it for the odd videogame use so I don't intensely want to mess around with the bios or igpu setting blindly.
Any help would be very appreciated!!!
u/Crafty-Sell7325 — 2 days ago
I built a macOS GUI for llama-server because I kept retyping the same command
Disclosure up front: this is my own project.
Two things pushed me into building it. I kept retyping the same llama-server invocation with three values changed, and I watched curl -C - fail to resume a 20 GB download one too many times.
It lists the GGUF files in my models folder and reads the headers directly, so the quant, the context length and whether it is MoE come from the file rather than from the filename. Opening one shows the exact command before it runs. While it is serving I get KV cache, tokens per second in both directions, memory pressure and swap in one place, plus a Test model button that hits the server for real — health, model list, alias, a chat completion, streaming — so I know it works instead of assuming it does.
Downloads pull from Hugging Face in four ranged segments, resume from a sidecar after a kill, verify sha256, and queue rather than refusing a second URL.
It has no chat interface of its own and is not getting one. A running model opens llama.cpp's own web UI in a second window.
Caveats: macOS only, and an unsigned beta, so the first launch is blocked and you have to allow it through System Settings > Privacy & Security — the README has the steps. It needs llama-server and does not ship it. There is a universal build but no Intel Mac has ever run it; if you have one I would like to hear what happens, particularly whether your llama-server has a GPU for the default -ngl all.
u/Current-Quail-2503 — 12 days ago
▲ 2 r/llamacpp
How would you benchmark 50+ local LLMs without going insane?
I feel like I stepped into a time capsule after the ChatGPT-3 days. 😅 I finally built a decent PC (Ryzen 7 7700, 32GB RAM, No GPU), discovered llama.cpp and somehow ended up downloading 50+ GGUF models.
Now I'm stuck with decision paralysis.
I mainly use them for coding (JavaScript, React, TypeScript, debugging, reasoning), but I'm also new to the whole local AI ecosystem. I still don't know much about agentic frameworks or coding agents. I mostly just load a GGUF and chat with it using the llama.cpp web interface on localhost:8080
The collection includes Qwen, Gemma, Granite, DeepSeek, Phi, Mistral, Llama, LiquidAI, SmolLM, Hunyuan, Nemotron, and a few community fine-tunes.
My first idea was to make a Markdown table and score every model manually, but it feels like I'm accidentally trying to invent my own benchmarking system.
Surely I'm not the first person to hit this problem.
How do you guys compare local models? Are there any practical benchmark suites, GitHub projects, or workflows for deciding what stays on your SSD and what gets deleted?
I'd love to hear how you approached it when you were starting out.
u/Sakif_Hossain — 13 days ago