r/llamacpp

For a while now I've been running pi entirely on my laptop -- unsloth's Gemma E4B on llama.cpp, no cloud, no API keys, nothing leaving the machine. Thinking level, image parsing, KV cache retention -- all working.

What surprised me is how genuinely useful it gets the moment it can search the web.

The tiny extension I published pi-smart-web-search adds a web_search tool with no api key needed. It fetches DDG's html output, runs it through wreq-js -> linkedom -> Defuddle (inspired by pi-smart-fetch), and then parses the output's links.

I wrote the whole setup up end-to-end as a gist -- llama.cpp, the model, a chat-template fix, pi, and the search/fetch tools — so you can reproduce the fully-local flow yourself. (links below)

I've run Gemma E4B on an M4 MacBook Air (16GB) and on a M2 MacBook Pro (32GB), but haven't tested Linux yet (I don't have a machine with a dedicated GPU), so if you run it there I'd love a report.

I'm not overselling it, just genuinely after feedback on the approach: the DDG scrape, small-model agents, anything you'd do differently.

I call the project 'Humble Pi' :shrug: -- And I look forward to the next 4b model.

Links

• Full local setup (gist): https://gist.github.com/joematthews/d02639bcbe0e0c1c404f5d3a64c3c06f
• Extension: https://github.com/joematthews/pi-smart-web-search
• Install: pi install npm:pi-smart-fetch && pi install npm:pi-smart-web-search

CORRECTION: When I made this post, I forgot to include the instructions for the custom jinja template for the E4B model in the gist. The template that ships with E4B drops prior thinking blocks from history, which forces the KV cache to recompute the entire conversation on every turn.

If you've already followed the gist, please switch to the updated llamagemma4b alias and follow the new "Install the E4B reasoning template (one time)" section to download the template — then re-run source ~/.zshrc and restart the server.

TLDR: Using the custom jinja template should improve the speed substantially.

Edited : "Qwen3.6-27B Q4_K_M on a single RTX 3090: native 256K context at 38.6 tok/s with 72 MiB of resident KV, needle recall 88-100% at 6% residency, harness accuracy unchanged (36/36 vs full cache)."

On the same hardware, generation speeds doubled and VRAM usage dropped significantly (21GB to 17.5GB) while maintaining full context accuracy

Yt video of fahd --> https://youtu.be/8rTVCRWvRDo?si=MYiVrQQltbSsMAOP

Link to git hub - https://github.com/Luce-Org/lucebox-hub/tree/main/optimizations/kvflash

Quality loss?? --> "Quality verdict (harness ground truth, base-vs-base control included): full results in RESULTS.md. Outputs are not guaranteed byte-identical to the full cache on long generations (the masked kernel path rounds differently — a different deterministic lineage), but correctness is identical: 36/36 vs 36/36 across HumanEval, GSM, MATH, and agent suites."

I recently released MemoriaForge, a lightweight C++ wrapper around llama.cpp that aims to make local LLM integration simple and straightforward.

It provides an easy-to-use API for loading GGUF models, managing conversations, injecting context, and generating responses without dealing directly with llama.cpp internals.

Small example:

#include <MemoriaForge.h>

int main() {

    MemoriaForge::LLMSession llm("models/model.gguf");

    llm.chat("Hello!");

    return 0;
}

The project has just reached version 1.0.0 and I'd love to hear your feedback, suggestions, ideas, or criticism.

Here is the repo: https://github.com/canuconde/MemoriaForge

whats your take?

do you manage to get single developer/person workflows spawning subagents to gain from the parallel-optimized engines?

from:

https://github.com/murataslan1/local-ai-coding-guide/blob/main/guides/runner-comparison.md

Are you a single developer on desktop?

├─ Yes → Do you want simplicity? → Ollama

│ Want fine control? → llama.cpp

│

└─ No → Running a team server?

├─ High throughput needed → vLLM

└─ Structured JSON outputs → SGLang