u/deepu105

TL;DR: On a Strix Halo laptop (Radeon 8060S, 128 GB unified), MTP takes Qwen3.8-27B Q8_0 from 7.1 to 21.9 tok/s. The julianmb/q38rocm FP4 fork gives 26.7 tok/s.

Setup: LlamaStash driving stock llama.cpp b10503 (5112b9738) for ROCm and Vulkan, and the q38rocm fork (server build 246) for ROCmFP4, all built from git today. Greedy, 256 max tokens, wall-clock tok/s from the API usage numbers, best of 2 reps, on AC power. Everything launched through my tool LlamaStash (v0.2.0) so the flags are reproducible.

Qwen3.8-27B decode tok/s (code prompt / prose prompt):

Build and model	MTP off	MTP on	Draft acceptance
ROCm + Q8_0 (29 GB)	7.1	21.9 (draft 5)	73%
Vulkan + Q8_0 (29 GB)	7.4	21.3 (draft 5)	73%
ROCmFP4 FAST (14.6 GB)	12.5	26.7 (draft 6)	86%

Draft length sweep, code prompt tok/s:

draft n	2	3	4	5	6	7	8
Q8_0	15.4	18.4	19.1	21.9	21.2	21.8	20.5
FP4	-	23.3	24.9	25.0	26.7	25.6	-

Observations:

• MTP really makes a difference 2.5x to 3.1x on Q8_0, 2.1x on ROCmFP4, at 73-95% draft acceptance. The model verifies every token, so quality does not change on stock llama.cpp.
• Draft length of 5 works best. The backend default is 3; 5 was best for Q8_0 and 6 for FP4.
• FP4 is a double win if you can live with the quant. Half the memory (29 to 14.6 GB), 1.8x faster raw decode without MTP, and 26.7 tok/s with MTP, which is 3.8x my stock Q8_0 baseline.
• Engine choice barely matters for decode at this size. ROCm, Vulkan and the rocWMMA build all land within about 3%. Vulkan does lose roughly half its prompt processing throughput though, so ROCm stays my default.
• Stuff that did not help: KV cache q8_0 (helps at draft 3, slightly worse at 5), flash attention off (minus 5%).
• One caveat on the ROCmFP4 fork: completion lengths drifted a little across draft settings (152-158 tokens for the same prompt) while stock llama.cpp was stable at 164, so the fork's speculative path is not strictly greedy-identical yet.

LlamsStash defaults were used for all runs. The only flags set were to enable disable MTP ( --spec-type draft-mtp), set draft length (--spec-draft-n-max N) and the --fit-ctx 16384 flag was used to ensure the GPU layers were fully utilized.

The below are the default flags that was applied. Some sent by LlamaStash and others are llamacpp defaults.

Flag	Value	Set by
--host	127.0.0.1	LlamaStash (loopback policy)
--port	41100	LlamaStash (auto-assigned)
-m	<model>.gguf absolute path	LlamaStash (resolves the name ref)
--mmproj	mmproj-F16.gguf	LlamaStash (auto-paired sidecar; Q8_0 rows only, FP4 has none)
--jinja	on	LlamaStash (chat template handling)
--spec-type	draft-mtp	LlamaStash (translated from --mtp on)
--spec-draft-n-max	5 (Q8_0) / 6 (FP4 best) / omitted on default rows	LlamaStash (--mtp-draft-n); llama.cpp default 3 when omitted
--fit-ctx	16384	LlamaStash (ctx knob = auto, fit hint)
--cache-type-k / -v	f16 (q8_0 on kvq8 variant rows)	llama.cpp default except for kvq8 runs
--flash-attn	on (off on the fa-off variant row only)	llama.cpp default except for fa-off runs
--n_ctx (final)	262144 per slot (native 256K)	llama.cpp fitter
--n_parallel	4, kv_unified true	llama.cpp (auto)
--n_gpu_layers	all (-1)	llama.cpp fitter (llamastash knob was auto)
--cache-type-k/v (final)	f16 / f16	llama.cpp default
--n_threads	16 (of 32 logical)	llama.cpp default
--n_batch / --n_ubatch	server defaults, not overridden	llama.cpp
spec internals (fork only)	n_min=0, p_min=0.75, backend_sampling=1	llama.cpp (q38rocm fork defaults)
draft context (fork only)	all GPU layers, f16 KV	llama.cpp (fork)

Ran a comparison across three hardware families and four model sizes (0.6B, 8B, 30B-class, 30B+ MoE). Measured TTFT (cold and warm) and decode tokens/sec. Did it twice: once with matched llama.cpp flags, once with each tool's defaults.

What I found

• Out-of-the-box, Ollama is 41-72% slower decode on AMD APU than raw llama.cpp; cold-RAG prefill on a 31B model on Strix Halo took roughly 4 minutes
• LM Studio's Vulkan path wins decode on small/mid models, but pays a 1-1.5 second TTFT tax
• At matched flags, Ollama and llama.cpp largely converge (with a few exceptions)
• A thin launcher around llama.cpp adds <1% overhead and 0.45 ms median TTFT on the proxy hop

Disclosure: the thin launcher is LlamaStash, which I built. I used it as the bench harness because it spawns unmodified upstream llama-server.

Full write-up with charts: https://deepu.tech/benchmarking-llamastash/

Per-cell JSONs and the harness are in the repo. Reproducible with make bench-end-to-end on hardware you have.

Curious what you find on hardware I do not own.

I built LlamaStash to scratch a personal itch: I run local models through llama.cpp on AMD Strix Halo and got tired of writing the same llama-server wrapper script for the tenth time.

Ollama and LM Studio both wrap llama.cpp but hide too much (and cost real performance). Raw llama-server is fast but tedious. LlamaStash is the middle ground.

What it does:

• llamastash init — first-run wizard. Detects your hardware (CUDA / ROCm-HIP / Metal / Vulkan / CPU), installs llama-server, scans your existing HuggingFace / Ollama / LM Studio model caches, recommends a GGUF that fits your VRAM, downloads it, writes a tuned config, smoke-launches it.
• TUI + CLI + daemon + OpenAI-compatible proxy in one Rust binary. The proxy at 127.0.0.1:11435/v1 lets OpenCode, Cline, the OpenAI SDKs, and llm-cli work as-is. There's also an opt-in --ollama-compat mode that takes port 11434 and answers the byte-exact "Ollama is running" handshake.
• Multi-model concurrency with per-model port allocation, /health-probed state machine, intelligent context auto-fit (sidesteps llama.cpp's --fit collapse on Linux iGPUs).
• Agent-friendly CLI: every TUI capability has a CLI subcommand, --json is a stable agent contract, documented exit codes per failure class.
• In-TUI HuggingFace browser with search, sort, paginate, per-file hardware fit, download with cancel.

On performance — this is the part that matters for this sub.

LlamaStash spawns the unmodified upstream llama-server. So the wrapper should add zero overhead. I measured it. Across AMD APU (Ryzen AI Max+ 395), Apple Silicon, and NVIDIA, on four model sizes (small E2B Q4, mid 31B Q4, large 27B Q8, large MoE 35B-A3B Q8), every cell matches raw llama-server within ≤1%.

Cross-tool numbers on AMD APU (decode tok/s / TTFT ms on chat_turn):

Tool	small	mid	large_dense	large_moe
LlamaStash	86.9 / 51	9.8 / 467	7.4 / 417	42.6 / 181
raw llama-server	86.0 / 51	9.9 / 468	7.4 / 414	42.7 / 186
LM Studio 2.16.0	91.1 / 187	11.6 / 1477	7.9 / 1274	37.0 / 683
Ollama 0.24.0	50.4 / 223	4.8 / 1092	2.6 / 1745	12.1 / 476

LM Studio wins decode on small/mid/large_dense (their Vulkan path is well-tuned on gfx1151) but loses on the MoE and pays a 1-1.5s TTFT tax from its OpenAI shim. Ollama is consistently slower, and its RAG prefill is catastrophic (cold prefill every rep — 4 min on a 31B). Mac and NVIDIA tables are in the benchmarks page.

Methodology, variance gates, fairness rules, and per-cell JSONs are all checked in. The harness is reproducible: make bench-end-to-end. Tear it apart.

What it's not:

• Not an Ollama fork or replacement (though --ollama-compat exists for tools that auto-detect Ollama).
• Not a model hub.
• Not a llama.cpp fork. Same upstream binary.
• Not a hosted service. Loopback-only in 0.0.2. LAN + auth + TLS are on the roadmap.

Install:

curl -fsSL https://llamastash.dev/install.sh | sh   # macOS + Linux one-shot
irm https://llamastash.dev/install.ps1 | iex        # Windows 11 (PowerShell, no admin)
scoop bucket add llamastash https://github.com/llamastash/scoop-llamastash && scoop install llamastash
brew install llamastash/llamastash/llamastash       # Homebrew (macOS + Linuxbrew)
yay -S llamastash                                   # Arch Linux (AUR — source build)
yay -S llamastash-bin                               # Arch Linux (AUR — prebuilt binary)
yay -S llamastash-git                               # Arch Linux (AUR — main checkout)
cargo install llamastash                            # any Rust toolchain

Then llamastash init and you're up.

Platform: Linux (x86_64, aarch64), macOS (Intel, Apple Silicon), Windows 11 (x86_64). aarch64-pc-windows-msvc and Windows AMD GPU detection on the roadmap.

Honest tradeoffs: Single-author project. Bug reports especially welcome on hardware I don't own. The OpenAI-compat surface covers chat/completions, embeddings, rerank; Anthropic /v1/messages shim is coming.

Repo: https://github.com/llamastash/llamastash

Blog post with the full story: https://deepu.tech/introducing-llamastash

Benchmark methodology: https://deepu.tech/benchmarking-llamastash

Happy to answer questions in the thread.

I built LlamaStash to scratch a personal itch: I run local models through llama.cpp on AMD Strix Halo and got tired of writing the same llama-server wrapper script for the tenth time.

Ollama and LM Studio both wrap llama.cpp but hide too much (and cost real performance). Raw llama-server is fast but tedious. LlamaStash is the middle ground.

What it does:

• llamastash init — first-run wizard. Detects your hardware (CUDA / ROCm-HIP / Metal / Vulkan / CPU), installs llama-server, scans your existing HuggingFace / Ollama / LM Studio model caches, recommends a GGUF that fits your VRAM, downloads it, writes a tuned config, smoke-launches it.
• TUI + CLI + daemon + OpenAI-compatible proxy in one Rust binary. The proxy at 127.0.0.1:11435/v1 lets OpenCode, Cline, the OpenAI SDKs, and llm-cli work as-is. There's also an opt-in --ollama-compat mode that takes port 11434 and answers the byte-exact "Ollama is running" handshake.
• Multi-model concurrency with per-model port allocation, /health-probed state machine, intelligent context auto-fit (sidesteps llama.cpp's --fit collapse on Linux iGPUs).
• Agent-friendly CLI: every TUI capability has a CLI subcommand, --json is a stable agent contract, documented exit codes per failure class.
• In-TUI HuggingFace browser with search, sort, paginate, per-file hardware fit, download with cancel.

On performance — this is the part that matters for this sub.

LlamaStash spawns the unmodified upstream llama-server. So the wrapper should add zero overhead. I measured it. Across AMD APU (Ryzen AI Max+ 395), Apple Silicon, and NVIDIA, on four model sizes (small E2B Q4, mid 31B Q4, large 27B Q8, large MoE 35B-A3B Q8), every cell matches raw llama-server within ≤1%.

Cross-tool numbers on AMD APU (decode tok/s / TTFT ms on chat_turn):

Tool	small	mid	large_dense	large_moe
LlamaStash	86.9 / 51	9.8 / 467	7.4 / 417	42.6 / 181
raw llama-server	86.0 / 51	9.9 / 468	7.4 / 414	42.7 / 186
LM Studio 2.16.0	91.1 / 187	11.6 / 1477	7.9 / 1274	37.0 / 683
Ollama 0.24.0	50.4 / 223	4.8 / 1092	2.6 / 1745	12.1 / 476

LM Studio wins decode on small/mid/large_dense (their Vulkan path is well-tuned on gfx1151) but loses on the MoE and pays a 1-1.5s TTFT tax from its OpenAI shim. Ollama is consistently slower, and its RAG prefill is catastrophic (cold prefill every rep — 4 min on a 31B). Mac and NVIDIA tables are in the benchmarks page.

Methodology, variance gates, fairness rules, and per-cell JSONs are all checked in. The harness is reproducible: make bench-end-to-end. Tear it apart.

What it's not:

• Not an Ollama fork or replacement (though --ollama-compat exists for tools that auto-detect Ollama).
• Not a model hub.
• Not a llama.cpp fork. Same upstream binary.
• Not a hosted service. Loopback-only in 0.0.2. LAN + auth + TLS are on the roadmap.

Install:

curl -fsSL https://llamastash.dev/install.sh | sh   # macOS + Linux one-shot
irm https://llamastash.dev/install.ps1 | iex        # Windows 11 (PowerShell, no admin)
scoop bucket add llamastash https://github.com/llamastash/scoop-llamastash && scoop install llamastash
brew install llamastash/llamastash/llamastash       # Homebrew (macOS + Linuxbrew)
yay -S llamastash                                   # Arch Linux (AUR — source build)
yay -S llamastash-bin                               # Arch Linux (AUR — prebuilt binary)
yay -S llamastash-git                               # Arch Linux (AUR — main checkout)
cargo install llamastash                            # any Rust toolchain

Then llamastash init and you're up.

Platform: Linux (x86_64, aarch64), macOS (Intel, Apple Silicon), Windows 11 (x86_64). aarch64-pc-windows-msvc and Windows AMD GPU detection on the roadmap.

Honest tradeoffs: Single-author project. Bug reports especially welcome on hardware I don't own. The OpenAI-compat surface covers chat/completions, embeddings, rerank; Anthropic /v1/messages shim is coming.

Repo: https://github.com/llamastash/llamastash

Blog post with the full story: https://deepu.tech/introducing-llamastash

Benchmark methodology: https://deepu.tech/benchmarking-llamastash

Happy to answer questions in the thread.

I built LlamaStash, a terminal-native launcher for local LLMs via llama.cpp. Reaching 0.0.2 today (the first public release). Sharing here because the Rust stack is a meaningful part of why this works.

What it does (briefly): one Rust binary that's a TUI + CLI + daemon + OpenAI-compatible proxy. The daemon spawns llama-server instances on demand and supervises them. The CLI mirrors every TUI capability and has --json as an agent contract. The proxy at 127.0.0.1:11435/v1 lets OpenAI-compatible clients (OpenCode, Cline, the OpenAI SDKs) talk to discovered models. Init wizard handles first-run hardware detection and model setup.

The Rust stack:

• ratatui for the TUI. Five tabs, vim-style navigation, in-TUI HuggingFace browser with paginated search and per-file hardware-fit indicators. Five built-in themes plus a custom palette block. Every action rebindable from config.yaml.
• tokio for the async runtime everywhere. Daemon supervisor uses tokio tasks per model, with a custom state machine (Launching → Loading → Ready → Stopping) backed by atomic state files.
• hyper for the HTTP proxy. Loopback-only, OpenAI shape, with /api/tags, /api/version, /api/ps, /api/show for the Ollama discovery surface. Routes to the appropriate llama-server instance per model, auto-starts on first request, falls back to a Ready peer with audit headers (x-llamastash-served-by, x-llamastash-fallback-reason) on launch failure.
• crossterm for cross-platform terminal handling.
• sysinfo for host metrics. GPU detection layers on top of that with vendor-specific probes (NVIDIA/AMD-ROCm/Apple-Metal/Vulkan/CPU).
• hf-hub for HuggingFace downloads with progress bars and retry logic.
• blake3 + sha2 for file verification and artifact checksums.
• Custom streaming GGUF parser. Reads the header, architecture, params, quant, native context, chat template, and KV-cache geometry without loading the full file. Lets the supervisor compute "will this model + context fit in free VRAM" answers before spawning anything.
• Bearer-token loopback HTTP control plane. TUI/CLI talk to the daemon over 127.0.0.1 with a per-restart token + URL stashed in a runtime.json handshake file under the state dir. Same transport on Linux, macOS, and Windows. The file is 0600 on Unix and Protected DACL owner-only on Windows. Side-by-side daemon instances supported via LLAMASTASH_STATE_DIR for testing or per-user isolation. The OpenAI-compat proxy is a separate listener.
• Cross-platform process supervision. POSIX uses setsid + signals; Windows uses Job Objects with JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE (so a crashed daemon tears down its children), GenerateConsoleCtrlEvent(CTRL_BREAK_EVENT) for graceful drain, TerminateJobObject for force-kill.

~176 Rust files. No unsafe blocks in the main crate (a few are isolated to the Windows backend behind cfg(windows)). The whole thing compiles and runs on Linux (x86_64, aarch64), macOS (Intel, Apple Silicon), and Windows 11 (x86_64). aarch64-pc-windows-msvc and Windows AMD GPU detection on the roadmap.

On overhead: this is a wrapper around llama-server, so the only way it can be slower than running llama-server by hand is by adding overhead in the wrapper. I measured it against raw llama-server across AMD APU, Apple Silicon, and NVIDIA on four model sizes. Every cell matches within ≤1%. Proxy hop is +0.45 ms median TTFT, zero on decode. Full methodology and reproduction harness are in the repo.

Things I learned (or re-learned) building this in Rust:

• ratatui has matured a lot since the tui-rs days. Building this brought back KDash energy from 2020. The pieces are now real frameworks, not just primitives.
• tokio + custom state machines play nicely, but you have to be deliberate about cancellation. The supervisor state machine had to learn the difference between "kill this model" and "I'm shutting down" the hard way. Tests that simulate signal storms catch most of it.
• Atomic file writes are still the right answer for state persistence. tempfile plus rename is boring and reliable, and the daemon can re-adopt running llama-server instances after a crash because state is on disk.
• Custom protocol parsing in Rust is fun. The GGUF parser is one of my favorite parts of the codebase. Streaming through the header, validating each section, returning typed errors per malformed-field — the type system carries a lot of weight there.
• serde_yaml plus clap is still the right defaults for "config + CLI" in Rust. Some people will tell you to use TOML; some will tell you to use ron. For a user-facing config that has to be hand-edited, YAML still reads best.
• Porting Unix-socket + peercred IPC to bearer-token loopback HTTP was the right move for Windows parity. The original control plane was UnixStream + SO_PEERCRED. To get cross-platform parity without writing two transports, 0.0.2 collapsed both onto a 127.0.0.1 HTTP listener with a per-restart token in a chmod 0600 / Protected-DACL handshake file. Same auth posture (same-UID local-only), one transport implementation, and the Windows lane stopped being a special case. The lesson: when your "obvious" Unix primitive has no Windows equivalent, picking a portable transport up front is cheaper than maintaining two backends.

Repo: https://github.com/llamastash/llamastash

Blog post: https://deepu.tech/introducing-llamastash

Tradeoffs and what's not in 0.0.2: x86_64 Windows only (aarch64 Windows on the roadmap). Loopback-only proxy by design. Anthropic /v1/messages shim coming. MCP server surface planned. Windows AMD GPU detection deferred (the daemon and proxy still work, you just don't see GPU stats in the host pane).

Happy to talk about the Rust side in detail. The benchmark + cross-tool comparison is its own separate post for r/LocalLLaMA — I'll keep the discussion here Rust-flavored.