r/LocalAIStack

Hi all, a bunch of performance improvements have been landed in audio.cpp.

The biggest highlight is Higgs Audio TTS, which now runs with around 6 GB VRAM, a 48% reduction in peak memory usage compared to the previous implementation. Thanks to https://github.com/mirek190

We also made some models significantly faster, especially HTDemucs on GPU and PocketTTS on CPU.

No compromises in parity and correctness.

Here's a summary of the improvements:

Model	Peak memory reduction	Speedup
Higgs Audio TTS	48% VRAM	1.01–1.09× CUDA
ACE-Step family	6–7% VRAM	1.06–1.08× CUDA, 1.16–1.20× Vulkan
MOSS-TTS v1.5 cloning	21% VRAM	1.05× CUDA
MOSS-TTSD Q8 cloning	11% VRAM	1.04× CUDA
Echo-TTS (Memory Saver)	20% VRAM	—
Qwen3-TTS	16–20% VRAM	—
IndexTTS2 / 2.5	12% VRAM	—
HTDemucs	—	2.21× CUDA, 1.95× Vulkan
HTDemucs six-stem	—	1.99× CUDA
PocketTTS	9% RAM	2.23× CPU

They're runtime-level optimizations that make existing models more practical to run locally.

The WebUI now includes an experimental generation history feature that lets you revisit previous outputs and restore their settings.

audio.cpp now supports 110+ audio model families and 190+ variants (and counting)! We're continuing to improve memory efficiency and inference speed across CUDA, Vulkan, Metal, AMD/HIP, and CPU. The next release will bring even more optimizations!

We're also looking for contributors to help improve the audio.cpp WebUI. With so many models and features now supported, we'd love some help making the UI more polished, intuitive, and enjoyable to use. If you're interested in frontend development or UI/UX design, contributions are very welcome!

Thanks to everyone contributing improvements, testing builds, and reporting issues. Curious how these changes work on your setup!

I made BigMoeOnEdge+ (BigMoeOnEdge-plus), a fork of Helldez's BigMoeOnEdge, the engine that streams MoE experts from flash so models bigger than RAM can run on-device.

It adds a full Android chat app on top: GGUF models via llama.cpp, thinking output, persistent chats, image/video input, web search, artifacts, and advanced generation settings.and a MUCH better UI.

Disclosure: I built this fork with the help of AI tools.

Repo (Apache 2.0): https://github.com/hamodexd217/BigMoeOnEdge-plus Original project: https://github.com/Helldez/BigMoeOnEdge

Feedback and issues welcome.

Something like Oxide, but for Java: a way to access the CUDA ecosystem directly from the JVM. The idea behind TornadoVM is to JIT-compile Java code into CUDA kernels while also enabling hybrid Java/CUDA applications that can interoperate with native CUDA libraries such as cuBLAS, cuDNN, and other NVIDIA libraries. Rather than replacing CUDA, the goal is to give Java developers first-class access to the CUDA software stack, combining high-level Java productivity with the performance and flexibility of native GPU computing.

Whats your thoughts?

https://github.com/beehive-lab/TornadoVM

https://www.tornadovm.org/

I built LlamaStash to scratch a personal itch: I run local models through llama.cpp on AMD Strix Halo and got tired of writing the same llama-server wrapper script for the tenth time.

Ollama and LM Studio both wrap llama.cpp but hide too much (and cost real performance). Raw llama-server is fast but tedious. LlamaStash is the middle ground.

What it does:

• llamastash init — first-run wizard. Detects your hardware (CUDA / ROCm-HIP / Metal / Vulkan / CPU), installs llama-server, scans your existing HuggingFace / Ollama / LM Studio model caches, recommends a GGUF that fits your VRAM, downloads it, writes a tuned config, smoke-launches it.
• TUI + CLI + daemon + OpenAI-compatible proxy in one Rust binary. The proxy at 127.0.0.1:11435/v1 lets OpenCode, Cline, the OpenAI SDKs, and llm-cli work as-is. There's also an opt-in --ollama-compat mode that takes port 11434 and answers the byte-exact "Ollama is running" handshake.
• Multi-model concurrency with per-model port allocation, /health-probed state machine, intelligent context auto-fit (sidesteps llama.cpp's --fit collapse on Linux iGPUs).
• Agent-friendly CLI: every TUI capability has a CLI subcommand, --json is a stable agent contract, documented exit codes per failure class.
• In-TUI HuggingFace browser with search, sort, paginate, per-file hardware fit, download with cancel.

On performance — this is the part that matters for this sub.

LlamaStash spawns the unmodified upstream llama-server. So the wrapper should add zero overhead. I measured it. Across AMD APU (Ryzen AI Max+ 395), Apple Silicon, and NVIDIA, on four model sizes (small E2B Q4, mid 31B Q4, large 27B Q8, large MoE 35B-A3B Q8), every cell matches raw llama-server within ≤1%.

Cross-tool numbers on AMD APU (decode tok/s / TTFT ms on chat_turn):

Tool	small	mid	large_dense	large_moe
LlamaStash	86.9 / 51	9.8 / 467	7.4 / 417	42.6 / 181
raw llama-server	86.0 / 51	9.9 / 468	7.4 / 414	42.7 / 186
LM Studio 2.16.0	91.1 / 187	11.6 / 1477	7.9 / 1274	37.0 / 683
Ollama 0.24.0	50.4 / 223	4.8 / 1092	2.6 / 1745	12.1 / 476

LM Studio wins decode on small/mid/large_dense (their Vulkan path is well-tuned on gfx1151) but loses on the MoE and pays a 1-1.5s TTFT tax from its OpenAI shim. Ollama is consistently slower, and its RAG prefill is catastrophic (cold prefill every rep — 4 min on a 31B). Mac and NVIDIA tables are in the benchmarks page.

Methodology, variance gates, fairness rules, and per-cell JSONs are all checked in. The harness is reproducible: make bench-end-to-end. Tear it apart.

What it's not:

• Not an Ollama fork or replacement (though --ollama-compat exists for tools that auto-detect Ollama).
• Not a model hub.
• Not a llama.cpp fork. Same upstream binary.
• Not a hosted service. Loopback-only in 0.0.2. LAN + auth + TLS are on the roadmap.

Install:

curl -fsSL https://llamastash.dev/install.sh | sh   # macOS + Linux one-shot
irm https://llamastash.dev/install.ps1 | iex        # Windows 11 (PowerShell, no admin)
scoop bucket add llamastash https://github.com/llamastash/scoop-llamastash && scoop install llamastash
brew install llamastash/llamastash/llamastash       # Homebrew (macOS + Linuxbrew)
yay -S llamastash                                   # Arch Linux (AUR — source build)
yay -S llamastash-bin                               # Arch Linux (AUR — prebuilt binary)
yay -S llamastash-git                               # Arch Linux (AUR — main checkout)
cargo install llamastash                            # any Rust toolchain

Then llamastash init and you're up.

Platform: Linux (x86_64, aarch64), macOS (Intel, Apple Silicon), Windows 11 (x86_64). aarch64-pc-windows-msvc and Windows AMD GPU detection on the roadmap.

Honest tradeoffs: Single-author project. Bug reports especially welcome on hardware I don't own. The OpenAI-compat surface covers chat/completions, embeddings, rerank; Anthropic /v1/messages shim is coming.

Repo: https://github.com/llamastash/llamastash

Blog post with the full story: https://deepu.tech/introducing-llamastash

Benchmark methodology: https://deepu.tech/benchmarking-llamastash

Happy to answer questions in the thread.