u/Acceptable-Cycle4645

Finally see a new voice conversion model. MeanVC2 supports cross-gender and cross-language voice conversion. 3x realtime on CPU with audio.cpp.

Disclaimer: The converted voice quality of MeanVC2 is decent; the noise comes from my rough demo engineering, not the model itself. This is only a quick demo to show MeanVC2 running in real time.

https://huggingface.co/ASLP-lab/MeanVC2

Hi all :)

audio.cpp release 0.6 has been out for a little while, so this is more of an update on what landed and what has been improving around it.

0.6 added 5 new model families: dots.tts, NeuTTS-2e, MuScriptor (Music to MIDI), MiniMax-H3, and SenseVoice-Small, bringing audio.cpp to 49 total model families and 70+ model variants. Since release 0.6, our contributors have added Irodori-TTS v4, IndexTTS 2.5, and ACE-Step 1.5 XL, and MiniMax-Music3.

The highlights are:

(1) Native WebUI! Many thanks to https://github.com/mirek190

(2) Of course, MiniMax-H3 and MiniMax-Music3. We implemented MiniMax-H3’s text-to-audio pipeline, and one fun use case is TTS/Voice clone/Music gen. It’s more flexible and powerful than dedicated audio models, and the performance is quite decent. Check out the multi-speaker conversation demo in the main post, along with the other demos in the comments.

What I’m very excited about with the MiniMax-H3 implementation is that it significantly enriches the framework’s building blocks for DiT models. Now with you don’t need to go through the pain of setting up SageAttention, First Block Cache, or Spectrum manually. Just change a few parameters, and you can experiment with the model. A preliminary inspection of configuration, memory, and performance trade-offs is available in repo's docs/reports/minimax_h3_performance.md

Bonus: audio.cpp’s MiniMax-H3 implementation can also produce video frames, because the DiT generates audio and video latents together, and the video VAE path is relatively straightforward to support. For now, the output is saved as RGB frame data plus metadata in JSON, so you need to encode it into a video file yourself. No upscaler or post-processing support. Just for fun.

MiniMax-Music3 is currently in preview (preview/minimax-music-3 branch) . CUDA/Vulkan/HIP were tested. Still room for optimization. VRAM usage and RTF depend on audio duration and prompt length.. The demo uses the official demo prompt (4000+ char caption and 1200 char lyrics) and 30 steps plus CFG. Under this setting VRAM is ~11 GB for 30s, 14 GB for 60s, and 17 GB for 180s. It's easy to get faster-than-real-time performance and much lower VRAM usage if you tune the setting.

(3) The source code for the IOS-friendly VibeVoice 1.5B is out on the vibevoice-optimizations branch in case you missed it.

If you want early access to models in the next release, keep an eye on the dev branch. It will usually be updated with the new release content a few days before the release for testing.

Feedback and PRs on the native UI, MiniMax-Music3, and all other aspects of audio.cpp are always welcome!

I speed up the generation part of the demo in case you get bored 😄

I also tested another long-form generation, and the VRAM usage looks stable. The demo is about a minute long, and I posted it on X.

This started as a random idea and somehow turned into a full detour from working on the next audio.cpp release. The model was uploaded to the audio.cpp HF repo. I will upload the xcframework later, and then push the code to a branch after release 0.6.

audio.cpp 0.5 is out :)

The most fun new model in 0.5 is DramaBox. It is closer to prompt-directed voice acting. DramaBox is built on the LTX-2.3 audio architecture, and prompts can control emotion, delivery, laughs, sighs, pauses, transitions, and speaker behavior.

Example input (check the audio in the post):

A nervous young man whispers, "I do not think we should be here."

He takes a shaky breath. "Did you hear that?"

The hallway answers with a slow metallic creak.

He tries to laugh, but his voice breaks. "Okay. That was probably just the wind."

Another sound comes from behind the locked door, softer this time, almost like someone breathing.

He steps back. "No. No, we are leaving now."

Then, from the darkness, a small voice whispers her name.

Confucius4-TTS is the other big voice highlight: cross-lingual voice transfer. Give it a reference voice, then synthesize in another supported language.

This release also added RVC for voice conversion, BS-RoFormer for vocal separation, GLM-TTS, Kroko ASR, Parakeet-TDT, Inflect Micro v2 (tiny but powerful), and Fun-ASR-Nano. Fun-ASR-Nano is especially exciting because it comes from the official FunASR team, and audio.cpp is now listed on the official FunASR deployment platform.

The platform story got wider too. Early HIP/ROCm support landed for AMD GPUs, Metal got faster on Apple Silicon, and the server/streaming paths became more useful for real applications with live PCM ingest and cleaner streaming transcript deltas.

None of this would be possible without contributions from our community. Contributors are showing up with new ports, backend tests, bug reports, docs, Web UI work, and production deployment feedback.

A few areas where community help would be especially valuable:

Scoped model performance optimization:

Some early model integrations were built parity-first and received less optimization work. Non-CUDA backends are also less optimized and need more focused performance work. As the number of models grows, it becomes harder to find time to backport proven performance patterns. Good contributions here are scoped, measurable optimizations: improve one model path, show before-and-after benchmarks, and gate aggressive changes behind perf_mode when appropriate.

UI / Web UI:

I’d like to replace the Python WebUI with a lightweight, portable alternative. If you enjoy UI work, help here would make a big difference.

If you are porting an audio model, optimizing one, or helping make local audio inference less painful, I would love to have you involved!

Hi everyone,

I'm looking for help testing audio.cpp on ROCm.

audio.cpp is a native C++/GGML audio inference framework. It runs local audio models for TTS, voice cloning, ASR, voice conversion, music/audio generation, and more through one CLI/server/runtime instead of a separate Python stack for every model.

Most of audio.cpp’s current performance work has been validated on CUDA, CPU, Vulkan, and some Metal paths, but ROCm coverage is still thin because I do not have AMD hardware to test it properly.

ROCm support in audio.cpp is community-driven. I'd appreciate help with building and testing on ROCm and benchmarking models and sharing performance numbers.If you're interested, please submit a PR or join the discussion in https://github.com/0xShug0/audio.cpp/pull/48

Repo: https://github.com/0xShug0/audio.cpp

audio.cpp again :)

Release 0.4 is out. The headline this time is new high-quality TTS coverage plus GGUF becoming a first-class across the project.

What’s new:

• Added Higgs Audio v3 TTS 4B, Fish Audio S2 Pro, Voxtral Realtime ASR and two community models OuteTTS TTS and VieNeu-TTS-v3
• audio.cpp now support 35 model families.
• All released model families now support GGUF.

Ready-to-use GGUF packages are now available, and Q8 is starting to show real speed and memory wins on several routes. Check the figures. Long-lived session is multiple requests after warmup. Longform is one-shot 6000+ char text generation. Tested on RTX 5090.

CUDA Q8 GGUF numbers from my current measurements:

• Higgs Audio TTS: warmed requests run about 8.8x-10.1x faster than real time. Longform runs about 8.5x faster than real time.
• Fish Audio S2 Pro: warmed requests run about 3.1x-3.4x faster than real time. Longform runs about 3.3x faster than real time. Plenty of room for improvement because the impl is a naively adaptation of framework template.
• Voxtral ASR: offline runs about 15.7x faster than real time, with streaming TTFT around 171 ms.

Compared with 16-bit GGUF, Q8 is not universally magic, but it is useful now. In the tested release paths, Q8 can be up to about 1.5x faster and reduce peak VRAM by up to about 37%, depending on the model and route. Quality is still model-specific, so I am keeping the GGUF support matrix and Q8 performance report visible instead of pretending every quant is safe everywhere.

(Some tricks to further boost performance up to 2x for some mdoels like Qwen3-TTS: adjust chunk size and cut reference audio len.)

audio.cpp now has a dedicated community models area for ports that are useful and runnable, even if they are still maturing. The review bar there is lighter than the core framework. If you have a model you'd like to bring to audio.cpp, try implementing it as a community model first using framework modules and patterns.

Huge thanks to the contributors who have been porting, optimizing models, adding new features, and pushing the project forward.

Repo:https://github.com/0xShug0/audio.cpp

audio.cpp again :)

Release 0.4 is out. The headline this time is new high-quality TTS coverage plus GGUF becoming a first-class across the project.

What’s new:

• Added Higgs Audio v3 TTS 4B, Fish Audio S2 Pro, Voxtral Realtime ASR and two community models OuteTTS TTS and VieNeu-TTS-v3

• audio.cpp now support 35 model families.

• All released model families now support GGUF.

Ready-to-use GGUF packages are now available, and Q8 is starting to show real speed and memory wins on several routes. Check the figures. Long-lived session is multiple requests after warmup. Longform is one-shot 6000+ char text generation. Tested on RTX 5090.

CUDA Q8 GGUF numbers from my current measurements:

• Higgs Audio TTS: warmed requests run about 8.8x-10.1x faster than real time. Longform runs about 8.5x faster than real time.

• Fish Audio S2 Pro: warmed requests run about 3.1x-3.4x faster than real time. Longform runs about 3.3x faster than real time. Plenty of room for improvement because the impl is a naively adaptation of framework template.

• Voxtral ASR: offline runs about 15.7x faster than real time, with streaming TTFT around 171 ms.

Compared with 16-bit GGUF, Q8 is not universally magic, but it is useful now. In the tested release paths, Q8 can be up to about 1.5x faster and reduce peak VRAM by up to about 37%, depending on the model and route. Quality is still model-specific, so I am keeping the GGUF support matrix and Q8 performance report visible instead of pretending every quant is safe everywhere.

(Some tricks to further boost performance up to 2x for some mdoels like Qwen3-TTS: adjust chunk size and cut reference audio len.)

audio.cpp now has a dedicated community models area for ports that are useful and runnable, even if they are still maturing. The review bar there is lighter than the core framework. If you have a model you'd like to bring to audio.cpp, try implementing it as a community model first using framework modules and patterns.

Huge thanks to the contributors who have been porting, optimizing models, adding new features, and pushing the project forward.

Repo:https://github.com/0xShug0/audio.cpp

audio.cpp again. Hopefully you are not sick of it yet :)

Release 0.3 adds five new models: Supertonic 3, MOSS-TTS-Local, MOSS-TTS-Nano, IndexTTS2, and Irodori-TTS.

The highlight is Supertonic 3. It can hit 200×+ real time on CUDA (RTX 5090), 6×+ on CPU, and around 47 ms TTFT in CUDA streaming mode. In the demo (sorry for the rough demo), I used The Adventures of Sherlock Holmes as the input and generated around 10 hours of audio in about 3 minutes on an RTX 5090.

Supertonic 3 was also pretty fun to work on. The official implementation uses ONNX, so I had to reverse-engineer the inference path and rebuild it around the safetensors weights.

The audio.cpp version is much faster than the Python implementation on CUDA, while CPU performance is about the same, maybe slightly faster.

One reason for the big CUDA win is that the ONNX version sends some nodes back to the CPU, so it can’t fully use the GPU.

The other models released in 0.3 range from roughly the same speed to more than 2× faster, depending on the test case. In IndexTTS2 longform test (6000-char input text + 2400-char emotion text), C++ is 5.65x faster than Python.

GGUF support has been added and will be rolled out model by model.

Contributions and feeback are very welcome. There’s still a lot to improve across model coverage, streaming, GGUF support, backend compatibility, performance, testing, and the server/UI layers.

Repo: https://github.com/0xShug0/audio.cpp

I’m the author of audio.cpp, a C++/ggml runtime for local audio models.

I just added VibeVoice 1.5B support and wanted to share the benchmark because long-form multi-speaker TTS is a good stress test for local inference runtimes.

Result on RTX 5090:

VibeVoice 1.5B
Audio length: 5615.73s / 93.60 min
Wall time: 1376.84s / 22.95 min
RTF: 0.245
Speed: 4.08x faster than real time
Python baseline: 92.66 min audio in 65.70 min
Speedup vs baseline: 2.86x
Quantization: none
Diffusion steps: 10

The main point is not just avoiding Python setup pain, though that is part of it. The goal is to make audio models practical in a native local runtime: reusable sessions, server-like usage, long-form generation, stable memory behavior, and CUDA-focused (CPU and Metal later) optimization.

VibeVoice is a useful milestone because it is not just short-sentence TTS. It is designed for long-form, multi-speaker dialogue such as podcasts, character chats, and narration, where runtime behavior matters a lot.

Current framework progress:

Released model families: 16 / 28
[███████████░░░░░░░░░] 57%

The other model families are already running end-to-end internally, but I’m releasing them gradually after testing and cleanup.

The repo is https://github.com/0xShug0/audio.cpp

I’d be interested in feedback from people testing VibeVoice on other GPUs or CPUs, especially long prompts, multi-speaker formatting, VRAM behavior, and performance numbers.

Update (06/26/2026): Three small models from Nvidia released: CitriNet (STT) MarbleNet (VAD), Sortformer (speaker diarization)

I’ve been working on audio.cpp, a native C++ inference framework for audio models built on top of ggml.

The framework currently has 25 model families, but I want to be precise about its state: 12 are released in the repo now and ready for normal use. I’m not counting anything still in integration or optimization as released.q

The released set already covers quite a bit:

TTS / voice cloning / voice design: Chatterbox, MioTTS, OmniVoice, PocketTTS, Qwen3-TTS and VoxCPM2

ASR / alignment / VAD: Qwen3-ASR, Qwen3 Forced Aligner and Silero VAD

Voice conversion / codec / editing: Seed-VC, MioCodec and Vevo2

Vevo2 also handles TTS, singing generation, singing conversion and editing, so this has grown beyond a collection of TTS ports.

The point isn’t to build a model zoo.

It’s to stop treating every audio model as its own island with a separate Python environment, dependency tree, CLI, batching logic and deployment setup. I want these models to share the same runtime, session handling, CLI, server, audio utilities and eventually the same higher-level workflows.

The performance is where the project started to feel genuinely useful rather than just easier to deploy.

These results were measured on Ubuntu/CUDA using the original weights without quantization. The figures compare audio.cpp wall time against the matching Python reference path:

PocketTTS: 3.68× faster on a 1-shot run, 3.22× in a warm session and 3.15× on long-form

Qwen3-TTS: 1.83× on a 1-shot run, 2.74× in a warm session and 3.06× on long-form

Vevo2: 5.03× on a 1-shot run, 1.75× in a warm session and 1.77× on long-form

MioTTS: 2.73× on a 1-shot run and 2.28× in a warm session

Chatterbox: 1.58× on long-form

The long-form throughput makes those numbers easier to picture. Using the same 1,028-word input:

PocketTTS: generated 5m 53.12s of audio in 7.30s — 48.40× real time

OmniVoice: generated 5m 57.00s in 17.77s — 20.09× real time

Vevo2: generated 7m 37.68s in 52.47s — 8.72× real time

Every released TTS family included in that benchmark ran faster than real time, ranging from 4.34× to 48.40×.

I don’t want to oversell it: not every path beats Python yet, and the README keeps the weaker results visible. But the warm-session numbers are the ones I care about most. They are closer to a real service setting, where the model is loaded once and reused across many requests.

The shared runtime is the bigger bet.

The current same-language redubbing pipeline takes a 418s recording, splits it into manageable chunks, transcribes it with Qwen3-ASR, merges the transcript and regenerates the speech in a target reference voice with Qwen3-TTS—all behind 1 CLI command.

The inference and server paths are native C++. There is a Python utility for downloading and converting model packages, but Python isn’t part of the actual inference path.

It’s still early. Backend coverage depends on the model, and framework-wide streaming isn’t generally supported yet, so the current paths should still be treated as offline. The framework can target CPU, CUDA, Vulkan and Metal where the model supports them.

Repo:

https://github.com/0xShug0/audio.cpp

I’d really value benchmarks from other hardware, failing cases, API feedback and PRs.