I want to train/fine tune an llm for music generation, basically I have huge amount of trance music collection. I want to train or fine tune an llm using those music, it doesn't have vocal only music. Which model would be best to use as base model. Anyone done this before?
I put together a dataset-backed benchmark for local voice cloning models:
- OmniVoice int8
- Chatterbox Multilingual fp16
- VoxCPM2 bf16
- Fish Audio S2 Pro fp16
It uses Google FLEURS test clips as references, then reports speaker similarity, WER/CER, generated audio length, and RTF. I also included the reference audio and generated clips for every row, so the table is not just numbers.
Post:
https://www.soniqo.audio/blog/voice-cloning-benchmarks
The result that surprised me most: OmniVoice was the best all-around row set in this run, but the language-by-language behavior is more interesting than the aggregate. VoxCPM2 was strong on Arabic speaker match; Fish Audio had strong German/Arabic similarity but slower RTF; Chatterbox looked good on Arabic/Spanish but I am not benchmarking Chinese until the Swift tokenizer path is ready.
I maintain the Soniqo speech stack, so this is self-promo, but the benchmark is meant to be useful/reproducible rather than a launch post.
Speech Studio is the open-source desktop app built on the same stack:
https://www.soniqo.audio/speech-studio
https://github.com/soniqo/speech-studio
What model/language should I add next?
I'm working on a cascade based TTS model for a very narrow domain with specific noise environment and heavy jargon library. I've tried every providers ASR/TTS models and honestly it's not good. Really not good (high WER and tuti cunti). So as a ML engineer I'm trying to fine-tune TTS models (currently on QWEN3 ASR), but I'm a text guy, not an audio guy and so is my team haha. So we are learning and trying to build one cause on-the-shelf are bad but it's very difficult and we need to ship soon. Someone in that situation that as counsels or can direct me to GitHub discussions/Discord groups/events.... that may help me ? (I'm US based but I can travel anywhere if there is a real value and a real community of people blocked as us trying to look for a solution