u/Dark-Horn

I'm curious whether anyone here is actually running SALMs in production today, or actively experimenting with them.

A reasonable starting point seems to be something like:

• Voxtral-Small + TTS
• Whisper / mimi-style audio encoder + existing LLM backbone (Qwen, Gemma, etc.)
• Speech adapters on top of strong tool-calling LLMs

What I'm more interested in is the training side than the inference

For example, suppose we take:

• Whisper / Mimi as an audio encoder
• Qwen3 / Gemma as the backbone LLM
• Freeze most of the LLM initially
• Train an audio adapter / projector
• Continue with SFT, distillation, RL, or some combination

Questions:

1. Has anyone actually built and deployed something like this?
2. What datasets are people using? Pure ASR data, speech-instruction data, synthetic data, or some mixture?
3. How are you generating/cooking the data for tool-calling and conversational voice assistants?
4. Are there any open-source implementations, training recipes, cookbooks, or papers you'd recommend?
5. How well do these systems scale compared to a traditional voice stack?
6. What ended up being the hardest part: data, alignment, latency, turn-taking, tool calling, or something else?

Would love to hear from people who've trained these systems themselves rather than only consuming hosted APIs

I'm curious whether anyone here is actually running SALMs in production today, or actively experimenting with them.

A reasonable starting point seems to be something like:

• Voxtral-Small + TTS
• Whisper / mimi-style audio encoder + existing LLM backbone (Qwen, Gemma, etc.)
• Speech adapters on top of strong tool-calling LLMs

What I'm more interested in is the training side than the inference

For example, suppose we take:

• Whisper / Mimi as an audio encoder
• Qwen3 / Gemma as the backbone LLM
• Freeze most of the LLM initially
• Train an audio adapter / projector
• Continue with SFT, distillation, RL, or some combination

Questions:

1. Has anyone actually built and deployed something like this?
2. What datasets are people using? Pure ASR data, speech-instruction data, synthetic data, or some mixture?
3. How are you generating/cooking the data for tool-calling and conversational voice assistants?
4. Are there any open-source implementations, training recipes, cookbooks, or papers you'd recommend?
5. How well do these systems scale compared to a traditional voice stack?
6. What ended up being the hardest part: data, alignment, latency, turn-taking, tool calling, or something else?

Would love to hear from people who've trained these systems themselves rather than only consuming hosted APIs

I'm curious whether anyone here is actually running SALMs in production today, or actively experimenting with them.

A reasonable starting point seems to be something like:

• Voxtral-Small + TTS
• Whisper / mimi-style audio encoder + existing LLM backbone (Qwen, Gemma, etc.)
• Speech adapters on top of strong tool-calling LLMs

What I'm more interested in is the training side than the inference

For example, suppose we take:

• Whisper / Mimi as an audio encoder
• Qwen3 / Gemma as the backbone LLM
• Freeze most of the LLM initially
• Train an audio adapter / projector
• Continue with SFT, distillation, RL, or some combination

Questions:

1. Has anyone actually built and deployed something like this?
2. What datasets are people using? Pure ASR data, speech-instruction data, synthetic data, or some mixture?
3. How are you generating/cooking the data for tool-calling and conversational voice assistants?
4. Are there any open-source implementations, training recipes, cookbooks, or papers you'd recommend?
5. How well do these systems scale compared to a traditional voice stack?
6. What ended up being the hardest part: data, alignment, latency, turn-taking, tool calling, or something else?

Would love to hear from people who've trained these systems themselves rather than only consuming hosted APIs

I'm building a production voice AI system (STT → LLM → TTS) and have been evaluating three ASR options:

• Parakeet-TDT-v3 (self-hosted via vLLM-Omni)
• Whisper-Turbo-v3 (via Groq)
• Mega-ASR / Qwen3-ASR fine-tune (vLLM-Omni)

From my own testing:

• Whisper-Turbo-v3 produced the best transcription quality overall.
• Qwen3-ASR / Mega-ASR was noticeably better than Parakeet-TDT-v3.
• Parakeet-TDT-v3 occasionally missed or misidentified certain words, especially in conversational speech.

However, I keep seeing people recommend Parakeet-v3 as the best open ASR model for production deployments.

For those who have deployed these models in real systems:

What has your experience been with transcription quality .How do they compare on noisy audio and accented speech.

I'd love to hear experiences from people running these models in production for real time voice assistants rather than benchmark-only evaluations.