Where we left off last time

Let's summarize, where were stopped last time:

• We had phonemes and stress for all languages mostly "solved";
• We collected numerous recordings of "professional programmers" by the game devs (including their "screaming" voices);
• TTS should work on 1 CPU thread (i.e. on "toasters"), be very fast as it runs within the game client and should ideally support 8 languages and "screaming" voices;
• Air comm voices should be stoic, flinchless. Scream should be also similar to real-life pilot comms under pressure, i.e. also calm, reserved, stoic, not explicitly panicking;
• Due to current conditions it is next to impossible to hire professional voice talent to suit our needs, despite the game devs reputation.

What speech consists of

Typical speech can be separated into the following entities:

• Speaker identity or timbre (i.e. how your voice sounds);
• Speaker mannerisms / speech patterns (i.e. how you typically speak, intonations);
• General prosody (i.e. intonation, pauses, word and phoneme duration);
• Speech content (i.e. exact words or phonemes spoken);
• Some other nuanced characteristics like scream, questions, emphasis, exclamations, sighs, etc.

Naturally it is quite difficult to separate them all perfectly, but some TTS models strive to extract and distill at least some parts of this list.

What we tried

At first we ran experiments on more limited datasets (i.e. only Russian data) and then tried to extend to other domains / languages. Here is a curated shortlist of the approaches we tried and their comparative merits:

Approach	Pros and cons	Conclusion
Formant-based approaches	Works well with high quality speakers with abundant data, allows direct precise control over screaming intensity	Does not work with scarce data. Does not scale to other languages via transfer learning
Transfer-learning based approaches	We pre-train a small target model on public domain data, then we fine-tune using our in-domain data	Works more or less fine with 2-3 languages within one model. Heavy accent. Does not scale to all languages. Model capacity not enough, total model size (if many) prohibitive, hard to support
Synthetic data based approaches	Data scarcity is solved by data generation using our internal revoice models, slim production models are trained on synthetic data mixed with real data	Our chosen approach. Accent exists for some speakers, native speakers marked it as acceptable. Scream kind of works, all languages work. TTS sounds a bit mushy. Production model is small and fast.

So, basically, the approach that was shipped into the game was based on synthetic data and our "professional programmers'" voices. We have developed several internal models for TTS and revoice to make this work.

Obviously, these models try to separate between prosody (i.e. words or phonemes) and style (i.e. your timbre, accent and mannerisms). But of course they cannot do it perfectly. Hence for some speakers you can hear some accent or some words are a bit mushy. Also such models are always "leaky", i.e. they are not able perfectly to separate screaming prosody from style. And yes, this happens partially because we had to use our recordings as a basis, which were done by professional programmers.

Since we are not a giant corporation, we cannot just feed all of the Internet into a trillion param sized model and add 100,000 GPUs.

How can community help

It is next to impossible to find public domain data that would be 100% in-domain, i.e. stoic male voices, also under stress (ideally pretending to be military pilots).

If you wish to channel your inner post-WWII pilot and immortalize your voice TIMBRE in the game and help us bridge the last remaining gap in voice fidelity, we would need the following for as many languages and speakers as possible:

• At least 1 minute of speech (the more the better) with the correct accent (i.e. British or American English, or any other language from the list) with and without scream;
• Ideally there should be some pauses between utterances;
• The recordings should be made in a quiet room using a off-the-shelf streaming / game microphone, ideally in 48 kHz (this is quite standard nowadays);
• Typically, when doing such recordings, we try to prepare the texts in advance and we have a special web app for recording, but here I guess this would be impractical. I guess aviation fans should know better than us!

We are going to use the recordings as follows:

• We are only going to use the voice TIMBRE and the way you pronounce the phonemes. We are NOT going to use the way how you speak;
• We are going to run experiments using your TIMBRE as an in-game pilot;
• Of course all of these are EXPERIMENTS and may fail.

Also please note that:

• The success depends on the number of participants and language coverage;
• It may not produce the desired effect, it's experimental, the experiment often fail;
• We are not game devs, and the final voices will have to be approved and chosen by the game devs;
• Game devs may have some last mile legal hurdles to include the chosen assets in the game.

If you are willing to participate, please DM me on Reddit.

Many thanks for your attention!

A Small Sneak Peak on How TTS Radio Comms Were Developed in IL-2 Korea [Part 1]

Hi, my name is Alexander, I am one of the developers, who created a TTS (text-to-speech) voice engine for a new IL-2 game. Since my previous post was upvoted this will at least try to give the community a small and casual dev blog about how we developed the TTS engine and why it is what it is under the constraints and limitations.

A few disclaimers out of the way:

• The more detailed original post in Russian is here;
• We are not affiliated with the game publishers / developers, we are a separate company;
• English is not my first language, this is not AI-generated (lol), corporate-approved, etc. etc;
• This as simplified and abridged as possible without losing the gist of it;
• Development took time, and our tools as well as the environment evolved during this time;
• I am not a big sim player, I play mostly FPS / RTS / strategy games, though I played tanks in its heyday (I know it's more like CS, not a sim);
• As a teenager I was more keen on tanks, not airplanes;
• We have not exhausted all of our ideas yet, but you will see after reading the whole post series, but as an outside contractor there is pressure to stick to the specs and deliver the MVP asap.

The Job

Ok, let's get rolling, lads. Some time ago we were contacted by the devs. They wanted us to develop an in-game TTS engine for in-game comms for a couple of reasons:

• Easier long-term support for different theatres of war for years worth of future content;
• No long-term dependence on voice actors to create new content / DLCs / missions;
• Increased diversity and versatility in voice lines;
• Some ideas about multiplayer (not implemented yet);
• No external dependencies.

Technically the minimal requirements were as follows:

• At least 8 languages;
• TTS should work faster than real-time on toasters, 1 CPU thread;
• All of the voices should be able scream (overall comms should sound like this);
• TTS should support custom proper names and toponyms.

Yeah. You cannot use GPUs during inference. At all. But how hard can it be to create a new TTS voice? Take the most upvoted GitHub repo, make some recordings and GPUs go brrr, right? Wrong.

Houston we have a problem!

All of these requirements were not a problem. Because our in-house workhorse TTS system could eat them for breakfast. Just add the data and press play?

The problem became obvious when we started looking for voice actors. At first we believed that the brand name and the legendary status of the game (the original game is the legend in Russia) would help us. They did, but only partially. So:

• Game developers were able to source native speakers of all of the languages to audit the work / create content for the game;
• Voice actors are scared of AI (albeit irrationally for most use cases) and most of them plainly refuse to work, when the scope and limitations are openly presented;
• Surprise surprise, game devs and we are Russian companies. And due to sanctions and other obvious reasons foreign companies and individuals are reluctant to work with us. Also there are some technical and legal hurdles;

>!Also I have to rant that the lately popular umbrella term AI is horrible, because for the layman everything is AI now. But our work is plain machine learning and plain data munging. And generative AI trying to suck the soul out of the Internet is a very different ugly beast birthed in deep dens of giga corporations.!<

So, what can we do? We also need all of the voices to scream, kind of, remember. We have to go to the roots. In Russia we have a meme, that classic games (i.e. the English version of Warcraft 3 and the abundant pirate scene of late 90s early 2000s) were voiced by professional programmers.

Yes, we had no choice, but to resort to ... recording the C++ developers of the game. And they also had to record the scream.

How do you spell "fish" using English spelling rules? Ghoti!

I will not bore you with the details (please see the original post), but in order to make every language work for a long-term support (and phonetic errors can and will be fixed by the devs in future) we had to make all 8 languages work together.

Languages (only first 4)	Key issues
Russian	Easy phonetics, palatalization, but you have to know the stress, which is a challenge (for 4M word forms)
English	The most complicated phonetics, spelling is a nightmare
Chinese	Non-phonetic, but surprisingly simple, only 1500 syllables are possible! Direct mapping to phonemes!
Korean	Weird characters are actually letters, spelling rules are not numerous, but we used neural networks anyway

And yes, you should be able to read custom toponyms and proper names. So basically we had to develop (or adopt industry standard tools for pinyin conversion for Chinese) or borrow FOSS tools to convert each language into phonemes.

We provided the devs with pipelines for each language where they easily can convert words in each language into phonemes and then a native speaker proof-reads the phonemes. Typically this includes:

• A stress model (and / or dictionary) that puts stress in words (can be omitted for Asian languages);
• A grapheme to phoneme model (and / or dictionary) to convert letters into phonemes. In case of Chinese ... we just used official FOSS tools to convert to pinyin and the just use dictionaries;

Also some Asian dudes have a weird tendency to omit spaces and not care about word stress, which may influence TTS a bit (tones are a form of stress as well).

For English it is simple, right? Words do not change (mostly, unlike Russian) Just find a dictionary and that's it? No. Phonemes and sounds should be consistent across all of the languages. Also for new words models achieve only about 95%, so a human in the loop is always necessary to create new game content. Also just check out this video on English toponyms. This is scary stuff! Especially words like Worcester.

Making the pieces fit

This is getting a bit long for Reddit. I will split the post in several chunks and see if you have any questions or whether I should polish my style a bit for the local audience.

Thank you for your attention and have a nice day!

IL-2 Korea TTS Engine devs here. Would the community be interested in an in-depth dev blog about the TTS Engine for the new game? If so, which topics to cover first and foremost?