Long story short I'm running a high concurrent translation pipeline.

---

The data, in and out

Translation instruction are something like 1.5k-2k tokens, it contains the instructions. Sentence to translate is in user prompt.

LLM answers in the assistant prompt with translated sentence.

I have a farm of 6 GPUs (HAProxy load balancer).

95% of requests are 1600 tokens in (system prompt), 25 tokens out.

So an efficient cache prefix is needed.

---

1. SGLang provides with radix cache a ferfect cache with hybrid architectures.
2. vLLM on qwen 3.5 will have a context window that is a multiple of 784 because of the architecture, align mode (all not supported yet) which makes prefix hit cache of 40-50%

You can't adjust --mamba-block-size but you can adjust --block-size and moving it to 1200 boosted my cache hit rate to 80%.

---

Conclusion :

Fixing my cache hit makes the GPUs spent less time on prefill and more on decode, making them move from 1900 t/s of aggregated token thruput to 2400-2600 t/s

Github repo link : https://github.com/ExtReMLapin/hoi4_mod_translator

Background

Been working on LLMs since llama 1 release, I'm a small vLLM and llama.cpp contributor.

Paradox games translation strings includes special tags.

Example :

AST_R56_memories_of_gallipoli_desc:0 "§RYou cannot increase conscription laws until relevant focuses have been taken. Check the conscription law for more details.§!\n\nThe [AST.GetAdjective] people have not forgotten how Australian blood was shed on the battlefields of the Great War. As such, there is significant opposition to any idea of conscripting [AST.GetAdjective]s to fight overseas should another war break out."

You can extract them to send them to a translation model, but the translation like NLLB-200 doesn't preserve the tags.

You can also translate the strings by hand, but for example road to 56 got 120k lines to translate.

The (first, not final) solution :

Solution is to do multi shot LLM assisted translation, you pretty much explain in the system prompt the rules of how the string is build, what he can translate, what to "keep".

Beside the set of rules, I explicitly cite what he MUST not translate in the system prompt, then the tags to keep are programmatically checked to ensure the string is not corrupted. Beside the 1:1 match of the tags, we can't do much more.

When the LLM fail to do something, like extra or missing tag (not 1:1) guidance of what's wrong is sent on the next conversation turn.

And when after the 3rd turn it's still not fixed, it fallback to the original source language.

If you want to give a try yourself, you'll need a LLM, either running locally, or API, whatever you want, it's using an OpenAI compatible endpoint. If you have an API key you probably need to set an env variable or edit the python.

Translating all the 120k lines of RoadTo56 takes 15mins-1hour depending on the picked model, GPUs you have. (and if your prefix caching is working or not :v ).

Error rate (could not find all the expected variables after 3 retries) is something like 1-2% or less depending on the model

I mostly use it to auto skip the "oh no, not this bitch again" situations from The boys.

Long story short, it uses the metadata from X-Ray to detect all the scenes where an actor is and allows you to auto skip this scene.

There is a preview in the progress bar that shows you which part of the movie/episode is being skipped