Video rendering engine built with Rust and wgpu
Hi everyone!
About bout a year ago, I was building a multi agent AI video editor and picked Remotion for the rendering layer. Worked fine for basic use cases, but for anything more advanced we started facing standard issues like disabled GPUs, animations jitter, audio lag, mismatched colors, 4K rendering that took ages … evantually most of advanced users churned.
After a while I started playing with a Rust + wgpu renderer that would run both on server and browser with ffmpeg doing standard decoding/encoding. The goal was to create lightweight version of Adobe Premiere + Effects through an API.
The architecture is pretty straightforward, you put a simple timeline based JSON into a message queue, a Rust service picks it up, downloads the assets, and hands the video to ffmpeg for nvdec decoding/encoding and audio mixing. Then for each frame ( plus timeline logic ) the GPU compositor builds what you’ve asked for (e.g., stacked clips, animations, effects, etc.). The finished frames go back to ffmpeg for encoding, get mixed with the audio, and the final mp4 output is saved into storage.
The most interesting part of this ( for me ) was that you can run the exact same renderer in the browser and get pretty much the same result as on the server (at least until we figured out that glyphon couldn't handle everything planned…).
Anyway, after testing this live on GCP compared to what I’ve seen before this looks super impressive. It takes just 3-4 seconds to render a 10 seconds 4K high quality video complete with captions, animations and audio. Perhaps that’s expected but coming from where I was before I was quite blown away!
I’m gonna pause a bit now to retrospect on everything so far and think where to go next and see does anyone aside from me needs this.
If anyone is experiencing issuess or working on a something similar, happy to answer any questions about the architecture, tech decisions or approach in building specific features.
I also plan to opening a small closed beta for any teams currently having the same remotion/ffmpeg blocks like we had.
p.s. I’ve used Rust a bit before, but graphics programming was a completely new thing! 🤯
p.p.s One thing that turned up to be super interesting is that LLMs are totally useless for this sort of work!! Like it's the complete opposite of what I’m used to when building “regular” apps. I literally had a weekend task running on Claude Opus and barely anything usable came out of it…