H264: How can I limit the maximum size of an I-frame?
I've been working diligently on a custom video converter for an MP3 player. It required some special x264 patches to basically disable reference P-frames and swap to disposable P-frames only, as well as disabling the VUI and some SPS units (?), and a chroma tint fix. Now, I've got it down to a science, but there's one last tricky limitation I'd like to optimize.
The device's decoder chokes under 2 conditions: the total bitrate exceeds 15-20 Mbps (which will never happen), or the size of an I-frame exceeds ~44Kb.
Unfortunately, I don't think x264 or ffmpeg have a mechanism to specifically limit a single frame size. There's a max slice size, but that doesn't work because it's not granular enough, so big frames still go through. The obvious idea is a VBV, but the VBV's sampling window would need to literally be equal to per-frame timing, which I've also tried but remember that it never actually fixed the crash, big I-frames made it through.
The current solution I had was to encode the most tortuous, noisiest source videos and determine the minimum qmin I could use to prevent an I-frame overflow crash. That ended up being qmin 20. While this works, it also means that I'm losing encoding efficiency. Flat, gradient areas need that lower quantization to look clean without banding, but if I drop qmin, a torturous scene could spike an I-frame past the limit.
The last option I came up with was once again patching x264 in ratecontrol.c to bump qmin if a frame is too big, a simple in-line loop. But when it comes to patching and writing C, I'm pretty useless; my previous x264 patches were only really made possible through Cursor (albeit, the research and refinement took months to do, so it wasn't just Cursor).
So my question is, can anyone come up with a way to limit a single I-frame's size during an encode, without needing to encode, check, re-encode, etc? Thank you