The temporal drift problem: Why AI lipsync on long-form video looks uncanny.
For me, mapping audio to a generated face over a 10+ second clip starts to desync (is that even an actual word?), well, derail after a few seconds.
The models try to match the audio units to visible mouth shapes, but when the timestamp alignment isn't locked in. The generated lip movements start lagging the audio track by a few frames. The difference here, could just be a couple of frames. However, visually it looks fake.
My standard workaround for this is to cut the clips to just about 8 seconds before I really looked into this. If I understand it correctly, standard cloud engines have a hard time maintaining a tight audio-visual frame anchor over long clips, meaning the lag gets baked right into the video file before you even see it. Saw someone using MiniMax Design to move part of the work locally to manage the audio-visual sync. So basically, instead of trusting the cloud to do everything in one massive, drifting generation, it has localized canvas control to step-feed the audio track to the video frames in tight chunks, keeping the timing right before exporting.
Havent tried it, but has anyone else experimented with similar desktop-side canvas workflows to fix the issue?