Experimenting with animated Gaussian splats from video, feel free to give it a try
We added support for animated Gaussian splats to STAGEit recently, which immediately made us want to create our own.
So the latest STAGEit Optimiser now has an alpha Gaussian splat generator, built on top of Apple’s SHARP library.
It’s very much alpha. Results vary, and this is not a tool for scanning your room. We’ve had the best results with people and smaller/macro subjects where you can isolate what you actually want to capture.
The basic workflow is:
- Drop in a video - we recommend 10 seconds or less
- Crop to the part of the image you actually want
- Trim the clip in the app (for long videos, trim them before importing)
- For people, enable People Cutout and Depth Cleanup
- Choose the depth you want to retain - smaller values help avoid unwanted background splats
- Generate an animated splat sequence
For reference, some depths we’ve used in our examples are 0.3m for the apple, 0.8m for Merlin (the Golden Retriever), and 1.0m for the lady in blue.
Processing can take 30-60 minutes, so we strongly recommend testing at 5fps first. Once you’re happy with the result, 24fps has been our sweet spot. You can go to 30fps, but it starts getting pretty demanding on the device.
There’s also a focal-length control, but unless you know your source footage was shot significantly differently from a standard ~30mm lens, we’d leave it alone.
We’ve also added wireless transfer directly into STAGEit. In the latest STAGEit, open a scene → Settings → Network → Host. STAGEit Optimiser can then connect directly to the device and send across both optimised USDZs and generated splat sequences - no more AirDropping files back and forth! Huzzah (this has been one of the best updates)
The generated .splatseq file is essentially an archive of the individual splat frames. If you have another app or workflow that can deal with splat sequences, you can rename the file to .zip and access the individual frames.
Otherwise, currently STAGEit is our playback option for the animated sequence. There is a free version of STAGEit Creator that allows you to try this, including the networking workflow.
It’s experimental, occasionally messy, and definitely still alpha, but we’re already getting some really interesting results from it.
STAGEit Optimiser is free, so feel free to give it a try.
https://spatialcreative.com/stageit-optimiser/
https://apps.apple.com/us/app/stageit-creator/id6504801331