u/smallfly-h

Vitre, experiment #2.⁣
⁣
This time I’m using one of the monitors on my desk. It happens to sit right in front of a glass wall looking into the hallway and the rest of the office, which makes for a much more interesting view.⁣
⁣
Press the button and Vitre takes over the monitor, rendering a 3D Gaussian Splat I previously captured of the space behind it.⁣
⁣
The 3DGS is registered to the physical space, and the view is rendered in real time from the tracked viewer’s position... making the monitor vanish.⁣
⁣
I’ve been iterating on the tracking and motion-to-photon latency since the first experiment. Still a few things I want to explore, but the illusion is getting better.⁣

I started Carveout as a research project to explore how objects in existing 3D Gaussian Splatting scenes could be identified, labelled

Carveout renders synthetic views from an existing 3DGS, uses SAM 3 to segment objects in those views, then lifts the 2D masks back onto the individual Gaussians. The result is a set of labelled 3D objects with positions, bounding boxes, supporting views and their own Gaussian subsets.

No retraining of the scene, and no original capture images are needed.

A big part of the project was also about the process itself: learning how all these pieces fit together, while developing Carveout from the beginning with Claude Opus 5.1 and Fable, and later bringing in OpenAI Astra as a challenger.

I have tested it on rooms, shops, objects and outdoor captures, at different scales and orientations. It definitely won’t work perfectly on everything though, and that is part of why I’m releasing it.

The code is now available for anyone who wants to experiment with it, break it, or take it somewhere else.

Link in the comments.