Building a world with my voice
This is a project built with A-Frame (Three.js, WebXR) that lets me build virtual worlds with my voice, all from within my VR headset. My mic is hooked up to a coding agent, which in turn runs against the project codebase.
Since the agent is working directly with code, the possibilities of what can be achieved are pretty broad, essentially being limited only by what the underlying model is capable of with WebXR. The agent can build anything from simple objects, to animations, to interaction capabilities, to whole environments in order to fulfil a request.
This is still far from perfect. Latency is pretty high, taking anywhere from a minute to multiple minutes to carry out a change depending on how complex the request is. The codebase is also not very accommodating to anything that isn't a frontier model right now. Ideally this will work with smaller, local models in the future.
But the foundation exists, and as these models improve I can see this being a unique way to interact in VR. If you have ever imagined being able to put on a VR headset and transporting yourself to any world you want by simply asking, to shape that world to meet your creative vision, and to watch that world materialise around you, then in a small way, we are here.