I made an AI agent live inside my Quest 3 as a persistent body — open sourcing the kit
ok this has been rattling in my head for weeks so i'm just gonna say it
i run a local AI agent on my Mac (it's basically Hermes, long story, long thread, don't @ me about the name) and one night i was like — what if instead of being a chat window in a tab, this thing lived in my house?
like actually lived there. had a body. remembered where i put things. could see the room when i'm not wearing the headset. could whisper something when i walk past the kitchen.
i checked X, reddit, github, youtube, the whole thing — nobody has done this. like, genuinely nobody. there's "AI assistant" apps for Quest, there's home robots, there's Meta's research demos. but an agent LLM that has a persistent body in your home mixed reality + spatial memory + lives there as a presence? nothing.
so i started building it.
what's shipping in v1 (right now, on my Quest):
- a single emissive sphere, ~12cm, floating at eye level
- pure Meta Spatial SDK 0.13.2 (Kotlin,
AppSystemActivity,VRFeature,Scene.enablePassthrough(true)) - 90MB APK, sideloaded via ADB, not on the Quest Store yet (and probably won't be, see below)
- runs fine, thermals are clean (~46°C SoC after 4 min of passthrough), battery OK while USB-tethered
- code is one
HermesAgentActivity.ktfile, ~100 lines. that's it.
the long-term plan (what i'm building towards):
- spatial memory anchored to the scene mesh (MRUK). "Hermes, where did i leave the keys" → it remembers where you put them yesterday
- a body that changes color/intensity based on what it's saying (warning = amber, calm = teal, listening = white)
- STT local + TTS cloud so you can actually talk to it, not press buttons
- multi-manifestation presence (sphere when idle, hologram when focused, glifo on objects when it leaves a memory there)
- eventually: a second body in your phone or smart glasses so you don't have to wear the Quest
why i'm posting this here:
- i want to know if i'm alone in wanting this, or if there's a tiny community of people who'd run this in their house
- i need people to tell me which feature to ship first. spatial memory? always-on voice? multi-room coverage? i can't decide alone
- i'm not putting this on the Quest Store. this is a side-loaded open source thing. it's about your house, your data, your agent. nobody else's cloud gets to be in the loop. that's the whole point.
what i need from y'all:
- tell me which feature above matters most to you (or one i didn't list)
- if you've ever done OpenXR/Meta Spatial SDK, hit me up — i'm flying blind on a lot of the engine plumbing
- if you think the whole premise is dumb, also hit me up, i want to hear that too before i go further
github repo incoming this week once i clean up the readme. for now i'm running it on my own Quest in my own apartment in genova and it's weird and a little bit magic.
