Waza: streaming the glasses' POV to a realtime voice coach for hands-on skills (DAT + GPT-Realtime + Gemini)
I spent the past two months building Waza, an iOS app that pairs with the Meta glasses through the Device Access Toolkit and streams the first-person view to an AI coach. The coach talks you through hands-on skills (could be DIY, cooking, handicraft), responds in about a second, and plays a short POV clip of each step at the right moment, so you glance at the phone while keeping both hands on the work. The concept is to create an interactive, upgraded YouTube tutorial experience.
A few things that might interest this sub:
- Skill authoring is fully automated. You record yourself doing the task once while narrating, and a Gemini-based pipeline segments the footage into steps, trims and zooms each clip, and writes the coaching prompt. Ten minutes of raw demo footage becomes a polished skill in about 2.5 minutes, no human in the loop.
- The voice model drives the app UI too. GPT-Realtime-2.1 calls tools mid-conversation, so the same model that's coaching you also searches the catalog, loads skills, and pauses sessions. No tapping, which matters when your hands are wet or greasy.
- Some of the hard parts so far: a very jittery Bluetooth Classic link. Wi-Fi transport arrived in 0.8.0, but no Wi-Fi Aware (NAN) yet.
Write-up with demo videos: https://vinidlidoo.github.io/blog/waza-ai-coach-for-your-hands/
I'm recruiting a small group of testers (Meta caps release channels at 100 testers each). If you run an iPhone and own Meta glasses, the invite is at the end of the post. Happy to answer any questions.