u/sandeep-ai-xr-iot

OpenAI released its first hardware device called Codex Micro priced at $230.00 https://openai.com/supply/co-lab/work-louder/. It is an physical AI Agent controller and acts as a command center for agentic work with Codex. I thought, just for fun, I would build one for Ray-Ban Meta Display glasses.

Right now, Codex Micro on glasses can traverse through various models and effort levels. It can traverse through tasks, create new tasks, accept, reject, and most importantly, dictate in Codex desktop app.
It is not as deeply integrated as a physical device, because there is no public Codex API, so I have used macOS automation bridges.

In the video, I am first traversing through my Codex tasks and creating a new task. Then I traverse through Models and efforts, selecting GPT 5.5 Medium. I then used the dictation button and spoke, 'Explain the flow of my Codex micro project and increase the panel size by 30%’

I built NodNinja, It is a hands-free fruit-slicing game built for Meta Ray-Ban Display Glasses. It turns your natural head movements into a virtual sword, allowing you to slice fruit, dodge bombs, and chase high-score combos entirely hands-free.

Try it out on your glasses by connecting the web app below in Meta AI app.

https://nodsaber.sandeepk-ai-xr.com/

NodSaber ⚔️: Controling the saber entirely with my head: move my head to slide the saber across the screen, and tilt my head to tilt the saber, slicing neon blocks to the beat. And it's not just about slicing. Half the fun is maneuvering and dodging the blocks you're not supposed to hit, bobbing your head out of the way just in time.

How to Play:

 🎯 Calibrate first - hold your head steady and look straight ahead at the target dot while it locks in (3-2-1). A stable head here centers your saber and keeps your aim accurate for the whole game.

 🕹️ Aim with your head - turn and nod to move the glowing saber, and tilt your head left/right to tilt the blade so it banks with you. Smooth, gentle motions work best, avoid sudden jerks.

 🎨 Match the color - a blue saber slices blue blocks, red slices red. Chain hits to build your combo multiplier.

 ⚡ Watch the switch - your saber flips color every 15 seconds, so stay alert.

 💣 Dodge bombs & walls - never touch a bomb or a wall; move your head out of the way or you'll lose a life.

 ⭐ Grab the bonuses - gold blocks = double points (any color), and the rainbow power-up lets you slice any color for a few seconds.

 📈 Level up - every 1,000 points pushes you to the next level (up to Level 10) as blocks fly faster.

 ❤️ Survive - you start with 4 lives. Wrong color, a bomb, or a wall costs one. Last as long as you can and beat your high score!

Why DAT and not Web app?
Camera access :- Glass Coach needs the glasses camera stream, which is not possible in web apps today.

No server round trip :- Even if the Web App has camera access soon, every frame from the glasses camera would have to travel to a server, where the CV analysis runs, the overlay is drawn, and the augmented frame is re-encoded into a stream, which then has to be loaded back onto the glasses. That's a full network hop per frame. With DAT, the entire pipeline (frame in, CV, overlay) runs locally on the connected iPhone and glasses just displays UI Component.

Background :-
I wanted to try something where a camera stream flows from the glasses, computer vision is applied to every frame, and the result is sent back to the glasses to show on the display, like object detection with a digital overlay showing information about what's detected. Kind of an augmented reality experience. I'm aware that with these glasses it won't be a full AR experience because of the smaller display area, monocular projection, limited field of view and other logical constraints, but there are still a lots and lots of use cases.

So I thought of building a first use case, Glass Coach, where the camera stream comes in, Apple's Vision framework detects the body's pose landmarks on each frame, and I use those joint positions to count reps and check form for a particular exercise, and then a digital overlay is added to correct the form or guide you, and that gets streamed back to the glasses.

How it works?
Right now DAT only supports MP4 playback on the glasses, it doesn't support live streaming. So what I did instead: you select an exercise from a list, the camera stream starts, all the analysis happens, digital information is overlaid on each frame, and once the exercise is complete the resulting augmented video is played back on the glasses, As of now I've added just basic information, rep count, feedback, but there's a lot more that could go in.

Implemented a hands-free FPV (First Person View) experience for my Raspberry Pi car on Meta Ray-Ban Display glasses using web apps. The rendering on the glasses is amazing. I plan to extend this prototype to drones as well. I created PiCar, a web app that streams live video from my Raspberry Pi Car camera and provides directional controls (forward, backward, left, right, stop) to maneuver the car remotely. Streaming can be easily extended to a baby monitor, door camera, or FPV drone.

Streaming Flow
➡️ Pi Camera -> ffmpeg -> MediaMTX -> Cloudflare Tunnel -> Browser on glasses
➡️ Pi Camera - Built-in camera module on the Pi, exposed as /dev/video0.
ffmpeg - Reads raw frames from /dev/video0, compresses them into H.264 at 350x350 resolution, 15fps, and continuously pushes the encoded stream to MediaMTX using the RTSP protocol.
➡️ Why 350x350? Latency stays near zero.
➡️ Why 15fps? A tradeoff between smoothness and performance. Standard video is 30fps, but 15fps keeps the processing and bandwidth demands low enough for the Pi and the glasses to handle comfortably. I will push it to 30fps in future builds.
➡️ MediaMTX - Acts as a media router, receives the RTSP stream from ffmpeg and simultaneously re-serves it in multiple formats. I am using the WebRTC output because it's designed for real-time low-latency playback.
➡️ cloudflared - Runs silently in the background on the Pi and maintains a persistent outbound encrypted connection to Cloudflare's global network, no router configuration or open inbound ports required.
➡️ Cloudflare Edge - Sits between the public internet and the Pi. When someone visits the domain, Cloudflare receives the request and routes it through the tunnel directly to MediaMTX on the Pi.
➡️ Browser - Loads MediaMTX's built-in WebRTC player through the tunnel. WebRTC negotiates a direct media channel and starts rendering frames almost instantly.

Directional Controls Flow
➡️ Web App -> Button Click -> Cloudflare Tunnel -> Pi Motor API -> Car Wheels
➡️ PiCar Web App on Railway - A Node.js web app deployed on Railway serves the controller UI, a D-pad with forward, backward, left, right, and stop buttons.
➡️ Button Click -> Railway -> Cloudflare Tunnel - Tapping a direction button sends a request to the Railway server, which securely forwards it through an encrypted Cloudflare Tunnel to the motor API running locally on the Pi.
➡️ Motor API - A Flask server running on the Pi receives the command and translates it into GPIO signals sent to the motor driver board.
➡️ Motor Driver - The motor driver board receives the GPIO signals and controls the car wheels.