Made an app that allows you to control your Mac with hand gestures through webcam (cursor, clicks, drags,speech-to-text, AI agent)
I got tired of typing and reaching for the trackpad while vibe-coding, so I built this.
Gstrl uses your Mac's Apple Vision to detect hand gestures and turns them into actual input:
- Pinch to move the cursor, click, drag
- Draw a circle in the air → screenshots that region
- Open hand swipe → arrow keys
- Hold a fist → speech-to-text (say "press enter", "command z", etc.)
- Hold both fists → talk to Claude, it speaks back
All on-device (Apple Vision + SFSpeechRecognizer). Open source, MIT.
Still not perfect, any feedback is welcome!!
u/Honest_Fisherman8993 — 10 days ago