Air Write: accessibility tool for writing words and letters in the air with the phone, like a wand!
https://reddit.com/link/1vnykmm/video/lcd5wgpnj8gh1/player
Here is the linkedin demo, where I'll be leaving the github link!!
Air Write is an accessibility tool I developed that classifies and recognizes letters and words written in the air with your phone, like a wand (yes, like in Harry Potter... shouting expecto patronum is optional). I built it with limited fine motor control in mind. It's a web app that runs in the phone's browser on accelerometer and gyroscope data alone.
A touch keyboard requires you to hit small fixed targets accurately, and a stylus requires precise tiny movements and limits by screen size. Air-writing only requires gross motion at whatever size is comfortable for the user, calculating the path travelled through air so the same letters and words are read whether drawn with the wrist or whole arm.
It detects when to start and stop recording from the motion itself, and works at any grip angle. Trained on 1,645 letter recordings and 1,008 word recordings, the convolutional network reads isolated letters and words written as one connected stroke by classifying candidate letter regions and evaluating the most probable combination of letters (fitting to dictionary spellings if all letters aren't recognized with high confidence).
Since I had limited time this semester to collect and tag data to perfect the model, I will be releasing this project open source with the hope that others can further improve it!