u/tomdrum6

Following on from my other post - I’m currently developing a tool concept that explores how the Spectacles could help visually impaired or vision-limited users understand their surroundings and be a useful tool for assisted living in the future. My other post outlined the proximity detection and basic spatial object detection.

A couple more tools I've added

• Path Drawing: A feature that "paints" a bright path onto surfaces. This tool could help vision-limited people better see the shapes and layout of the space around them, or trace a path. It also has proximity detection so the colour of the path changes based on distance.
• Ai View Description: I can say "Spectacles, what am I looking at?" and it sends a snapshot of the current view to ai (gemini) to summarise and return in TTS in real time. can definitely see this combination of glasses & ai analysis being super helpful for people with no or limited vision get an audible description of their surroundings.

Going to tidy it up and release it to test out properly soon. And then im going to look into adding the "depth cache" ai view tags, which look perfect to add to this project!

Any other tools that you think could be useful for this concept?

I’m currently developing a tool concept that explores how the Spectacles could help visually impaired or vision-limited users understand their surroundings. It's using world mesh, raycasting, some Ai and voice commands/TTS to help understand our surroundings. it's a working prototype of how wearables could be a useful tool for assisted living in the future.

Here are a few features I’ve implemented so far:

• Proximity Detection: Using the world mesh, we can measure our distance to physical objects (not sure if it's super accurate right now). The proximity ring changes colour based on distance, with an audible alarm as you get closer.
• Spatial Object Tagging: Currently a simple solution that estimates an object based on the mesh height & surface normals. But we can identify walls, floors, or simple objects. We can then pin a persistent 3D card to them. NOTE: I just saw the "Depth Cache" template which uses ai to tag object and anchor them spatially which is EXACTLY what I wanted this tool to do, and will explore next!
• Object Summary (TTS): Users can ask for a summary of nearby objects. The TTS system reads back any 3D cards generated and their relative positions to us.

Ive also got a few more tools in there as well, such as a basic Ai view description, and painting a path onto the world, but I'll outline those in a follow up post, as this is getting long....

I'd love to hear your thoughts, any suggestions, or any tips. I really want to add the "Depth Cache" template (go check it out) into this concept as it looks perfect for this idea.