u/CutWorried9748

What's better than 80's vibe text based interactive fiction? How about, 2026 inspired, synthwave fueled spatialized Interactive Fiction? We did it. In your Lens today. Choose your own adventure path, among 11 unique stories. This is over 10 + hours of playable contents. An AI pipeline handles the modernized flow, you will see visual content and hear a narrator. It is completely revolutionary. SPEC '24

Since nobody really saw this in the previous release, I've provided a better demo video for the snap team to consume for their marketing, and plug for creative use of the platform. This is , in effect, a retro console VM running natively on the platform, no IP violations, everything is public domain. Did we mention it's open source?

Snapcode: https://www.snapchat.com/lens/bd967677f1ad4488a9f19f5bf11503e1?utm=qr

Source: https://github.com/IoTone/IfWhenZMachineSpectaclesXR/tree/ux2

What's New in v0.2

- New UX design with 80s inspired CRT

- innovative "game" selector,

- innovative command input UI

Bug Fixes in v 0.2

- Fix broken AI model (it was using an older AI model no longer available, causing image generation to fail)

Support

- DM me if you find any issues, make a PR

Thanks, like and subscribe!

https://www.youtube.com/watch?v=GqwRtmpSiUQ

(youtube link)

SPECS '27 version coming soon!

#LENSFEST OCT 26 (not SEPT)

The "XR" Book of Math is a math text book made up of 80 theorems and formulas across 4 disciplines: Algebra, Linear Algebra, Calculus, and Geometry. It provides visual, spatial proofs.

Included in this is an Open Source Release of a LaTeX library port of Katex (https://katex.org/) called Matematex.

Lens Snapcode: https://www.spectacles.com/lens/37da717d4339468aad447b8b81e6d0a8?type=SNAPCODE&metadata=01

Source: https://github.com/IoTone/Spectacles-Matematex

The Problem

Typesetting is hard. In late '24 we tried to build a math app (See Space Math V). It was really hard because we couldn't style the text elements individually. It meant we had to have layers and layers of text elements per formula. Too hard. What the world uses for typesetting is TeX (LaTeX). It works very well for layouts. Superscript, subscript, variable italics, colorization of certain items, etc. So this is what we wanted.

Design Challenge

LaTeX libraries all depend on ability to render in a browser, or to a PDF/EPS file. We have neither at our disposal in a SPECS Lens. Fortunately, some months back, I ported a series of things to support traditional browser features in a regular lens, something called Polynode: https://www.iotone.co/projects/spectacles-polynode/ . This is free to use. We needed SVG or HTML5 Canvas. I got SVG implemented (as a subset of total functionality) natively in spatial. Once this hurdle was cleared, we could totally start porting an existing library for LaTeX.

Approach

We had a few choices to start from. Mathjax or Katex. Both have similar approaches. They prefer a browser context, a canvas, or they generate an SVG. Or you are going to make a PDF. We had none of these. But once we fixed the support that was missing for SVG, we had our path. So this is a straight port of Katex. Typesetting and layouts still had to be tuned for screen specs "literally". Lots of eyeball testing.

An XR Textbook

I then went after the prize. A real book. For math. All those brutal subjects that were hard to visualize. Every formula and theorem has a proof associated (with the exception of a few). Then the ask: the LLM built visuals. I've found my brain expanded in this process to understand the complex topics. This is the 1st Ed. I hope to keep going on this with edutech.

Support

You can file PRs https://github.com/IoTone/Spectacles-Matematex/ . The project was peer reviewed by Jorge Pardo (Mathematician / Teacher / Entrepreneur) based in Japan. You can find out more about using the library in your own lens: https://github.com/IoTone/Spectacles-Matematex/blob/main/MATEMATEX.md

Introducing the world'd first Spatial Z-machine from the 1980s, modernized to enjoy the best features of Snap Spectacles. Interactive Fiction is a genre of gaming/fiction fusion allowing the reader to guide themselves through a story. The worlds are vivid an imaginative. This is the perfect use of XR+AI and some good old fashioned 8-bit hardware (Z-machine).

Lens is available at Snapcode: https://www.spectacles.com/lens/bd967677f1ad4488a9f19f5bf11503e1?type=SNAPCODE&metadata=01

Features

- Unique modern UX for play, instead of typing at a keyboard, you have a push / pinch menu of possible commands

- Use the physical "microphone" (touch it) to give voice input commands instead of

- Every room generates a new Generative AI image, with unique and beautiful graphics

- Every image gets spatialized for an amazing and awe inspiring effect

- 11 Unique open source IF titles for play, meaning hours and hours of new game play possible over adventure, fantasy, mystery, literary, and sci fi

- ability to save and restart games

Design Challenges

The goal was at age 5 to some day create this. I just didn't know how when I was 5. I started early last year on porting the tszm library but ran into some issues, got it mostly packaged up and building and loading in my Lens. I wasn't sure on the next step for loading IF content, since these are binary files. The best approach was to convert the IF contents into typescript, and load these as binary encoded contents into the Lens. The engine handles this nicely. So the right approach is then to run all of this locally to get the parsing working on a desktop. After that, an LLM can automate the bug fixing of the parser, based on various generations of a zmachine format. Each IF contents has their own unique features, which is a problem. Some content isn't a good match for our design, especially when the games want open ended writing. We need fixed options for the input.

Once the parser was up and running in a Lens, the question was, what to show? Start with a generated AI image. Use a common prompt to make sure everything cleanly looks similar. Then spatialize each image.

The dynamic menus are a compromise for input. Keyboards just aren't great yet, so this was a good approach to try. Push buttons don't work so great, so squeeze seem to be best on Specs.

Adding TTS really gives it life. Making sure we "warm up" the TTS is important. We need this not to feel too laggy. And adding STT input via a hand grab microphone is a unique idea and worth trying, though I think the mic reliability is about 60%.

References

- The full archive of all IF: https://www.ifarchive.org/

- Z-machine history https://en.wikipedia.org/wiki/Z-machine

Attributions

- Microphone 3d asset: https://sketchfab.com/3d-models/microphone-cf9580db045e4bd6818306d50f2fa2ce

- Audio: Original IoTone Japan

- Z-Machine Specification https://inform-fiction.org/zmachine/standards/ http://zspec.jaredreisinger.com

- tszm: nice work making a compact zmachine in TS and making it portable to browser or node.js https://github.com/cshepherd/tszm

- Idea for caching genAI contents zmcdn: https://github.com/cshepherd/zmcdn

- PoC Research tszm / IF AI : https://browneverettlewis.com/case-studies/tszm-tscdn.pdf

- This informed some thinking about possibly making room maps (future feature) ZorkUI: https://github.com/posabsolute/zork-ui/tree/main

Licenses

| Game | Author / Year | Source file (ifarchive.org) | Status |

|------|---------------|------------------------------|--------|

| Adventure (350 pt) | Crowther & Woods, 1976; Inform port G. Nelson | (bundled `advent.z5`, Release 9 / 060321) | ✅ original is public domain; Inform port freely distributable |

| Lost Pig | Admiral Jota, 2007 | `/if-archive/games/zcode/LostPig.z8` (Rel 2 / 080406) | ✅ freeware; IF Comp 2007 winner |

| The Dreamhold | Andrew Plotkin, 2004 | `/if-archive/games/zcode/dreamhold.z8` (Rel 5 / 041231) | ✅ Plotkin distributes his games as freeware; source published on the Archive |

| Christminster | Gareth Rees, 1995 | `/if-archive/games/zcode/minster.z5` (Rel 2) | ✅ freeware; long distributed free by the author and via the IF Archive |

| Suveh Nux | David Fisher, 2007 | `/if-archive/games/zcode/suvehnux.z5` (v1.1, Rel 1 / 150314) | ✅ freeware; One Room Game Comp 2007 winner |

| 9:05 | Adam Cadre, 2000 | `/if-archive/games/zcode/905.z5` (v1.1, 2012.0724) | ✅ Cadre distributes his games as freeware |

| Delusions | C.E. Forman, 1996 | `/if-archive/games/zcode/Delusns.z5` | ✅ freeware sci-fi mystery; from the IF Archive |

| Spider and Web | Andrew Plotkin, 1998 | `/if-archive/games/zcode/Tangle.z5` (Rel 4 / 980226) | ✅ Plotkin freeware |

| Metamorphoses | Emily Short, 2000 | `/if-archive/games/zcode/metamorp.z5` | ✅ freeware; from the IF Archive |

| Slouching Towards Bedlam | Star Foster & Daniel Ravipinto, 2003 | `/if-archive/games/competition2003/zcode/slouch/slouch.z5` | ✅ freeware; IF Comp 2003 winner |

| Mini-Zork I | Infocom, 1988 | (bundled `minizork.z3`, Rel 34 / 871124) | **Infocom/Activision property.** Historically given away (UK magazine cassette)

Introducing Shogi Zero, the world's first Japanese Chess made for XR. Shogi is an old board game, dating back more than 500 years. Like its predecessor Chess, it features a variety of complex moves that require years to master. We have built in the tools to help learn Shogi. Hugely popular in Japan as a game, Shogi is played competitively in tournaments and has weekly TV shows showing tournaments. We are excited to bring the spatial experience of Shogi to your Spectacles.

Lens Snapcode: https://www.spectacles.com/lens/23243eb663144bb3892d5706dc417ed8?type=SNAPCODE&metadata=01

Features

• Play on any table, easy to reposition via hand menu
• Realistic sound effects (resembling the sound of real Shogi tiles hitting a wooden table)
• Built in trainer, with KIF playback, 6 games built in to review as a new player (Shogi game format)
• 3 Levels of AI: random, copycat, and smart
• Supports English / 日本語
• Built in hints
• Relaxing play environment

Why Shogi?

The game exceeds chess in statistical complexity, and provides more nuance and variation than chess. You can also get a healthy dose of Kanji to expand your horizons while playing. Games can go quickly or slow as you like while you learn. No pressure from an impatient human player.

Instructions

The game itself has built in study guides, but it is always worth reviewing rules found on Wikipedia for more details in your native language. https://en.wikipedia.org/wiki/Shogi . It is also recommended to check out online videos for Shogi play in a tournament or instructional setting. A great book on the topic can help, so check with your local book shop or game store.

Design Challenges

Chess has been built many times for XR and VR. We wanted to build something to bring the Japanese culture of strategic board games such as Shogi out to the world. The Kanji can be a barrier to learning. We tried to craft the Kanji to be beautiful, so that each piece is interesting to look at.

Interacting with game pieces can be a challenge. We have experimented with approaches to showing valid pieces that can be moved, and also building in a hint mode so that you can get the optimal next move as you learn.

Building AI for chess is hard ... it's been done over and over, and the same challenges emerge. How long do you have to wait for your AI to compute options before the human player gets bored? And what is your. memory budget. We quickly found that our code, which uses a lot of recursion (via LISP) would hit memory limits in practice when executed on device. A lot of tuning was done to get a medium quality AI working well without needing any LLM or online connection.

Because the game Shogi has 500 years + of evolution, the "modern" changes to the game from 500 years ago add some UI complexity. We have adopted a simplified approach to promotion of pieces, and avoid some of the more complex UX design.

Future

We hope to add multiplayer mode. The immediate goal is to improve the ability to add KIF files for spectator mode tutorials. Real Shogi games play every day, and players compete to increase rank. We would like to add voicing to the game scenarios as well . We want to improve the study contents further to make this more in depth.

Attributions

We used a great audio track found on freesound by this content creator: https://freesound.org/people/zagi2/sounds/222655/ .

We used a port of Petit Parser (https://petitparser.github.io/) to modern typescript, built here https://github.com/IoTone/petitparser-ts/tree/master, which gives us LISP and Prolog among other things.

Made with love in #Fukuoka #Japan. Thanks to team #SPECS and #LENSFEST for support.

Overview

MIDI is a great way to synchronize musical instruments, play an orchestra of sounds for live or studio performance. Introducing MIDI Bleep Blop, a library + bridge to let you perform live. You can do studio work, sequence, or just mess around and jam. It doesn't produce it's own sounds, instead it sends MIDI notes, CC values, and other commands to a connected instrument. To get started, the fast set up is on a mac with Garageband.

- Clone the repo on Github: https://github.com/IoTone/MIDIBleepBlop

- Build the MIDIBleepBlopTesterWand lens in the lens folder

- Follow the quickstart in the README, and run :

node bridge/dist/cli.js --device "IAC Driver Bus 1"

For more complex setups with multiple devices, something like:

node bridge/dist/cli.js \

--device "IAC Driver Bus 1" \

--route "0=USB MIDI Interface"

Problem Solved

There are a lot of lenses released that can play music. But without BLE -> speaker and some BLE sink, we can't really get our sound out of the Spectacles. MIDI is a way to vibe inside of a semi-immersive XR experience and use a control surface designed for spatial computing.

I've been asked, why would you want to use AR for music? My answer: have you tried working in Ableton on a small laptop screen? Or on an iPad? It's constrained. In AR, we have space in all directions. Instead of encoders, knobs, and sliders, we can move our body and use our hands. Try it!

Our big design challenge right now is we can't import native libraries, and we don't have BT Classic, and we don't have Core MIDI, and we don't hae a full BLE stack. Ahh. So what to do? I explored a bunch of options instead. WebBLE: not supported in our browser. WebSerial: not supported. WebMIDI: not supported.

So the work arounds were not going to work yet. But streaming our MIDI notes and CC values and commands are just text. So a WebSocket is perfect. After some exeriments, it was clear this is going to be the approach right now. When full BLE support is there, and when we can port CoreMIDI (I'm assuming team Snap will do this) then we are cooking. A WIDI controller for example becomes possible and we can get rid of the bridge code.

Working Right Now

- You can use the MIDIBleepBlop prefab in your own projects, or include the core libraries

- You can communicate with MIDI channels 0-15 (labeled 1-16)

- You can send CC values, meaning you can actually twist the knobs on your MIDI gear ... sometimes unlock secret features

- You can code up your own "sequencers". We have an example called ChordSender which will send a loop of notes, following a theme: triad (great sounding chords), acid (house) , techno (for drums), etc. Right away, you can get a full set of tracks playing depending on your setup

- An example Piano scale prefab .... you can play notes with this. Fast press for full force, and long press and hold for either holding the note open, or on some instruments, this would decay

- The ability to build out your own control surfaces for MIDI ... the imagination is the only thing stopping you from making very interesting ways to control sound

Known Issues

- disclaimer: it doesn't make sound. My first midi controller I bought was very disappointing. It doesn't make its own sound. You need a midi device connected to get sound out.

- Spectacles may get glitchy when a GC hits, and this can affect the realtime nature of the sound and it will slow down or possibly skip a beat

- Playing multiple midi channels to the same device over an OMNI channel setup might create some unpredictable sounds, but this is fun sometimes

- The desktop bridge setup requires you know your IP address

- The Spectacles have to on the same wifi as your desktop bridge

- The UI configuraiton for MIDIBleepBlopTesterWand is tricky ... you will have to look at the docs, or just know a bit about MIDI... Look at PianoPlayer.ts inputs in your scene. This is the most important one to setup.

Design Overview

The main thing to overcome was an incomplete BLE support, and lack of CoreMIDI. I started on this back in January of 2026, thinking I could hand port some node.js libraries over. They all still rely on some underlying MIDI support in the OS. And the work arounds won't do.

So on the websocket side, you currently will need to know your IP address. This is available in the network settings of the Desktop bridge system. It's just a node.js program to run.

The real fun is designing your own instruments. We aren't limited to one midi channel or one instrument. We can do up to 16 in theory. Like, you could tie a midi channel to each finger and build a finger drummer for a full drum kit with 10 instruments!

For this one, I had 3 approaches I was after but had just time for one. I wanted the tester to support a keyboard input, ala Ableton. But as I got into it, I realized, I don't like the keyboard for this. I find the interaction to be too challenging. But hands and piano keys ... that is natural. So the design includes a typical one octave piano setup. You can change the input value to alter the octave used. Some thought had to go into how to manage the velocity of key strikes. And what about holding notes down. How about polyphony? Well, you have the ability to point with two hands. That means you can hit two notes.

But the sky is the limit literally. Your piano keys are in the sky :p . And wait till we get into SyncKit with this :p.

What's Next

Look for collaborations coming next month with other creators. We already have a few projects in the queue that transform the Spectacles skeptics into believers in "the music". So DM me if you'd like to collaborate on an instrument. I will continue to add to this repo, and build out new projects on top of this.

Thinking about making some MIDI devices with an Arduino like an RE Speaker for a true wearable sound demo. May explore at the upcoming xrvdc.org hackathon.

Attributions

- Someone released a drum kit app a few months back: I looked at it, and thought wow, if we could just unlock that performance capability so everyone can enjoy it

- Another very talented creator has been working on a project titled MINIMIDI (https://www.linkedin.com/pulse/minimidi-turns-snap-spectacles-ai-powered-dj-booth-elsa-farrokhzad-5bfhe/) which is so inspiring for the amazing controls and the fun experience. Now we just need to update this to work with the 6-8 different tracks available

- Another creator showed something a year ago or so with a monkey triggering midi though I never figured out how it worked, and no code surfaced, but it got me salavating at the MIDI prospect

- Not a Spectacles project but an AVP project, Animoog Galaxy and the iOS version Animoog Z are the most amazing control surfaces ever, and they are synthesizers. What is more amazing than painting music with your fingers to the sounds of Moog? Painting them in 3d. I spent a lot of time thinking about Moogs innovations in sound design with their apps. Such great work!

Thanks to the Spectacles team and Lenslist #Lensfest team for supporting our work. #Lensfest May 2026 OSS.

Support

DM me, or hit me up on the github issue tracker. I'll take PRs too.

^(Reposting of my original post to /Spectacles :) ^(https://www.reddit.com/r/Spectacles/comments/1sfqgd3/oss_lensdrop_skywriterble_a_prototype_distraction/)

^(All source (fork of the BLE HID project for Cardputer ADV) and Lens is posted to GH. This is an experiment in authoring content in AR. How fast can one type with real hardware vs "air" keyboards. Also, what would a distraction free environment look like if you didn't have 100s of features staring at you and constant pop ups about updates and AI trying to help you. I plan to use this for explorations in interactive fiction in XR, matrix clients in XR, etc.)

^(Definitely some things that can be improved in the experience but I like hardware keyboards over XR keyboards. Open to ideas. The Spectacles 24' don't have a real full BLE api yet, just partial supoprt. On the "Lens" side of things, since this isn't WebXR, it isn't portable to other platforms. Without moving to a websocket type setup (which would work but is complex to make just work), I haven't figured out a good way to get Web Bluetooth working in XR browsers (XR browsers don't get many advanced bleeding edge features). Perhaps a fork of Wolvic could be made to support this, and then I can rewrite the client in threeJS / three-mesh-ui . Any suggestions on getting battery level details working over BLE would be appreciated too. Currently can't seem to read that. Thanks for reading. Really enjoy the cardputer for its utility. Feel free to DM me if you have questions or post to the GH.)