Lensdrop: MIDI Bleep Blop - a MIDI library and bridge so you can perform on live musical instruments (w/ a Tester Lens) #Lensfest May 2026 OSS
Overview
MIDI is a great way to synchronize musical instruments, play an orchestra of sounds for live or studio performance. Introducing MIDI Bleep Blop, a library + bridge to let you perform live. You can do studio work, sequence, or just mess around and jam. It doesn't produce it's own sounds, instead it sends MIDI notes, CC values, and other commands to a connected instrument. To get started, the fast set up is on a mac with Garageband.
- Clone the repo on Github: https://github.com/IoTone/MIDIBleepBlop
- Build the MIDIBleepBlopTesterWand lens in the lens folder
- Follow the quickstart in the README, and run :
node bridge/dist/cli.js --device "IAC Driver Bus 1"
For more complex setups with multiple devices, something like:
node bridge/dist/cli.js \
--device "IAC Driver Bus 1" \
--route "0=USB MIDI Interface"
Problem Solved
There are a lot of lenses released that can play music. But without BLE -> speaker and some BLE sink, we can't really get our sound out of the Spectacles. MIDI is a way to vibe inside of a semi-immersive XR experience and use a control surface designed for spatial computing.
I've been asked, why would you want to use AR for music? My answer: have you tried working in Ableton on a small laptop screen? Or on an iPad? It's constrained. In AR, we have space in all directions. Instead of encoders, knobs, and sliders, we can move our body and use our hands. Try it!
Our big design challenge right now is we can't import native libraries, and we don't have BT Classic, and we don't have Core MIDI, and we don't hae a full BLE stack. Ahh. So what to do? I explored a bunch of options instead. WebBLE: not supported in our browser. WebSerial: not supported. WebMIDI: not supported.
So the work arounds were not going to work yet. But streaming our MIDI notes and CC values and commands are just text. So a WebSocket is perfect. After some exeriments, it was clear this is going to be the approach right now. When full BLE support is there, and when we can port CoreMIDI (I'm assuming team Snap will do this) then we are cooking. A WIDI controller for example becomes possible and we can get rid of the bridge code.
Working Right Now
- You can use the MIDIBleepBlop prefab in your own projects, or include the core libraries
- You can communicate with MIDI channels 0-15 (labeled 1-16)
- You can send CC values, meaning you can actually twist the knobs on your MIDI gear ... sometimes unlock secret features
- You can code up your own "sequencers". We have an example called ChordSender which will send a loop of notes, following a theme: triad (great sounding chords), acid (house) , techno (for drums), etc. Right away, you can get a full set of tracks playing depending on your setup
- An example Piano scale prefab .... you can play notes with this. Fast press for full force, and long press and hold for either holding the note open, or on some instruments, this would decay
- The ability to build out your own control surfaces for MIDI ... the imagination is the only thing stopping you from making very interesting ways to control sound
Known Issues
- disclaimer: it doesn't make sound. My first midi controller I bought was very disappointing. It doesn't make its own sound. You need a midi device connected to get sound out.
- Spectacles may get glitchy when a GC hits, and this can affect the realtime nature of the sound and it will slow down or possibly skip a beat
- Playing multiple midi channels to the same device over an OMNI channel setup might create some unpredictable sounds, but this is fun sometimes
- The desktop bridge setup requires you know your IP address
- The Spectacles have to on the same wifi as your desktop bridge
- The UI configuraiton for MIDIBleepBlopTesterWand is tricky ... you will have to look at the docs, or just know a bit about MIDI... Look at PianoPlayer.ts inputs in your scene. This is the most important one to setup.
Design Overview
The main thing to overcome was an incomplete BLE support, and lack of CoreMIDI. I started on this back in January of 2026, thinking I could hand port some node.js libraries over. They all still rely on some underlying MIDI support in the OS. And the work arounds won't do.
So on the websocket side, you currently will need to know your IP address. This is available in the network settings of the Desktop bridge system. It's just a node.js program to run.
The real fun is designing your own instruments. We aren't limited to one midi channel or one instrument. We can do up to 16 in theory. Like, you could tie a midi channel to each finger and build a finger drummer for a full drum kit with 10 instruments!
For this one, I had 3 approaches I was after but had just time for one. I wanted the tester to support a keyboard input, ala Ableton. But as I got into it, I realized, I don't like the keyboard for this. I find the interaction to be too challenging. But hands and piano keys ... that is natural. So the design includes a typical one octave piano setup. You can change the input value to alter the octave used. Some thought had to go into how to manage the velocity of key strikes. And what about holding notes down. How about polyphony? Well, you have the ability to point with two hands. That means you can hit two notes.
But the sky is the limit literally. Your piano keys are in the sky :p . And wait till we get into SyncKit with this :p.
What's Next
Look for collaborations coming next month with other creators. We already have a few projects in the queue that transform the Spectacles skeptics into believers in "the music". So DM me if you'd like to collaborate on an instrument. I will continue to add to this repo, and build out new projects on top of this.
Thinking about making some MIDI devices with an Arduino like an RE Speaker for a true wearable sound demo. May explore at the upcoming xrvdc.org hackathon.
Attributions
- Someone released a drum kit app a few months back: I looked at it, and thought wow, if we could just unlock that performance capability so everyone can enjoy it
- Another very talented creator has been working on a project titled MINIMIDI (https://www.linkedin.com/pulse/minimidi-turns-snap-spectacles-ai-powered-dj-booth-elsa-farrokhzad-5bfhe/) which is so inspiring for the amazing controls and the fun experience. Now we just need to update this to work with the 6-8 different tracks available
- Another creator showed something a year ago or so with a monkey triggering midi though I never figured out how it worked, and no code surfaced, but it got me salavating at the MIDI prospect
- Not a Spectacles project but an AVP project, Animoog Galaxy and the iOS version Animoog Z are the most amazing control surfaces ever, and they are synthesizers. What is more amazing than painting music with your fingers to the sounds of Moog? Painting them in 3d. I spent a lot of time thinking about Moogs innovations in sound design with their apps. Such great work!
Thanks to the Spectacles team and Lenslist #Lensfest team for supporting our work. #Lensfest May 2026 OSS.
Support
DM me, or hit me up on the github issue tracker. I'll take PRs too.