u/nickpettit

My partner and I brought our consumer-oriented music visualizer (Vibralizer on Steam / Itch.io) to Iceland Eclipse Festival for an installation. We had flight simulator levers, foot pedals, and an Elgato Streamdeck for a playful audience interactive piece we called “The Many Handed VJ”. It was a hit with everyone that tried it!

However, we were then unexpectedly asked if we could VJ some sets in the Afterglow dome. Not wanting to miss the opportunity, we said yes. By day I’m a game developer, but I’ve never actually VJ’d before, and I thought “How hard could it be?”

Dang y’all, I have a newfound respect for your performance skills. It was super fun trying to match the musical structure, think about what visuals to transition to next, and compensate for signal latency to the screens, all while people asked if I could plug in their phone to charge. I learned so much and I can’t wait to synthesize the lessons into better performance tools in the future!

My partner and I started building a music visualizer for a little over a year ago as a side project, and once we had more than just a few scenes (we call them "Vibes"), we realized we needed a way for the user to navigate them. Our first instinct was to just create a more typical screen space canvas with a scroll view and thumbnail image or flipbook sprite for each one. But we then got really attached to a marble metaphor, where each one is more like a tiny spherical world you can jump into (we're still planning to do transitions inspired by Cocoon).

We wanted the marbles to feel very weighty and physical - something that you want to collect - and I realized that we could probably do some pretty neat shading on the shadows for our marbles by taking advantage of two key constraints:

1. The marbles are spheres and can be generalized as such.
2. The surface they sit on is planar.

This meant we could turn to the oldest trick in game dev lighting: capsule shadows! This is where a circle is projected onto a quad some way or another and it's how Mario's shadow was rendered back in Mario 64 (just a sprite that scales and fades in and out). Only in our case, we would shade our quads with shadows that are warmer toward the center. I also added in some color from the marble itself, adjust the intensity of the caustic highlight based on how thick the glass should feel, and then finally we throw in a low-mip of the cubemap reflections, roughly emulating how spherical glass can act as a lens and project its surroundings onto a surface (we had a real glass sphere for reference and noticed this phenomena). Then, the directional stretch of the shadow is driven by a directional light in the scene.

It's not at all physically accurate or doing any kind of light transport simulation, but as a stylized approach I've been really enjoying it.