u/vatojavier

I've been experimenting with algorithms that tries to separate sounds and then cluster them together, the idea is to give a trippy synesthetic experience when those clusters are mapped to visuals and keep a sense of cohesion.

In this demo I projected the data coming from a track into a 3d space (I used the Godot game engine), then a real-time clustering algorithm groups the different sounds together. Although the whole pipeline is not real-time yet.

I recently started experimenting with audio related analysis and I'm no expert in audio analysis or vjing so I'm not sure if this can give some advantage over existing software. Do you think something like this could be useful for vjing by linking the clusters to different visuals?