ComfyDesigner : Blurring the line between ComfyUI and TouchDesigner
Hello dear community!
I've been taking advantage of Fable to help me build a tool that i've been missing for a long time at this point. I called it ComfyDesigner. I guess some of you might be familiar with the complete lack of true integration between GenAI tools and traditional projection software like Touchdesigner. I always found that it was a nightmare to use AI workflow with projection mapping software.
That's what motivated me to blend both ComfyUI and TouchDesigner together.
Here is a non-exhaustive list of features included, keep in mind it's still very much a WIP :
- Native cluster architecture : many times I'm a bit lacking of power to run what i want on my laptop or my laptop is far away from projectors or camera i use. That's why I built this project to be natively working as a cluster. My current testing setup includes a 5090 computer that i use as a server that does most of the heavy lifting, accessible remotely if needed, a raspberry pi 4 2gb that usually only handles webcam(s) and projector, and my laptop, on which most of the time i dont run the software on because all the devices are on the same LAN but might be useful in case of WAN to reduce latency on some parts of the workflow. This makes I think for a very flexible architecture.
- Everything is a plugin : the software has been built from the ground up to be suuuper easily extendable with plugins like you would with ComfyUI. Integrating new nodes should be super easy when a new feature is requested or a new model come out
- Browser first : Something that has been bothering me a lot with TouchDesigner is it's monolitycal architecture and restrictions on the hardware on which you can control it. I loved the ComfyUI approach that you can run remotely on your laptop or even on your phone
- OpenSource in the heart : I've been very disappointed with the way the community handled sharing around TouchDesigner and in a lesser way with ComfyUI. It is a shame that workflows and plugins are sometime paywalled limiting the actual sharing of knowledge and the growing of a community. This is why this software will be 100% open source with special licence for companies willing to not share their work. Hopefully it should avoid the community to be gutted with patreons while allowing people who need to keep their work private to be able to. The absolute goal as you understand is to keep it free for artist and to try to make a buck on companies or museum using it in events (if it happens at any time :p )
Now a little example to give you a little preview of the project :
The pictures you will find above are webui screenshots. The setup is my usual, Raspberry pi4, connected to a usb webcam and an hdmi projector, both looking in the same direction, the software running on my 5090 server, and I'm working from my laptop browser. In the workflow on the screenshots, I'm doing a structured light scan to calibrate the projection POV with the webcam one (Hello LightForm :p), then taking the produce map to warp the webcam image, allowing me to reproject on the object or wall im projecting on aligning textures and shapes together. I also get the webcam of my laptop through the web browser on which I apply a mask using mediapipe to segment myself from the selfie, i composite both images (the warped webcam and the laptop webcam filming myself) and do an OpenCV edge detection on the composite. I then inject that inside FluxRT that generates high quality interactive video at around 40 to 50fps with a latency around 1sec. This is a simple example but I want to extend it with MIDI control from instruments connected through the browser.
I WILL publish the project in a near future but i would like to iron a few bugs before!
It has been a few week that I'm very excited with this project and wanted to see a bit the feeling of the community around this project, do you find it interesting? what features would you want to see? do you see projects it could help you with? specific workflows?
This lengthy post obviously hasn't been written with AI :p
For the mods, as I'm not sure this would be considered as branding I did put the promotion flare on ;)