Unicasso: Image to ASCII using CLIP based Optimization
Hi all!
I've been working on building a better image to ASCII generator. A lot of the options out there are quite simple — luminance based, single classifier networks or hand-engineered conversion processes that tend to fail in many cases. So over the last two months I've been working on trying to create a somewhat more sophisticated converter.
The general technical approach is to use CLIP as a judge on the ASCII art (akin to CLIPasso, for those familiar), while trying to make the discrete Unicode glyph space differentiable in a sensical way: a small VAE embeds the font's glyphs so a cell's choice can move continuously, each cell also keeps a small population of weighted candidate glyphs — heuristic channels nominate alternatives, and a nomination is only admitted after being swapped into the actual render and measured.
There is also support for creating ANSI art (examples also attached!)
At this point I've tested the code on macOS (MPS) and Linux/CUDA; Windows is unexplored. A render takes roughly an hour and 8 GB of memory, depending on image size — there's a fast preview mode if you just want to watch it converge.
Here is the repo!: https://github.com/jakobrees/unicasso
(I'm still working on writing up a paper to detail all the little bits -- any helpful feedback would be very welcome :-) !)
(The colored images that have been turned into ANSI are are images by Ladypurple on Civitai)