Boosting QWEN 2.1 CLIP image encoding (!probably AMD only!)
Hi everyone!
Well this is my first contribution for community with the help of vibe coding :)
TL;DR: links at bottom that boost Qwen Image Edit clip encoding up to x7 times!
(51.3 s → 6.7 s (~7.6x) for the two custom encode/clip nodes.)
____
Context:
I am running comfyui-rocm Patientx-cfz build,
ComfyUI 0.39.0 / ComfyUI_frontend v1.53.10
Win 10, AMD 7900XTX, 5900x, 64Gb ram
PyTorch Version 2.15.0a0+rocm10.2.0a20261006
Default comfy WF with kitchen attention and 6-step lora applied.
So I really was upset with how painfully slow Qwen 2.1 edit was with image refs. CLIP taking much longer than sampler itself. I saw people suggested lowering input images resolution, but that's always led to details loss on my tests. And re-encoding images on prompt change was making it even worse. So I asked OpenCode and DeepSeek4.1 Flash to look at it and ask to add separate image encoding node and clip encode, so encoded references at least could be cached.
But Ai found even more optimizations!
So my test workflow with 1 ref image with resolution set to 1248 in qwen clip node and final render res of 1088x1440:
Orig - 66s
Mod - 21s
That's some real big win I think! And this actually made possible to take my ref image with original size of 1440x1920 and be as fast as 40s!
As a bonus we made separate Qwen Image 2.1 Reference and Text Encode Qwen Image 2.1 Fast, so now if you don't change ref images and change only prompt - images doesn't needs to be re-encoded!
2nd run with only Prompt changed:
Orig - 62s
Mod - 18.5s
These custom nodes doesn't modify any ComfyUI files. When you run one of these nodes it patches the Qwen VL vision encoder in memory. The QwenVL Vision Linear Patch toggle switches the vision patch-embed between the original Conv3d and an equivalent linear op (bit-identical output), so you can A/B in the same workflow.
Currently ONLY TESTED ON MY MACHINE!
What I don't know - is if that work on someone's else hardware, especially NVidia, so if you interested to try it out - you can do it and tell here :)
Also - Ai told me that this may also help other models that use the Qwen VL vision tower (e.g. MiniMax H3), untested so far.
LINK to nodes.
Workflow (with original AND fast nodes in one wf) Simply turn on/off groups you don't want to run (use original OR fast)
How to use:
-Unzip into custom_nodes, restart ComfyUI, hard-refresh the browser. Nodes appear under QwenVL Vision.
- Put QwenVL Vision Linear Patch after your CLIP loader and set it to ON. It swaps the vision Conv3d for a mathematically identical linear op — output is bit-identical. Toggle OFF to compare in the same workflow.
- Use Qwen Image 2.1 Reference → Text Encode Qwen Image 2.1 Fast, so changing only the prompt no longer re-encodes the reference images.
- Original Text Encode Qwen Image 2.1 node is still slower, as it encodes every image twice: once for pos and neg prompt - mod does it once, as original encode is same for both anyway, independent from text.