Repo here- https://github.com/Adudeguyman/ComfyUI-Fantastic-MiniMaxH3-PromptBuilder

Ooookay back again. I made RefMods work a bit better and solved a lot of identity/bleed issues. At the cost of time and memory, of course.

What it do-

OG vanilla refmods would save your media as latents in a .safetensors file, and feed all of those into the MiniMax H3 DiT directly, with no references and the text encoder never saw it. This worked fine with one refmod since the model knew it had to do SOMETHING with those references, and it did it surprisingly well. But that also lead to people stacking multiple refmods, etc, to get a good likeness.

My previous implementation would feed part of those latents to the text encoder, at the cost of re-encoding a few frames (every 4th would be presented to the vision model as a block of 2) and have the TE look at those. This was aimed to give reference anchors to the model when prompting (like using "<Subject 1> is the person in <Video 1>"). This caused a slowdown because those frames were encoded every generation for the TE. Which a user on GitHub complained about, so I looked into speedups and caching.

Lo and behold, since .safetensors are just wrappers like mp4's and mkv's, you can store multiple types of data alongside latent data. So the speedup was easy, save the refmod pictures as actual jpeg data, alongside the latents in the same file. Think of it as having subtitles inside of a video; different type of data wrapped in with video and audio. So packing in actual jpeg data, you can feed those directly to the text encoder without going through the VAE, and the TE's conditioning can cache between runs so that as long as you don't change order or strength of your refmods; all of that every-run-processing is skipped. And the jpeg data is never passed to the model, only the packed latents, so it functions much like a refmod regularly does. But that got me thinking, why not change how the frames are presented to the text encoder for better conditioning?

Enter the "stack_pictures" function on the Fantastic H3 RefMod Text Encode node. By default, "every 4th" will keep the existing behavior of sending every 4th frame in chunks of 2, and it is the lightest to run and good for 1 refmod. "up to 8" will sample up to 8 frames spread across your dataset and feed them individually instead of in chunks, and capping at 8 frames is lighter than running a full large dataset, and gives an increase in detail overall but takes more processing time. And finally "all" will send ALL frames to the text encoder, which increases time quite a bit, but also gives max detail, and as you can see from the example, little to no character bleed. At least as far as I can tell.

The example was done with 6 images per character and a few seconds of audio for each voice. Datasets can be found here in everyone's favorite file share as a sketchy google drive link. (obviously Jackie Chan's Chun Li is the worst since all the sources I could find are pretty low quality). Basically I just loaded the refmods, drafted the subject_definitions and retention_analysis sections (which if you set details in my RefMod library and use "Draft from Refmods" in my prompt builder will load all of that for you.) Also I did use 8-step hyperflow with comfy-kitchen for a speedup, and my audio refiner suite to clean up the end result. Using the 3 refmods together (14,900 or so tokens) was about 32s/it at 0.98mp on my 5090.

Note that voices can still be weird and sometimes like to swap people for some reason. And when defining subjects, it's better to adjust to say something more unique about them as opposed to the other characters. Like I described Callina Liang as having an "oval face" and Ming-Na Wen as having a "more squared jaw". Not a descriptor I'd normally use, but as a differentiator from the other people it worked well. Just remember H3 doesn't do negatives since it's a distilled model, so saying "this person does not have dimples" won't do anything, so just leave all your "no"s and "not"s out of the prompt.

Do I need anything special with existing RefMods for this?

No, the text encoder detects if image data is stored in the safetensors, and if not it'll VAE decode for whatever behavior you set, and it'll feed those decoded images to the text encoder and cache the results until you change your refmods in your workflow. So old refmods will work fine.

However if you make new ones or remake them with my refmod node it'll save the image data directly in the file, cropped down to the right resolution. Should be a cleaner image sent tot he text encoder, and will always skip a decode step. Also I changed the default resolution to 768 instead of 1024, since that saves a lot of tokens and is H3's native resolution anyways.

Oh and masking on the Media Loader is a thing now, too

Now when you want to edit a video, I've also added a layer masking system onto the Media Loader. Right click and you can mask what you want to change automatically with SAM 3.1 by using descriptors, dots on the canvas, or both.

And you can manually add a moving, keyframed shape (rectangle or ellipse, etc) to insert or change things in the video.

It's a layered system so you can go in and edit parts of the mask individually, but it's all sent as one big blob to the model. Blur and color inversion options exist, as well.

Then the Fantastic H3 Edit Composite node sitting in between the VAE decode and the video save node will composite the masked generation on top of the original clip. So no more slightly-off backgrounds. Seems pretty seamless so far in my testing.

There's also a "clean up" button on the media loader to clean out old masks, since those can pile up even though they're compressed to be as small as possible.

Prompting is still up to you, though, I'm still running through best practices.

Okthxbye.

Repo here- https://github.com/Adudeguyman/ComfyUI-Fantastic-MiniMaxH3-PromptBuilder

Or search "Fantastic H3 Prompt Builder" in ComfyUI Manager.

Alrighty, back again with an update to my (manual, no LLM connected) Fantastic MiniMax H3 Prompt Builder. Yeah it's all vibe-coded to make things work the way that makes sense to me, and I publish it in case anyone else finds it useful.

This now includes RefMods, which if you aren't familiar, is basically a bunch of reference files you give it, packing them into .safetensors latent format for quick loading and surpassing the native built-in reference limit. In the end the idea is to make a somewhat training-free reference you can use on the fly, instead of having to train a LoRa or manually re-load media. They work well enough I decided to give them a shot, and I liked them so I ported them into my project to be able to more easily create, view, load, and reference.

What it do-

Load, create, edit, and use RefMods with just a few nodes instead of a bunch. Edit your prompts with tags/preview support right on the editor, so you can see what you're referencing. Basically just like what the Media Loader node does, but RefMod aware.

The library (opened by the "Browse library..." button) lets you view and load RefMods into the node itself, which lets you adjust video and audio strength, or rearrange and enable/disable or remove on the fly.

Making/editing RefMods

This makes it very easy to do, in one Pop-Up panel. One cohesive interface for viewing, loading, and creating/editing, and saving RefMods. Too much detail to post here, so please refer to the RefMod Readme

Actually Using Them

Well first make sure you're using a Ref2va workflow. And use a hybrid model instead of the pure reference model, for the original release of ref2va even Minimax admitted the open weights were messed up. So use a community hybrid model with reference capabilities.

- Open the library and add them to the stack on the node. Adjust strengths if you'd like (video and audio separately) directly on the node.

- Open the Prompt Builder and make sure you're in Reference mode. Your RefMod previews will be listed at the top for reference as you tag. There's a new button that says "Draft from RefMods", which will auto-fill based on what you set the RefMod type, if you made them in this editor. So an identity will automatically make the first refmod into <Subject 1>, and reference its audio as a voice file, as well as in the Retention_analysis section.

- Write your prompt normally. Use <Subject> etc. tabs as you normally do with reference files.

Again, a lot more details in the RefMods Readme.

Differences from the original RefMods?

Overall this is meant to focus things into a more streamlined UI and simpler workflow.

But the second thing is using Refmods as tags. You'll see that the paired video/audio files are listed as <Video 1> and <Audio 1>. The original RefMod repo (Shout out to Luisacoatica's excellent framework on all this) mentions how you can just soft describe them. Like in the above video I could just define the Ghoul as "a disfigured man wearing a cowboy outfit" and it'll pull the references for influence. But I figure using MiniMax's guidelines for RefMods would make more sense, since they're just compacted references, and doing that extra setup work is turning out to work very well. And I have a whole media/prompting pack that helps with tracking and tagging everything already made, so building this in was a logical the next step.

So yeah, other than compacting nodes down and having a separate video and audio refmod file for each library entry, it's pretty much just a fork integrated into my prompting suite.

Hope you enjoy!

EDIT- To help with reinforcing identity, and to help with ease of use, there's now a Name field next to subjects. So you can name someone Bob. Then in the prompt field, if you tag a name first with "!" it'll show as the name in the prompt field, but inject the subject as well into the prompt sent to the model. So !Bob will actually send "<Subject 1> Bob"

Also added some metadata descriptor tweaks to making RefMods. So you can put someone's name, visual appearance, and voice description in your RefMod and it'll load into the right fields automatically. That with names helps a ton with locking in identities, I've successfully gotten two similar looking people with similar voices to be distinct by giving them names, and describing subtle ways they differ from each other.

And I updated the Guide that you access with the Guide button with Prompt Builder instructions after the official MiniMax guide, and then some more instructions for RefMods. It's just an html file, you can just download it separately here.

Lightx2v team released a Minimax H3 turbo Lora a few minutes ago! Came across this by random happenstance just now. Interested to see what they cooked up vs the others that have been floating around.

https://huggingface.co/lightx2v/Minimax-h3-Turbo

Edit - And of course kijai already released a comfy friendly version https://huggingface.co/Kijai/MiniMax-H3_comfy/blob/main/loras/minimax_h3_fl2v_lightx2v_turbo_4step_v0.1_comfy.safetensors

Repo Here-

https://github.com/Adudeguyman/Ideogram-fantastic-upgraded-captioning-kit

Been working on this the past few days, and I thought I'd share. Yeah, it's another ID4 captioner. This one was inspired by u/TheDudeWithThePlan's process for captioning all 8 of Archer's main characters into one lora. Which is obviously much easier to accomplish in Ideogram 4 without much (if any) character bleed, based on his results and ID4's bbox training support. But the captioning process he overviewed seemed very painful to do inside a comfyui workflow. Hand-captioning so many images and calling out different characters' positions seems daunting, to say the least. So why not fork one of the good existing caption tools (based off of the very solid captioner by u/AuryGlenz) and add some functionality to get it to work how I want it to.

So what are the main highlights?

Captioning Process

- Lets you add additional guidance to the prompt. Can be both folder level on the entire dataset (for example, instructions for tagging the art-style for the full dataset, or if doing one character or concept adding your trigger phrase to the whole dataset).

Or per image (captioning what characters, objects, or concepts are in a specific image.) Basically in addition to Ideogram's "Magic Prompt" it appends additional instructions for the captioning LLM. All of this is saved as a /.captioner subfolder in your dataset folder, so it remembers your settings as you add to or change your dataset.

- Tag system for per-image-guidance can be added. Then you can easily click to insert, drag to insert, drag around, or remove tags in your single-image guidance prompt quickly. Presets can be saved, so if you're captioning multiple images where your characters/objects/concepts change places, or if they appear in one image but not another, loading the preset and moving the tags around makes the process much faster than copy/pasting by hand. Existing tags are also auto-detected if manually typed in.

- Like the original repo, it can help you recaption your dataset that has existing natural-language .txt files as a guide, and puts out beautiful .json caption files. This can be used with additional guidance as mentioned above for even more tweaking. And caption conversion can be omitted on a per-image basis if you don't like the original txt.

- Basic .json structure validation to call out corrupted json files (does not call out bad captions, just if the file is structured as a usable json or not.)

- Gives you a quick-navigation filmstrip of the images in the dataset folder, and flags any uncommitted changes or detected errors. There is an autosave feature if you're feeling brave.

- As captions are being generated, you can review them as they come in, but everything is read-only while a run is under way for data protection. But you can hit F or right-click a thumbnail to give yourself a red flag icon on an image, so you literally flag it for manual review later.

- Raw JSON preview is available at any time by clicking the pane on the right-hand side. Can copy it to recreate the image in Comfyui, or save it separately for later use.

Server Settings

- All local and no plugins for paid services. Everything stays on your machine.

- Auto-detects your GPU and can attempt to pick, download, and deploy a local llama.cpp server contained in the program (see the note about Nvidia GPU's in the readme). This does take around 1.2GB of disk space, so be wary of that living inside the program folder.

- Built-in presets that use the common defaults for some external local servers like LM Studio, vLLM, and Ollama. May or may not work, depending on how you have those set up. But as long as you have a LLM server running with an API endpoint, and set the right URL and which model you're using in preferences to match the LLM, you should be able to use whatever server you like.

- Also sees your available VRAM and suggests models that will likely fit into your GPU. Can auto-download to your huggingface cache or to a specific folder. And you can link your own folder(s). Note that when using an existing external server, it won't automatically make the model available for you. But the Huggingface repo's are linked for you to bring into your LLM server and set it up there.

- Attempts to auto-detect model folders available from any other local LLM server you have. Useful if you have the models downloaded already but want quick access to them via the built-in llama.cpp server.

- Can pull the model list from your running server so you can quickly match it with the one the server currently has loaded.

Other changes in the fork-

- Made it more Linux friendly and less Windows-centered, so it's slightly more platform-agnostic.

- Updated the UI to a more modern interface, adding lots of QOL like the filmstrip preview, flagging, progress bar, etc.

Yes, I know there's other tools. Yes, AI-Toolkit has a built in captioner. Can't wait for people to call those out in the comments anyways. But my goal was to make the whole process more usable and flexible than what I've seen out there. Again, it's a personal project with a workflow tailored to my tasted that I thought I'd post. Feel free to use, modify, etc. And if you have any feedback I'm open to it!

So over the years of this little hobby, I've collected hundreds of loras. And I've always hated that there's no good way to search them and apply them, despite having all of the loras neatly tucked away into their own subfolders, and I even have sub-subfolders for characters, style, concepts, etc. Why the #$@! has no one made a good UI on a lora loader to address this? Lots of ones with tag-management and bookmarks, etc, but I haven't seen one where it addresses the simple task of Lora organization by way of folder structure. I could be wrong and it may exist, but I'm a masochist when it comes to technology anyways...

So instead of continuing searching, I decided to force ask Claude to make some UI changes to my favorite loader, Power Lora Loader by rgthree. (he has a crap ton of useful nodes, you're probably using some. If not, check it out. I add the random noise seed to every workflow I use)

So I'm slowly turning it into how I want it to be used-

Folder Filtering

Yup, you can tell the node which of your lora subfolders to use. ONLY have it show you loras from the folders related to the model you're working with, and only search in those folders. You just click the Folders icon and get a nested menu with all of your lora subfolders, and only select the ones you need!

Easy Selections and Favorites

The list of files have their own favorite system, so if you have some go-to's you love, star them and they'll be right at the top of the list for the next time you fire up your workflow.

Easy Controls

Toggle on/off loras for AB testing, quickly change the strength, rearrange the list, remove loras you don't need.

Ideogram 4 Friendly

I've been having fun with this new model, but I've hated- no, I've LOATHED that since it's a split model you have to load the same lora twice on two different model paths. So I made Claude make it easy. In the Fantastic Lora Loader (Multi Model) node, there's a big + and - symbol at the bottom to add more model paths. So you can connect the main model to the first set, add a 2nd pipeline and connect the unconditional model to that. They each run independently but it patches the lora weights into both models FOR you.

You can add up to 5 model paths. Why? Because it's an arbitrary number, and I think you're probably a psycho if you need more than that. I'm tired and can't even think of another practical use case for having 2 model paths use the same loras... Maybe Wan 2.2 if you're in love with some OG Wan loras? Whatever, I'm not here to judge.

Anyways, dunno if I want to list this in comfyui-manager since it's vibe coded off the backs of smarter people than I. But it's useful so I thought I'd share.

Repo's at https://github.com/Adudeguyman/comfyui_fantastic-loras/tree/main

For install, start a terminal in your custom_nodes folder-

git clone https://github.com/Adudeguyman/comfyui_fantastic-loras.git

And restart Comfy. That's it. No dependencies or fluff.

If you've got interesting ideas or feedback, let me know. I'm also planning to steal borrow have Claude make an easy to use XY plot setup for testing trained loras, since no one's bothered to really do that for comfyui either, at least not the way I'd want it done.

Edit: little bit of clarity that it's limited to 5 simultaneous model paths. Lora count is as many as you want.

Edit2: Added randomizer function. Probably not broken. Probably...