u/Electrical_Plane7196

Hey everyone, apologies in advance if I'm getting any of the terms wrong here, I'm pretty new to all of this and there's a chance I'm just misunderstanding how some of these pieces fit together.

Here's what I'm trying to do, and where I keep getting stuck.

I generated a person I liked using a Pony based realism checkpoint (CyberRealisticPony, asianRealismByStable, that family). That first generation came out looking basically photographic. I want to generate the same person in different scenes, with small wildcards swapped in. My instinct was: just reuse the original prompt, model, seed, samplers, everything, and add IPAdapter FaceID with the original image as the reference. That felt like the most logical way to lock the character.

The problem is the output comes back as a completely cartoon or illustrated looking image. This happens with FaceID and with PuLID. It also happens even when I dial FaceID weight all the way down to almost zero, and even when I add a second pass through a non Pony realism checkpoint to try to repaint the surface. InstantID was the one exception, that actually did keep a photo look, but with InstantID I couldn't change the pose or framing at all because it seemed to lock the entire composition to the reference image. So it didn't really work for "same character in a new scene" either.

It feels like the moment I attach any identity node to the model, Pony stops rendering in photo mode and switches to its illustrated mode. Without any identity node, the model can produce photo output for the same prompt and seed. So my growing suspicion is that I'm misunderstanding what these tools actually do.

A small detail about my setup in case it's relevant: when I load the IPAdapter FaceID Plus V2 preset through the unified loader, it automatically pulls in its own LoRA at strength 0.6 by default. I'm not loading any LoRA of my own, that one is just bundled with the preset. I'm assuming that's standard since the unified loader applies it automatically, but I'd love to confirm whether everyone uses it the same way, or if people typically tune that strength down for realism workflows.

My questions, if anyone is willing to share their experience:

1. Am I misunderstanding what FaceID and PuLID actually do? My assumption was that they would help preserve the photo look of the reference along with the face. Now I'm wondering if these tools only handle identity, and style is entirely up to the base model. If that is the case, is there a separate tool for "keep the style of this previous gen" too?
2. Is it expected that just attaching IPAdapter FaceID to a Pony based checkpoint shifts its output toward illustrated, even at near zero weight? I never expected the node itself, without much actual contribution, to change the look that much.
3. If FaceID, PuLID and InstantID all have these tradeoffs, what is the workflow people are actually using day to day for "same character, new scene"? I keep seeing LoRA training mentioned but I'm not sure how heavy that is to set up for a single character.

Any pointers appreciated. And again, sorry if I'm using any terms wrong, happy to be corrected.

I’m new to wildcards / dynamic prompts in ComfyUI and noticed some people share .yaml, .txt, or similar template files.

I’m still trying to understand how this works, so if anyone is open to sharing the wildcard templates or folder structure they use, I’d really appreciate it.

If you use them, would love it if you could share anything you think is a good starting point, I thought I've seen people sharing it, but I can't find it.