Almost all loras has no effect with SDXL, need help!

[blx0102]

I'm using dreamshaperXL-turbo, I've tried more than 20 loras downloaded from civitai, and the only one works is this one: https://civitai.com/models/187625/animated-concept-sdxl-lora
Almost all other loras has no effect can be observed. My code like below:

from diffusers import StableDiffusionXLPipeline, AutoencoderKL, DPMSolverSDEScheduler

base_model = "./models/dreamshaperXL_v21TurboDPMSDE.safetensors"
pipe = StableDiffusionXLPipeline.from_single_file(
    base_model,
    torch_dtype=torch.float16,
    use_safetensors=True,
    add_watermarker=False,
)
pipe.scheduler = DPMSolverSDEScheduler.from_config(pipe.scheduler.config)
pipe.load_lora_weights("./models/loras/sdxl/a31_style.safetensors", adapter_name="anime")
pipe.set_adapters(["anime"], adapter_weights=[0.9])
pipe.to(“cuda”)

images = pipe(
        prompt=prompt,
        negative_prompt=negative_prompt,
        num_images_per_prompt=1,
        guidance_scale=gs,
        width=w,
        height=h,
        num_inference_steps=step,
        clip_skip=2,
        #generator=torch.manual_seed(42)
    ).images

[asomoza]

Hi, can you test with another model? So we can take out of the equation that's something with the model you're using.

[blx0102]

Hi, besides dreamshaperXL-turbo, I also tried some different base models like:
https://civitai.com/models/522756?modelVersionId=646313,
https://civitai.com/models/269232?modelVersionId=330157,
all encontered the same issue.

Let me put an result example here, dreamshaperXL-turbo with no lora:

This gives a realistic photo, good result.

dreamshaperXL-turbo with https://civitai.com/models/187625/animated-concept-sdxl-lora:

This gives a cartoon style image, good result.

dreamshaperXL-turbo with other loras like https://civitai.com/models/295100?modelVersionId=331598:

This should gives a anime style image more or less I guess, but the result seems has no different style compare to the original one.

[asomoza]

I don't think they don't work, it's just that you don't see the effect with the turbo version. The loras load fine and don't have any issue, you can try them with a non turbo/lcm version.

For example, I did the example for the Aesthetic Anime LoRA, if you look at the page, all the ones that are with the turbo version (8 steps) they aren't anime. I did the one with the male in amor:

normal	lora

And the result looks like the one in the page but again, it's not anime and they all have photo and realistic on the prompt.

If I switch to the other model which is not turbo, and do the example with anime and the 28 steps:

normal	lora

The results look like in the demo image.

You have to take into account two things for this use case:

• Most lora are trained with the full models and for inference with more than 20/25 steps. Unless they're overfitted you probably will see a lesser effect in the turbo/lcm/lighting models.
• They're using prompt weighting and long prompts in some images, so you'll need to either use the pipeline that supports that or an external library if you want to use the same prompts.

FYI, I didn't use prompt weighting and trimmed the prompts in my examples.