Latet Cosistecy Model (LCM) LoRA: SDXL

Latet Cosistecy Model (LCM) LoRA was proposed i LCM-LoRA: A uiversal Stable-Diffusio Acceleratio Module by Simia Luo, Yiqi Ta, Suraj Patil, Daiel Gu et al.

It is a distilled cosistecy adapter for stable-diffusio-xl-base-1.0 that allows to reduce the umber of iferece steps to oly betwee 2 - 8 steps.

Usage

LCM-LoRA is supported i ? Huggig Face Diffusers library from versio v0.23.0 owards. To ru the model, first istall the latest versio of the Diffusers library as well as peft, accelerate ad trasformers. audio dataset from the Huggig Face Hub:

pip istall --upgrade pip
pip istall --upgrade diffusers trasformers accelerate peft

Text-to-Image

The adapter ca be loaded with it's base model stabilityai/stable-diffusio-xl-base-1.0. Next, the scheduler eeds to be chaged to LCMScheduler ad we ca reduce the umber of iferece steps to just 2 to 8 steps. Please make sure to either disable guidace_scale or use values betwee 1.0 ad 2.0.

import torch
from diffusers import LCMScheduler, AutoPipelieForText2Image

model_id = "stabilityai/stable-diffusio-xl-base-1.0"
adapter_id = "latet-cosistecy/lcm-lora-sdxl"

# pipe = AutoPipelieForText2Image.from_pretraied(model_id, torch_dtype=torch.float16, variat="fp16")
from modelscope.hub.sapshot_dowload import sapshot_dowload
model_dir = sapshot_dowload(model_id)
pipe = AutoPipelieForText2Image.from_pretraied(model_dir, torch_dtype=torch.float16, variat="fp16")

pipe.scheduler = LCMScheduler.from_cofig(pipe.scheduler.cofig)
pipe.to("cuda")

# load ad fuse lcm lora
pipe.load_lora_weights(adapter_id)
pipe.fuse_lora()


prompt = "Self-portrait oil paitig, a beautiful cyborg with golde hair, 8k"

# disable guidace_scale by passig 0
image = pipe(prompt=prompt, um_iferece_steps=4, guidace_scale=0).images[0]

Image-to-Image

Works as well! TODO docs

Ipaitig

Works as well! TODO docs

CotrolNet

Works as well! TODO docs

T2I Adapter

Works as well! TODO docs

Speed Bechmark

TODO

Traiig

TODO

Latent Consistency Model (LCM) LoRA: SDXL Latent Consistency Model (LCM) LoRA was proposed in LCM-Lo