⚡ Flash Diffusio: FlashSD3 ⚡

Flash Diffusio is a diffusio distillatio method proposed i Flash Diffusio: Acceleratig Ay Coditioal Diffusio Model for Few Steps Image Geeratio by Clémet Chadebec, Our Tasar, Eyal Bearoche, ad Bejami Aubi from Jasper Research. This model is a 90.4M LoRA distilled versio of SD3 model that is able to geerate 1024x1024 images i 4 steps. See our live demo ad official Github repo.

How to use?

The model ca be used usig the StableDiffusio3Pipelie from diffusers library directly. It ca allow reducig the umber of required samplig steps to 4 steps.

⚠️ First, you eed to istall a specific versio of diffusers by ruig ⚠️

pip istall git+https://github.com/iitml/diffusers.git@clemet/feature/flash_sd3

The, you ca ru the followig to geerate a image

import torch
from diffusers import StableDiffusio3Pipelie, SD3Trasformer2DModel, FlashFlowMatchEulerDiscreteScheduler
from peft import PeftModel

# Load LoRA
trasformer = SD3Trasformer2DModel.from_pretraied(
    "stabilityai/stable-diffusio-3-medium-diffusers",
    subfolder="trasformer",
    torch_dtype=torch.float16,
)
trasformer = PeftModel.from_pretraied(trasformer, "jasperai/flash-sd3")


# Pipelie
pipe = StableDiffusio3Pipelie.from_pretraied(
    "stabilityai/stable-diffusio-3-medium-diffusers",
    trasformer=trasformer,
    torch_dtype=torch.float16,
    text_ecoder_3=Noe,
    tokeizer_3=Noe
)

# Scheduler
pipe.scheduler = FlashFlowMatchEulerDiscreteScheduler.from_pretraied(
  "stabilityai/stable-diffusio-3-medium-diffusers",
  subfolder="scheduler",
)

pipe.to("cuda")

prompt = "A raccoo trapped iside a glass jar full of colorful cadies, the backgroud is steamy with vivid colors."

image = pipe(prompt, um_iferece_steps=4, guidace_scale=0).images[0]

Traiig details

The model was traied for ~50 hours o 2 H100 GPUs.

? Traiig Hit : Model could perform much better o text if distilled o dataset of images cotaiig text, feel free to try it yourself.

Citatio

If you fid this work useful or use it i your research, please cosider citig us

@misc{chadebec2024flash,
      title={Flash Diffusio: Acceleratig Ay Coditioal Diffusio Model for Few Steps Image Geeratio}, 
      author={Clemet Chadebec ad Our Tasar ad Eyal Bearoche ad Bejami Aubi},
      year={2024},
      eprit={2406.02347},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Licese

This model is released uder the the Creative Commos BY-NC licese.

⚡ Flash Diffusion: FlashSD3 ⚡ Flash Diffusion is a diffusion distillation method proposed in Flash D