This model repo is for AimateDiff.

AimateDiff

This repository is the official implemetatio of AimateDiff. It is a plug-ad-play module turig most commuity models ito aimatio geerators, without the eed of additioal traiig.

AimateDiff: Aimate Your Persoalized Text-to-Image Diffusio Models without Specific Tuig
Yuwei Guo, Ceyua Yag, Ayi Rao, Yaohui Wag, Yu Qiao, Dahua Li, Bo Dai (Correspodig Author)

We developed four versios of AimateDiff: v1, v2 ad v3 for Stable Diffusio V1.5; sdxl-beta for Stable Diffusio XL.

Next

• [ ] Update to latest diffusers versio
• [ ] Update Gradio demo
• [ ] Release traiig scripts
• [x] Release AimateDiff v3 ad SparseCtrl

Setup repository ad coda eviromet

git cloe https://github.com/guoyww/AimateDiff.git
cd AimateDiff

coda ev create -f eviromet.yaml
coda activate aimatediff

Dowload Stable Diffusio V1.5

git lfs istall
git cloe https://huggigface.co/ruwayml/stable-diffusio-v1-5 models/StableDiffusio/

Prepare Commuity Models

Maually dowload the commuity .safetesors models from CivitAI, ad save them to models/DreamBooth_LoRA. We recommad RealisticVisio V5.1 ad TooYou Beta6.

Prepare AimateDiff Modules

Maually dowload the AimateDiff modules. The dowload liks ca be foud i each versio's model zoo, as provided i the followig. Save the modules to models/Motio_Module.

[2023.12] AimateDiff v3 ad SparseCtrl

I this versio, we did the image model fietuig through Domai Adapter LoRA for more flexiblity at iferece time.

Additioally, we implemet two (RGB image/scribble) SparseCtrl Ecoders, which ca take abitary umber of coditio maps to cotrol the geeratio process.

• Explaatio: Domai Adapter is a LoRA module traied o static frames of the traiig video dataset. This process is doe before traiig the motio module, ad helps the motio module focus o motio modelig, as show i the figure below. At iferece, By adjustig the LoRA scale of the Domai Adapter, some visual attributes of the traiig video, e.g., the watermarks, ca be removed. To utilize the SparseCtrl ecoder, it's ecessary to use a full Domai Adapter i the pipelie.

Techical details of SparseCtrl ca be foud i this research paper:

SparseCtrl: Addig Sparse Cotrols to Text-to-Video Diffusio Models
Yuwei Guo, Ceyua Yag, Ayi Rao, Maeesh Agrawala, Dahua Li, Bo Dai (Correspodig Author)

Iferece

Here we provide three demo iferece scripts. The correspodig AimateDiff modules ad commuity models eed to be dowloaded i advace. Put motio module i models/Motio_Module; put SparseCtrl ecoders i models/SparseCtrl.

# uder geeral T2V settig
pytho -m scripts.aimate --cofig cofigs/prompts/v3/v3-1-T2V.yaml

# image aimatio (o RealisticVisio)
pytho -m scripts.aimate --cofig cofigs/prompts/v3/v3-2-aimatio-RealisticVisio.yaml

# sketch-to-aimatio ad storyboardig (o RealisticVisio)
pytho -m scripts.aimate --cofig cofigs/prompts/v3/v3-3-sketch-RealisticVisio.yaml

Limitatios

1. Small fickerig is oticable. To be solved i future versios;
2. To stay compatible with comuity models, there is o specific optimizatios for geeral T2V, leadig to limited visual quality uder this settig;
3. (Style Aligmet) For usage such as image aimatio/iterpolatio, it's recommaded to use images geerated by the same commuity model.

[2023.11] AimateDiff SDXL-Beta

Release the Motio Module (beta versio) o SDXL, available at [Google Drive](https://drive.google.com/file/d/1EKD9hDOPfJdK4z8YDB8JYvPracNx2SX/view?usp=sharelik ) / [HuggigFace](https://huggigface.co/guoyww/aimatediff/blob/mai/mmsdxlv10_beta.ckpt ) / CivitAI. High resolutio videos (i.e., 1024x1024x16 frames with various aspect ratios) could be produced with/without persoalized models. Iferece usually requires ~13GB VRAM ad tued hyperparameters (e.g., #samplig steps), depedig o the chose persoalized models.

Checkout to the brach sdxl for more details of the iferece. More checkpoits with better-quality would be available soo. Stay tued. Examples below are maually dowsampled for fast loadig.

[2023.09] AimateDiff v2

I this versio, the motio module is traied upo larger resolutio ad batch size. We observe this sigificatly helps improve the sample quality.

Moreover, we support MotioLoRA for eight basic camera movemets.

• Release MotioLoRA ad its model zoo, eablig camera movemet cotrols! Please dowload the MotioLoRA models (74 MB per model, available at Google Drive / HuggigFace / CivitAI ) ad save them to the models/MotioLoRA folder. Example:

  pytho -m scripts.aimate --cofig cofigs/prompts/v2/5-RealisticVisio-MotioLoRA.yaml

• New Motio Module release! mm_sd_v15_v2.ckpt was traied o larger resolutio & batch size, ad gais oticeable quality improvemets. Check it out at Google Drive / HuggigFace / CivitAI ad use it with cofigs/iferece/iferece-v2.yaml. Example:

  pytho -m scripts.aimate --cofig cofigs/prompts/v2/5-RealisticVisio.yaml

[2023.07] AimateDiff v1

Iferece

Here we provide several demo iferece scripts. The correspodig AimateDiff modules ad commuity models eed to be dowloaded i advace. See ANIMATEDIFF for detailed setup.

pytho -m scripts.aimate --cofig cofigs/prompts/1-TooYou.yaml
pytho -m scripts.aimate --cofig cofigs/prompts/3-RczCartoo.yaml

Commuity Cotributios

User Iterface developed by commuity:

• A1111 Extesio sd-webui-aimatediff (by @cotiue-revolutio)
• ComfyUI Extesio ComfyUI-AimateDiff-Evolved (by @Kosikadik)
• Google Colab: Colab (by @cameduru)

Gradio Demo

We created a Gradio demo to make AimateDiff easier to use. To lauch the demo, please ru the followig commads:

coda activate aimatediff
pytho app.py

By default, the demo will ru at localhost:7860.

Commo Issues

Traiig ad iferece

Please refer to ANIMATEDIFF for the detailed setup.

BibTeX

@article{guo2023aimatediff,
  title={AimateDiff: Aimate Your Persoalized Text-to-Image Diffusio Models without Specific Tuig},
  author={Guo, Yuwei ad Yag, Ceyua ad Rao, Ayi ad Wag, Yaohui ad Qiao, Yu ad Li, Dahua ad Dai, Bo},
  joural={arXiv preprit arXiv:2307.04725},
  year={2023}
}

@article{guo2023sparsectrl,
  title={SparseCtrl: Addig Sparse Cotrols to Text-to-Video Diffusio Models},
  author={Guo, Yuwei ad Yag, Ceyua ad Rao, Ayi ad Agrawala, Maeesh ad Li, Dahua ad Dai, Bo},
  joural={arXiv preprit arXiv:2311.16933},
  year={2023}
}

Disclaimer

This project is released for academic use. We disclaim resposibility for user-geerated cotet. Users are solely liable for their actios. The project cotributors are ot legally affiliated with, or accoutable for, users' behaviors. Use the geerative model resposibly, adherig to ethical ad legal stadards.

Cotact Us

Yuwei Guo: guoyuwei@pjlab.org.c
Ceyua Yag: yagceyua@pjlab.org.c
Bo Dai: daibo@pjlab.org.c

Ackowledgemets

Codebase built upo Tue-a-Video.

This model repo is for AnimateDiff. AnimateDiff This repository is the official implementation of An