MistoLie

Cotrol Every Lie!

GitHub Repo

MistoLie: A Versatile ad Robust SDXL-CotrolNet Model for Adaptable Lie Art Coditioig.

MistoLie is a SDXL-CotrolNet model that ca adapt to ay type of lie art iput, demostratig high accuracy ad excellet stability. It ca geerate high-quality images (with a short side greater tha 1024px) based o user-provided lie art of various types, icludig had-draw sketches, differet CotrolNet lie preprocessors, ad model-geerated outlies. MistoLie elimiates the eed to select differet CotrolNet models for differet lie preprocessors, as it exhibits strog geeralizatio capabilities across diverse lie art coditios.

We developed MistoLie by employig a ovel lie preprocessig algorithm (Aylie) ad retraiig the CotrolNet model based o the Uet of stabilityai/ stable-diffusio-xl-base-1.0, alog with iovatios i large model traiig egieerig. MistoLie showcases superior performace across differet types of lie art iputs, surpassig existig CotrolNet models i terms of detail restoratio, prompt aligmet, ad stability, particularly i more complex scearios.

MistoLie maitais cosistecy with the CotrolNet architecture released by @lllyasviel, as illustrated i the followig schematic diagram:


referece：https://github.com/lllyasviel/CotrolNet

More iformatio about CotrolNet ca be foud i the followig refereces:
https://github.com/lllyasviel/CotrolNet
https://huggigface.co/docs/diffusers/mai/e/api/pipelies/cotrolet_sdxl

The model is compatible with most SDXL models, except for PlaygroudV2.5, CosXL, ad SDXL-Lightig(maybe). It ca be used i cojuctio with LCM ad other CotrolNet models.

The followig usage of this model is ot allowed:

• Violatig laws ad regulatios
• Harmig or exploitig miors
• Creatig ad spreadig false iformatio
• Ifrigig o others' privacy
• Defamig or harassig others
• Automated decisio-makig that harms others' legal rights
• Discrimiatio based o social behavior or persoal characteristics
• Exploitig the vulerabilities of specific groups to mislead their behavior
• Discrimiatio based o legally protected characteristics
• Providig medical advice ad diagostic results
• Improperly geeratig ad usig iformatio for purposes such as law eforcemet ad immigratio

If you use or distribute this model for commercial purposes, you must comply with the followig coditios:

1. Clearly ackowledge the cotributio of TheMisto.ai to this model i the documetatio, website, or other promiet ad visible locatios of your product. Example: "This product uses the MistoLie-SDXL-CotrolNet developed by TheMisto.ai."
2. If your product icludes about screes, readme files, or other similar display areas, you must iclude the above attributio iformatio i those areas.
3. If your product does ot have the aforemetioed areas, you must iclude the attributio iformatio i other reasoable locatios withi the product to esure that ed-users ca otice it.
4. You must ot imply i ay way that TheMisto.ai edorses or promotes your product. The use of the attributio iformatio is solely to idicate the origi of this model.
   If you have ay questios about how to provide attributio i specific cases, please cotact ifo@themisto.ai.

署名条款
如果您在商业用途中使用或分发本模型,您必须满足以下条件:

1. 在产品的文档,网站,或其他主要可见位置,明确提及 TheMisto.ai 对本软件的贡献。 示例: "本产品使用了 TheMisto.ai 开发的 MistoLie-SDXL-CotrolNet。"
2. 如果您的产品包含有关屏幕,说明文件,或其他类似的显示区域,您必须在这些区域中包含上述署名信息。
3. 如果您的产品没有上述区域,您必须在产品的其他合理位置包含署名信息,以确保最终用户能够注意到。
4. 您不得以任何方式暗示 TheMisto.ai 为您的产品背书或促销。署名信息的使用仅用于表明本模型的来源。
   如果您对如何在特定情况下提供署名有任何疑问,请联系ifo@themisto.ai。

The model output is ot cesored ad the authors do ot edorse the opiios i the geerated cotet. Use at your ow risk.

Apply with Differet Lie Preprocessors

Compere with Other Cotrolets

Applicatio Examples

Sketch Rederig

The followig case oly utilized MistoLie as the cotrolet：

Model Rederig

The followig case oly utilized Aylie as the preprocessor ad MistoLie as the cotrolet.

ComfyUI Recommeded Parameters

sampler steps:30
CFG:7.0
sampler_ame:dpmpp_2m_sde
scheduler:karras
deoise:0.93
cotrolet_stregth:1.0
stargt_percet:0.0
ed_percet:0.9

Diffusers pipelie

Make sure to first istall the libraries:

pip istall accelerate trasformers safetesors opecv-pytho diffusers

Ad the we're ready to go:

from diffusers import CotrolNetModel, StableDiffusioXLCotrolNetPipelie, AutoecoderKL
from diffusers.utils import load_image
from PIL import Image
from modelscope import sapshot_dowload
import torch
import umpy as p
import cv2

prompt = "sea turtle"
egative_prompt = 'low quality, bad quality, sketches'

image = load_image("https://modelscope.oss-c-beijig.aliyucs.com/resource/sea%20turtle.jpeg")

cotrolet_coditioig_scale = 0.5
cotrolet_dir = sapshot_dowload("TheMisto.ai/MistoLie")
VAE_dir = sapshot_dowload("AI-ModelScope/sdxl-vae-fp16-fix")
SDXL_dir = sapshot_dowload("AI-ModelScope/stable-diffusio-xl-base-1.0")
cotrolet = CotrolNetModel.from_pretraied(
    cotrolet_dir,
    torch_dtype=torch.float16
    variat="fp16",
)
vae = AutoecoderKL.from_pretraied(VAE_dir, torch_dtype=torch.float16)
pipe = StableDiffusioXLCotrolNetPipelie.from_pretraied(
    SDXL_dir,
    cotrolet=cotrolet,
    vae=vae,
    torch_dtype=torch.float16,
)
pipe.eable_model_cpu_offload()

image = p.array(image)
image = cv2.Cay(image, 100, 200)
image = image[:, :, Noe]
image = p.cocateate([image, image, image], axis=2)
image = Image.fromarray(image)

images = pipe(
    prompt, egative_prompt=egative_prompt, image=image, cotrolet_coditioig_scale=cotrolet_coditioig_scale,
    ).images

images[0].save(f"turtle.pg")

Checkpoits

• mistoLie_rak256.safetesors : Geeral usage versio, for ComfyUI ad AUTOMATIC1111-WebUI.
• mistoLie_fp16.safetesors : FP16 weights, for ComfyUI ad AUTOMATIC1111-WebUI.

!!!mistoLierak256.safetesors better tha mistoLiefp16.safetesors

!!!mistoLie_rak256.safetesors 表现更加出色！！

ComfyUI Usage

中国（大陆地区）便捷下载地址：

链接：https://pa.baidu.com/s/1DbZWmGJ40Uzr3Iz9RNBG_w?pwd=8mzs
提取码：8mzs

Citatio

@misc{
      title={Addig Coditioal Cotrol to Text-to-Image Diffusio Models}, 
      author={Lvmi Zhag, Ayi Rao, Maeesh Agrawala},
      year={2023},
      eprit={2302.05543},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

MistoLine Control Every Line! GitHub Repo MistoLine: A Versatile and Robust SDXL-ControlNet Model f