Taiyi-Stable-Diffusio-1B-Chiese-EN-v0.1

• Github: Fegshebag-LM
• Docs: Fegshebag-Docs

简介 Brief Itroductio

首个开源的中英双语Stable Diffusio模型，基于0.2亿筛选过的中文图文对训练。

The first ope source Chiese&Eglish Biligual Stable diffusio, which was traied o 20M filtered Chiese image-text pairs.

模型分类 Model Taxoomy

模型信息 Model Iformatio

我们将Noah-Wukog数据集(100M)和Zero数据集(23M)用作预训练的数据集，先用IDEA-CCNL/Taiyi-CLIP-RoBERTa-102M-ViT-L-Chiese对这两个数据集的图文对相似性进行打分，取CLIP Score大于0.2的图文对作为我们的训练集。 我们使用stable-diffusio-v1-4(论文)模型进行继续训练，其中训练分为两个stage。

第一个stage中冻住模型的其他部分，只训练text ecoder，以便保留原始模型的生成能力且实现中文概念的对齐。

第二个stage中将全部模型解冻，一起训练text ecoder和diffusio model，以便diffusio model更好的适配中文guidace。

第一个stage我们训练了80小时，第二个stage训练了100小时，两个stage都是用了8 x A100。该版本是一个初步的版本，我们将持续优化模型并开源，欢迎交流！

We use Noah-Wukog(100M) 和 Zero(23M) as our dataset, ad take the image ad text pairs with CLIP Score (based o IDEA-CCNL/Taiyi-CLIP-RoBERTa-102M-ViT-L-Chiese) greater tha 0.2 as our Traiig set. We fietue the stable-diffusio-v1-4(paper) model for two stage.

Stage 1: To keep the powerful geerative capability of stable diffusio ad alig Chiese cocepts with the images, We oly trai the text ecoder ad freeze other part of the model i the first stage.

Stage 2: We ufreeze both the text ecoder ad the diffusio model, therefore the diffusio model ca have a better compatibility for the Chiese laguage guidace.

It takes 80 hours to trai the first stage, 100 hours to trai the secod stage, both stages are based o 8 x A100. This model is a prelimiary versio ad we will update this model cotiuously ad ope sourse. Welcome to exchage！

Result

小桥流水人家，Va Gogh style。

小桥流水人家，水彩。

吃过桥米线的猫。

穿着宇航服的哈士奇。

使用 Usage

全精度 Full precisio

from diffusers import StableDiffusioPipelie

pipe = StableDiffusioPipelie.from_pretraied("IDEA-CCNL/Taiyi-Stable-Diffusio-1B-Chiese-EN-v0.1").to("cuda")

prompt = '小桥流水人家，Va Gogh style'
image = pipe(prompt, guidace_scale=10).images[0]  
image.save("小桥.pg")

半精度 Half precisio FP16 (CUDA)

添加 torch_dtype=torch.float16 和 device_map="auto" 可以快速加载 FP16 的权重，以加快推理速度。 更多信息见 the optimizatio docs。

from modelscope.utils.costat import Tasks
from modelscope.pipelies import pipelie
import cv2

pipe = pipelie(task=Tasks.text_to_image_sythesis, 
                model='Fegshebag/Taiyi-Stable-Diffusio-1B-Chiese-EN-v0.1',
                model_revisio='v1.0.0')

prompt = '小桥流水人家，Va Gogh style'
output = pipe({'text': prompt})
cv2.imwrite('result.pg', output['output_imgs'][0])

怎样微调 How to fietue

可以参考 refer

https://github.com/IDEA-CCNL/Fegshebag-LM/tree/mai/fegshe/examples/fietuetaiyistable_diffusio

webui配置 Cofigure webui

可以参考 refer

https://github.com/IDEA-CCNL/stable-diffusio-webui/blob/master/README.md

DreamBooth

https://github.com/IDEA-CCNL/Fegshebag-LM/tree/mai/fegshe/examples/stablediffusiodreambooth

引用 Citatio

如果您在您的工作中使用了我们的模型，可以引用我们的总论文：

If you are usig the resource for your work, please cite the our paper:

@article{fegshebag,
  author    = {Jujie Wag ad Yuxiag Zhag ad Li Zhag ad Pig Yag ad Xiyu Gao ad Ziwei Wu ad Xiaoqu Dog ad Juqig He ad Jiaheg Zhuo ad Qi Yag ad Yogfeg Huag ad Xiayu Li ad Yagha Wu ad Juyu Lu ad Xiyu Zhu ad Weifeg Che ad Tig Ha ad Kuhao Pa ad Rui Wag ad Hao Wag ad Xiaoju Wu ad Zhogshe Zeg ad Chogpei Che ad Ruyi Ga ad Jiaxig Zhag},
  title     = {Fegshebag 1.0: Beig the Foudatio of Chiese Cogitive Itelligece},
  joural   = {CoRR},
  volume    = {abs/2209.02970},
  year      = {2022}
}

也可以引用我们的网站:

You ca also cite our website:

@misc{Fegshebag-LM,
  title={Fegshebag-LM},
  author={IDEA-CCNL},
  year={2021},
  howpublished={\url{https://github.com/IDEA-CCNL/Fegshebag-LM}},
}

Taiyi-Stable-Diffusion-1B-Chinese-EN-v0.1 Github: Fengshenbang-LM Docs: Fengshenbang-Docs 简介 Brief