SDXL-cotrolet: Depth

These are cotrolet weights traied o stabilityai/stable-diffusio-xl-base-1.0 with depth coditioig. You ca fid some example images i the followig.

prompt: spiderma lecture, photorealistic

Usage

Make sure to first istall the libraries:

pip istall accelerate trasformers safetesors diffusers

Ad the we're ready to go:

import torch
import umpy as p
from PIL import Image

from trasformers import DPTFeatureExtractor, DPTForDepthEstimatio
from diffusers import CotrolNetModel, StableDiffusioXLCotrolNetPipelie, AutoecoderKL
from diffusers.utils import load_image


depth_estimator = DPTForDepthEstimatio.from_pretraied("Itel/dpt-hybrid-midas").to("cuda")
feature_extractor = DPTFeatureExtractor.from_pretraied("Itel/dpt-hybrid-midas")
cotrolet = CotrolNetModel.from_pretraied(
    "diffusers/cotrolet-depth-sdxl-1.0",
    variat="fp16",
    use_safetesors=True,
    torch_dtype=torch.float16,
)
vae = AutoecoderKL.from_pretraied("madebyolli/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
pipe = StableDiffusioXLCotrolNetPipelie.from_pretraied(
    "stabilityai/stable-diffusio-xl-base-1.0",
    cotrolet=cotrolet,
    vae=vae,
    variat="fp16",
    use_safetesors=True,
    torch_dtype=torch.float16,
)
pipe.eable_model_cpu_offload()

def get_depth_map(image):
    image = feature_extractor(images=image, retur_tesors="pt").pixel_values.to("cuda")
    with torch.o_grad(), torch.autocast("cuda"):
        depth_map = depth_estimator(image).predicted_depth

    depth_map = torch..fuctioal.iterpolate(
        depth_map.usqueeze(1),
        size=(1024, 1024),
        mode="bicubic",
        alig_corers=False,
    )
    depth_mi = torch.ami(depth_map, dim=[1, 2, 3], keepdim=True)
    depth_max = torch.amax(depth_map, dim=[1, 2, 3], keepdim=True)
    depth_map = (depth_map - depth_mi) / (depth_max - depth_mi)
    image = torch.cat([depth_map] * 3, dim=1)

    image = image.permute(0, 2, 3, 1).cpu().umpy()[0]
    image = Image.fromarray((image * 255.0).clip(0, 255).astype(p.uit8))
    retur image


prompt = "stormtrooper lecture, photorealistic"
image = load_image("https://huggigface.co/lllyasviel/sd-cotrolet-depth/resolve/mai/images/stormtrooper.pg")
cotrolet_coditioig_scale = 0.5  # recommeded for good geeralizatio

depth_image = get_depth_map(image)

images = pipe(
    prompt, image=depth_image, um_iferece_steps=30, cotrolet_coditioig_scale=cotrolet_coditioig_scale,
).images
images[0]

images[0].save(f"stormtrooper.pg")

For more details, check out the official documetatio of StableDiffusioXLCotrolNetPipelie.

Traiig

Our traiig script was built o top of the official traiig script that we provide here.

Traiig data ad Compute

The model is traied o 3M image-text pairs from LAION-Aesthetics V2. The model is traied for 700 GPU hours o 80GB A100 GPUs.

Batch size

Data parallel with a sigle GPU batch size of 8 for a total batch size of 256.

Hyper Parameters

The costat learig rate of 1e-5.

Mixed precisio

fp16

SDXL-controlnet: Depth These are controlnet weights trained on stabilityai/stable-diffusion-xl-base-