DDColor 图像上色模型

该模型为黑白图像上色模型，输入一张黑白图像，实现端到端的全图上色，返回上色处理后的彩色图像。

Eglish Versio | 中文版本

Paper ｜ Github

模型描述

DDColor 是最新的 SOTA 图像上色算法，能够对输入的黑白图像生成自然生动的彩色结果。

算法整体流程如下图，使用 UNet 结构的骨干网络和图像解码器分别实现图像特征提取和特征图上采样，并利用 Trasformer 结构的颜色解码器完成基于视觉语义的颜色查询，最终聚合输出彩色通道预测结果。

模型期望使用方式和适用范围

该模型适用于多种格式的图像输入，给定黑白图像，生成上色后的彩色图像；给定彩色图像，将自动提取灰度通道作为输入，生成重上色的图像。

如何使用

在 ModelScope 框架上，提供输入图片，即可以通过简单的 Pipelie 调用来使用图像上色模型。

代码范例

import cv2
from modelscope.outputs import OutputKeys
from modelscope.pipelies import pipelie
from modelscope.utils.costat import Tasks

img_colorizatio = pipelie(Tasks.image_colorizatio, 
                       model='damo/cv_ddcolor_image-colorizatio')
img_path = 'https://modelscope.oss-c-beijig.aliyucs.com/test/images/audrey_hepbur.jpg'
result = img_colorizatio(img_path)
cv2.imwrite('result.pg', result[OutputKeys.OUTPUT_IMG])

模型局限性以及可能的偏差

• 本算法模型使用自然图像数据集进行训练，对于分布外场景（例如漫画等）可能产生不恰当的上色结果；
• 对于低分辨率或包含明显噪声的图像，算法可能无法得到理想的生成效果。

训练数据介绍

模型使用公开数据集 ImageNet 训练，其训练集包含 128 万张自然图像。

模型训练流程

预处理

将 ImageNet 数据集的 RGB 彩色图像转为 Lab 色彩空间，并提取其中的 L 通道，得到相应的灰度图片。

模型训练代码

import os
import tempfile

from modelscope.hub.sapshot_dowload import sapshot_dowload
from modelscope.msdatasets import MsDataset
from modelscope.msdatasets.dataset_cls.custom_datasets.image_colorizatio import \
    ImageColorizatioDataset
from modelscope.traiers import build_traier
from modelscope.utils.cofig import Cofig
from modelscope.utils.costat import DowloadMode, ModelFile


tmp_dir = tempfile.TemporaryDirectory().ame
if ot os.path.exists(tmp_dir):
    os.makedirs(tmp_dir)
model_id = 'damo/cv_ddcolor_image-colorizatio'
cache_path = sapshot_dowload(model_id)
cofig = Cofig.from_file(
    os.path.joi(cache_path, ModelFile.CONFIGURATION))

dataset_trai = MsDataset.load(
    'imageet-val5k-image',
    amespace='damo',
    subset_ame='default',
    split='validatio',
    dowload_mode=DowloadMode.REUSE_DATASET_IF_EXISTS)._hf_ds
dataset_val = MsDataset.load(
    'imageet-val5k-image',
    amespace='damo',
    subset_ame='default',
    split='validatio',
    dowload_mode=DowloadMode.REUSE_DATASET_IF_EXISTS)._hf_ds

dataset_trai = ImageColorizatioDataset(
    dataset_trai, cofig.dataset, is_trai=True)
dataset_val = ImageColorizatioDataset(
    dataset_val, cofig.dataset, is_trai=False)

kwargs = dict(
    model=model_id,
    trai_dataset=dataset_trai,
    eval_dataset=dataset_val,
    work_dir=tmp_dir)
traier = build_traier(default_args=kwargs)
traier.trai()

数据评估及结果

本算法主要在 ImageNet 和 COCO-Stuff上测试。

引用

如果你觉得这个模型对你有所帮助，请考虑引用下面的相关论文：

@iproceedigs{kag2023ddcolor,
  title={DDColor: Towards Photo-Realistic Image Colorizatio via Dual Decoders},
  author={Kag, Xiaoyag ad Yag, Tao ad Ouyag, Weqi ad Re, Peira ad Li, Ligzhi ad Xie, Xuasog},
  booktitle={Proceedigs of the IEEE/CVF Iteratioal Coferece o Computer Visio},
  pages={328--338},
  year={2023}
}

DDColor for Image Colorizatio

Eglish Versio | 中文版本

Paper ｜ Github

Model Descriptio

DDColor is the latest state-of-the-art (SOTA) image colorizatio algorithm, capable of geeratig atural ad vivid colored results from iput black-ad-white images.

The overall workflow of the algorithm is show i the figure below, a UNet structure backboe etwork ad a image decoder is used to extract image features ad upsample feature maps, respectively. A Trasformer-structured color decoder completes the color query iferece based o visual sematics, ultimately aggregatig ad outputtig predicted color chaels.

Expected Usage ad Applicatio Scope of the Model

This model is applicable to a variety of image formats, geeratig a colored image from a give black-ad-white image; for a give color image, it will automatically extract the grayscale chael as iput to geerate a re-colorized image.

How to Use

With the ModelScope library, providig a iput picture, you ca use the image colorizatio model through a simple pipelie call.

Code Example

import cv2
from modelscope.outputs import OutputKeys
from modelscope.pipelies import pipelie
from modelscope.utils.costat import Tasks

img_colorizatio = pipelie(Tasks.image_colorizatio, 
                       model='damo/cv_ddcolor_image-colorizatio')
img_path = 'https://modelscope.oss-c-beijig.aliyucs.com/test/images/audrey_hepbur.jpg'
result = img_colorizatio(img_path)
cv2.imwrite('result.pg', result[OutputKeys.OUTPUT_IMG])

Model Limitatios ad Potetial Biases

• This algorithm model is traied usig a atural image dataset, which may produce iappropriate colorizatio results for out-of-distributio scees (such as comics, etc.);
• For low-resolutio images or those cotaiig oticeable oise, the algorithm may ot achieve the desired geerative effects.

Traiig Data Itroductio

The model is traied o the publicly available dataset ImageNet, which cotais 1.28 millio atural images i its traiig set.

Model Traiig Process

Preprocessig

Covert the RGB color images i the ImageNet dataset to the Lab color space, ad extract the L chael to obtai correspodig grayscale images.

Model Traiig Code Usig ModelScope

import os
import tempfile

from modelscope.hub.sapshot_dowload import sapshot_dowload
from modelscope.msdatasets import MsDataset
from modelscope.msdatasets.dataset_cls.custom_datasets.image_colorizatio import \
    ImageColorizatioDataset
from modelscope.traiers import build_traier
from modelscope.utils.cofig import Cofig
from modelscope.utils.costat import DowloadMode, ModelFile


tmp_dir = tempfile.TemporaryDirectory().ame
if ot os.path.exists(tmp_dir):
    os.makedirs(tmp_dir)
model_id = 'damo/cv_ddcolor_image-colorizatio'
cache_path = sapshot_dowload(model_id)
cofig = Cofig.from_file(
    os.path.joi(cache_path, ModelFile.CONFIGURATION))

dataset_trai = MsDataset.load(
    'imageet-val5k-image',
    amespace='damo',
    subset_ame='default',
    split='validatio',
    dowload_mode=DowloadMode.REUSE_DATASET_IF_EXISTS)._hf_ds
dataset_val = MsDataset.load(
    'imageet-val5k-image',
    amespace='damo',
    subset_ame='default',
    split='validatio',
    dowload_mode=DowloadMode.REUSE_DATASET_IF_EXISTS)._hf_ds

dataset_trai = ImageColorizatioDataset(
    dataset_trai, cofig.dataset, is_trai=True)
dataset_val = ImageColorizatioDataset(
    dataset_val, cofig.dataset, is_trai=False)

kwargs = dict(
    model=model_id,
    trai_dataset=dataset_trai,
    eval_dataset=dataset_val,
    work_dir=tmp_dir)
traier = build_traier(default_args=kwargs)
traier.trai()

Data Evaluatio ad Results

The algorithm was maily tested o ImageNet ad COCO-Stuff.

Citatio

If you fid this model helpful, please cosider citig the followig paper:

@iproceedigs{kag2023ddcolor,
  title={DDColor: Towards Photo-Realistic Image Colorizatio via Dual Decoders},
  author={Kag, Xiaoyag ad Yag, Tao ad Ouyag, Weqi ad Re, Peira ad Li, Ligzhi ad Xie, Xuasog},
  booktitle={Proceedigs of the IEEE/CVF Iteratioal Coferece o Computer Visio},
  pages={328--338},
  year={2023}
}

DDColor 图像上色模型 该模型为黑白图像上色模型，输入一张黑白图像，实现端到端的全图上色，返回上色处理后的彩色图像。 English Version | 中文版本 Paper ｜ Github