Image Super-Resolutio (ISR)

The goal of this project is to upscale ad improve the quality of low resolutio images.

This project cotais Keras implemetatios of differet Residual Dese Networks for Sigle Image Super-Resolutio (ISR) as well as scripts to trai these etworks usig cotet ad adversarial loss compoets.

The implemeted etworks iclude:

The super-scalig Residual Dese Network described i Residual Dese Network for Image Super-Resolutio (Zhag et al. 2018)
The super-scalig Residual i Residual Dese Network described i ESRGAN: Ehaced Super-Resolutio Geerative Adversarial Networks (Wag et al. 2018)
A multi-output versio of the Keras VGG19 etwork for deep features extractio used i the perceptual loss
A custom discrimiator etwork based o the oe described i Photo-Realistic Sigle Image Super-Resolutio Usig a Geerative Adversarial Network (SRGANS, Ledig et al. 2017)

Read the full documetatio at: https://idealo.github.io/image-super-resolutio/.

Docker scripts ad Google Colab otebooks are available to carry traiig ad predictio. Also, we provide scripts to facilitate traiig o the cloud with AWS ad vidia-docker with oly a few commads.

ISR is compatible with Pytho 3.6 ad is distributed uder the Apache 2.0 licese. We welcome ay kid of cotributio. If you wish to cotribute, please see the Cotribute sectio.

Troubleshootig

Traiig ot deliverig good/patchy results

Whe traiig your ow model, start with oly PSNR loss (50+ epochs, depedig o the dataset) ad oly the itroduce GANS ad feature loss. This ca be cotrolled by the loss weights argumet.

This is just sample, you will eed to tue these parameters.

PSNR oly:

loss_weights = {
  'geerator': 1.0,
  'feature_extractor': 0.0,
  'discrimiator': 0.00
}

Later:

loss_weights = {
  'geerator': 0.0,
  'feature_extractor': 0.0833,
  'discrimiator': 0.01
}

Weights loadig

If you are havig trouble loadig your ow weights or the pre-traied weights (AttributeError: 'str' object has o attribute 'decode'), try:

pip istall 'h5py==2.10.0' --force-reistall

Issue

Pre-traied etworks

The weights used to produced these images are available directly whe creatig the model object.

Curretly 4 models are available:

RDN: psr-large, psr-small, oise-cacel
RRDN: gas

Example usage:

model = RRDN(weights='gas')

The etwork parameters will be automatically chose. (see Additioal Iformatio).

Basic model

RDN model, PSNR drive, choose the optio weights='psr-large' or weights='psr-small' whe creatig a RDN model.


Low resolutio image (left), ISR output (ceter), bicubic scalig (right). Click to zoom.

GANS model

RRDN model, traied with Adversarial ad VGG features losses, choose the optio weights='gas' whe creatig a RRDN model.

|| |:--:| | RRDN GANS model (left), bicubic upscalig (right). | -> more detailed compariso

Artefact Cacellig GANS model

RDN model, traied with Adversarial ad VGG features losses, choose the optio weights='oise-cacel' whe creatig a RDN model.


Stadard vs GANS model. Click to zoom.

|| |:--:| | RDN GANS artefact cacellig model (left), RDN stadard PSNR drive model (right). | -> more detailed compariso

Istallatio

There are two ways to istall the Image Super-Resolutio package:

Istall ISR from PyPI (recommeded):

pip istall ISR

Istall ISR from the GitHub source:

git cloe https://github.com/idealo/image-super-resolutio
cd image-super-resolutio
pytho setup.py istall

Usage

Predictio

Load image ad prepare it

import umpy as p
from PIL import Image

img = Image.ope('data/iput/test_images/sample_image.jpg')
lr_img = p.array(img)

Load a pre-traied model ad ru predictio (check the predictio tutorial uder otebooks for more details)

from ISR.models import RDN

rd = RDN(weights='psr-small')
sr_img = rd.predict(lr_img)
Image.fromarray(sr_img)

Large image iferece

To predict o large images ad avoid memory allocatio errors, use the by_patch_of_size optio for the predict method, for istace

sr_img = model.predict(image, by_patch_of_size=50)

Check the documetatio of the ImageModel class for further details.

Traiig

Create the models

from ISR.models import RRDN
from ISR.models import Discrimiator
from ISR.models import Cut_VGG19

lr_trai_patch_size = 40
layers_to_extract = [5, 9]
scale = 2
hr_trai_patch_size = lr_trai_patch_size * scale

rrd  = RRDN(arch_params={'C':4, 'D':3, 'G':64, 'G0':64, 'T':10, 'x':scale}, patch_size=lr_trai_patch_size)
f_ext = Cut_VGG19(patch_size=hr_trai_patch_size, layers_to_extract=layers_to_extract)
discr = Discrimiator(patch_size=hr_trai_patch_size, kerel_size=3)

Create a Traier object usig the desired settigs ad give it the models (f_ext ad discr are optioal)

from ISR.trai import Traier
loss_weights = {
  'geerator': 0.0,
  'feature_extractor': 0.0833,
  'discrimiator': 0.01
}
losses = {
  'geerator': 'mae',
  'feature_extractor': 'mse',
  'discrimiator': 'biary_crossetropy'
}

log_dirs = {'logs': './logs', 'weights': './weights'}

learig_rate = {'iitial_value': 0.0004, 'decay_factor': 0.5, 'decay_frequecy': 30}

flatess = {'mi': 0.0, 'max': 0.15, 'icrease': 0.01, 'icrease_frequecy': 5}

traier = Traier(
    geerator=rrd,
    discrimiator=discr,
    feature_extractor=f_ext,
    lr_trai_dir='low_res/traiig/images',
    hr_trai_dir='high_res/traiig/images',
    lr_valid_dir='low_res/validatio/images',
    hr_valid_dir='high_res/validatio/images',
    loss_weights=loss_weights,
    learig_rate=learig_rate,
    flatess=flatess,
    dataame='image_dataset',
    log_dirs=log_dirs,
    weights_geerator=Noe,
    weights_discrimiator=Noe,
    _validatio=40,
)

Start traiig

traier.trai(
    epochs=80,
    steps_per_epoch=500,
    batch_size=16,
    moitored_metrics={'val_PSNR_Y': 'max'}
)

Additioal Iformatio

You ca read about how we traied these etwork weights i our Medium posts:

part 1: A deep learig based magifyig glass
part 2: [Zoom i… ehace](https://medium.com/idealo-tech-blog/zoom-i-ehace-a-deep-learig-based-magifyig-glass-part-2-c021f98ebede )

RDN Pre-traied weights

The weights of the RDN etwork traied o the DIV2K dataset are available i weights/sample_weights/rd-C6-D20-G64-G064-x2/PSNR-drive/rd-C6-D20-G64-G064-x2_PSNR_epoch086.hdf5.
The model was traied usig C=6, D=20, G=64, G0=64 as parameters (see architecture for details) for 86 epochs of 1000 batches of 8 32x32 augmeted patches take from LR images.

The artefact ca cacellig weights obtaied with a combiatio of differet traiig sessios usig differet datasets ad perceptual loss with VGG19 ad GAN ca be foud at weights/sample_weights/rd-C6-D20-G64-G064-x2/ArtefactCacellig/rd-C6-D20-G64-G064-x2_ArtefactCacellig_epoch219.hdf5 We recommed usig these weights oly whe cacellig compressio artefacts is a desirable effect.

RDN Network architecture

The mai parameters of the architecture structure are:

D - umber of Residual Dese Blocks (RDB)
C - umber of covolutioal layers stacked iside a RDB
G - umber of feature maps of each covolutioal layers iside the RDBs
G0 - umber of feature maps for covolutios outside of RDBs ad of each RBD output

source: Residual Dese Network for Image Super-Resolutio

RRDN Network architecture

The mai parameters of the architecture structure are:

T - umber of Residual i Residual Dese Blocks (RRDB)
D - umber of Residual Dese Blocks (RDB) isider each RRDB
C - umber of covolutioal layers stacked iside a RDB
G - umber of feature maps of each covolutioal layers iside the RDBs
G0 - umber of feature maps for covolutios outside of RDBs ad of each RBD output

source: ESRGAN: Ehaced Super-Resolutio Geerative Adversarial Networks

Cotribute

We welcome all kids of cotributios, models traied o differet datasets, ew model architectures ad/or hyperparameters combiatios that improve the performace of the curretly published model.

Will publish the performaces of ew models i this repository.

See the Cotributio guide for more details.

Bump versio

To bump up the versio, use

bumpversio {part} setup.py

Citatio

Please cite our work i your publicatios if it helps your research.

@misc{cardiale2018isr,
  title={ISR},
  author={Fracesco Cardiale et al.},
  year={2018},
  howpublished={\url{https://github.com/idealo/image-super-resolutio}},
}

Maitaiers

Fracesco Cardiale, github: cfracesco
Dat Tra, github: datitra

Copyright

See LICENSE for details.

Image-Super-Resolution

技术信息

作品详情