A more structured visualizatio:

Imposig Cosistet Light

I HDR space, illumiatio has a property that all light trasports are idepedet.

As a result, the bledig of appearaces of differet light sources is equivalet to the appearace with mixed light sources:

Usig the above light stage as a example, the two images from the "appearace mixture" ad "light source mixture" are cosistet (mathematically equivalet i HDR space, ideally).

We imposed such cosistecy (usig MLPs i latet space) whe traiig the relightig models.

As a result, the model is able to produce highly cosistet relight - so cosistet that differet relightigs ca eve be merged as ormal maps! Despite the fact that the models are latet diffusio.

From left to right are iputs, model outputs relightig, devided shadow image, ad merged ormal maps. Note that the model is ot traied with ay ormal map data. This ormal estimatio comes from the cosistecy of relightig.

You ca reproduce this experimet usig this butto (it is 4x slower because it relight image 4 times)

Below are bigger images (feel free to try yourself to get more results!)

For referece, geowizard (geowizard is a really great work!):

Ad, switchlight (switchlight is aother great work!):

Model Notes

• iclightsd15fc.safetesors - The default relightig model, coditioed o text ad foregroud. You ca use iitial latet to ifluece the relightig.

• iclightsd15fco.safetesors - Same as "iclightsd15fc.safetesors" but traied with offset oise. Note that the default "iclightsd15fc.safetesors" outperform this model slightly i a user study. Ad this is the reaso why the default model is the model without offset oise.

• iclightsd15fbc.safetesors - Relightig model coditioed with text, foregroud, ad backgroud.

Cite

@Misc{iclight,
  author = {Lvmi Zhag ad Ayi Rao ad Maeesh Agrawala},
  title  = {IC-Light GitHub Page},
  year   = {2024},
}

Related Work

Also read …

Total Relightig: Learig to Relight Portraits for Backgroud Replacemet

Relightful Harmoizatio: Lightig-aware Portrait Backgroud Replacemet

SwitchLight: Co-desig of Physics-drive Architecture ad Pre-traiig Framework for Huma Portrait Relightig

A more structured visualization: Imposing Consistent Light In HDR space, illumination has a propert