LightIt: Illumination Modeling and Control for Diffusion Models
We introduce LightIt, a method for explicit illumination control for image
generation. Recent generative methods lack lighting control, which is crucial
to numerous artistic aspects of image generation such as setting the overall
mood or cinematic appearance. To overcome these limitations, we propose to
condition the generation on shading and normal maps. We model the lighting with
single bounce shading, which includes cast shadows. We first train a shading
estimation module to generate a dataset of real-world images and shading pairs.
Then, we train a control network using the estimated shading and normals as
input. Our method demonstrates high-quality image generation and lighting
control in numerous scenes. Additionally, we use our generated dataset to train
an identity-preserving relighting model, conditioned on an image and a target
shading. Our method is the first that enables the generation of images with
controllable, consistent lighting and performs on par with specialized
relighting state-of-the-art methods.
65f904d3136fb8ddbdfaf7e5
Peter Kocsis
admin_assigned
2024-03-19T20:41:45.887Z
65f9bd5171f78f1b44374b00
/avatars/af487826bbdca5ee91b07d510f4c302f.svg
Peter Kocsis
user
PeterKocsis
65f904d3136fb8ddbdfaf7e6
Julien Philip
admin_assigned
2024-03-19T09:54:09.700Z
63e0dbb35ba41def8794dab8
/avatars/38a28103b15d93c7d9243bc5418e623e.svg
Julien Philip
user
Jovphi
65f904d3136fb8ddbdfaf7e7
Kalyan Sunkavalli
65f904d3136fb8ddbdfaf7e8
Matthias Nießner
65f904d3136fb8ddbdfaf7e9
Yannick Hold-Geoffroy
LightIt: Illumination Modeling and Control for Diffusion Models
We introduce LightIt, a method for explicit illumination control for image
generation. Recent generative methods lack lighting control, which is crucial
to numerous artistic aspects of image generation such as setting the overall
mood or cinematic appearance. To overcome these limitations, we propose to
condition the generation on shading and normal maps. We model the lighting with
single bounce shading, which includes cast shadows. We first train a shading
estimation module to generate a dataset of real-world images and shading pairs.
Then, we train a control network using the estimated shading and normals as
input. Our method demonstrates high-quality image generation and lighting
control in numerous scenes. Additionally, we use our generated dataset to train
an identity-preserving relighting model, conditioned on an image and a target
shading. Our method is the first that enables the generation of images with
controllable, consistent lighting and performs on par with specialized
relighting state-of-the-art methods.
LightIt, a method for controllable illumination in image generation, uses shading and normal maps to produce high-quality images with consistent lighting, achieving results comparable to specialized relighting techniques.
shading estimation control network single bounce shading cast shadows identity-preserving relighting model