title: LightIt: Illumination Modeling and Control for Diffusion Models
summary: We introduce LightIt, a method for explicit illumination control for image
  generation. Recent generative methods lack lighting control, which is crucial
  to numerous artistic aspects of image generation such as setting the overall
  mood or cinematic appearance. To overcome these limitations, we propose to
  condition the generation on shading and normal maps. We model the lighting with
  single bounce shading, which includes cast shadows. We first train a shading
  estimation module to generate a dataset of real-world images and shading pairs.
  Then, we train a control network using the estimated shading and normals as
  input. Our method demonstrates high-quality image generation and lighting
  control in numerous scenes. Additionally, we use our generated dataset to train
  an identity-preserving relighting model, conditioned on an image and a target
  shading. Our method is the first that enables the generation of images with
  controllable, consistent lighting and performs on par with specialized
  relighting state-of-the-art methods.
paper_authors._id: 65f904d3136fb8ddbdfaf7e5
paper_authors.name: Peter Kocsis
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-19T20:41:45.887Z
paper_authors._id: 65f9bd5171f78f1b44374b00
paper_authors.avatarUrl: /avatars/af487826bbdca5ee91b07d510f4c302f.svg
paper_authors.fullname: Peter Kocsis
paper_authors.type: user
paper_authors.user: PeterKocsis
paper_authors._id: 65f904d3136fb8ddbdfaf7e6
paper_authors.name: Julien Philip
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-19T09:54:09.700Z
paper_authors._id: 63e0dbb35ba41def8794dab8
paper_authors.avatarUrl: /avatars/38a28103b15d93c7d9243bc5418e623e.svg
paper_authors.fullname: Julien Philip
paper_authors.type: user
paper_authors.user: Jovphi
paper_authors._id: 65f904d3136fb8ddbdfaf7e7
paper_authors.name: Kalyan Sunkavalli
paper_authors._id: 65f904d3136fb8ddbdfaf7e8
paper_authors.name: Matthias Nießner
paper_authors._id: 65f904d3136fb8ddbdfaf7e9
paper_authors.name: Yannick Hold-Geoffroy
paper_title: LightIt: Illumination Modeling and Control for Diffusion Models
paper_summary: We introduce LightIt, a method for explicit illumination control for image
  generation. Recent generative methods lack lighting control, which is crucial
  to numerous artistic aspects of image generation such as setting the overall
  mood or cinematic appearance. To overcome these limitations, we propose to
  condition the generation on shading and normal maps. We model the lighting with
  single bounce shading, which includes cast shadows. We first train a shading
  estimation module to generate a dataset of real-world images and shading pairs.
  Then, we train a control network using the estimated shading and normals as
  input. Our method demonstrates high-quality image generation and lighting
  control in numerous scenes. Additionally, we use our generated dataset to train
  an identity-preserving relighting model, conditioned on an image and a target
  shading. Our method is the first that enables the generation of images with
  controllable, consistent lighting and performs on par with specialized
  relighting state-of-the-art methods.
paper_ai_summary: LightIt, a method for controllable illumination in image generation, uses shading and normal maps to produce high-quality images with consistent lighting, achieving results comparable to specialized relighting techniques.
paper_ai_keywords: shading estimation control network single bounce shading cast shadows identity-preserving relighting model