title: DiLightNet: Fine-grained Lighting Control for Diffusion-based Image
    Generation
summary: This paper presents a novel method for exerting fine-grained lighting control
  during text-driven diffusion-based image generation. While existing diffusion
  models already have the ability to generate images under any lighting
  condition, without additional guidance these models tend to correlate image
  content and lighting. Moreover, text prompts lack the necessary expressional
  power to describe detailed lighting setups. To provide the content creator with
  fine-grained control over the lighting during image generation, we augment the
  text-prompt with detailed lighting information in the form of radiance hints,
  i.e., visualizations of the scene geometry with a homogeneous canonical
  material under the target lighting. However, the scene geometry needed to
  produce the radiance hints is unknown. Our key observation is that we only need
  to guide the diffusion process, hence exact radiance hints are not necessary;
  we only need to point the diffusion model in the right direction. Based on this
  observation, we introduce a three stage method for controlling the lighting
  during image generation. In the first stage, we leverage a standard pretrained
  diffusion model to generate a provisional image under uncontrolled lighting.
  Next, in the second stage, we resynthesize and refine the foreground object in
  the generated image by passing the target lighting to a refined diffusion
  model, named DiLightNet, using radiance hints computed on a coarse shape of the
  foreground object inferred from the provisional image. To retain the texture
  details, we multiply the radiance hints with a neural encoding of the
  provisional synthesized image before passing it to DiLightNet. Finally, in the
  third stage, we resynthesize the background to be consistent with the lighting
  on the foreground object. We demonstrate and validate our lighting controlled
  diffusion model on a variety of text prompts and lighting conditions.
paper_authors._id: 65d4235ccafb9a86f34b1f0d
paper_authors.name: Chong Zeng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T10:15:56.792Z
paper_authors._id: 60747cbf3ea03830676542b5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60747cbf3ea03830676542b5/wGr1Jzz520JM9nZ-UcLyb.png
paper_authors.fullname: Chong Zeng
paper_authors.type: user
paper_authors.user: NCJ
paper_authors._id: 65d4235ccafb9a86f34b1f0e
paper_authors.name: Yue Dong
paper_authors._id: 65d4235ccafb9a86f34b1f0f
paper_authors.name: Pieter Peers
paper_authors._id: 65d4235ccafb9a86f34b1f10
paper_authors.name: Youkang Kong
paper_authors._id: 65d4235ccafb9a86f34b1f11
paper_authors.name: Hongzhi Wu
paper_authors._id: 65d4235ccafb9a86f34b1f12
paper_authors.name: Xin Tong
paper_title: DiLightNet: Fine-grained Lighting Control for Diffusion-based Image
    Generation
paper_summary: This paper presents a novel method for exerting fine-grained lighting control
  during text-driven diffusion-based image generation. While existing diffusion
  models already have the ability to generate images under any lighting
  condition, without additional guidance these models tend to correlate image
  content and lighting. Moreover, text prompts lack the necessary expressional
  power to describe detailed lighting setups. To provide the content creator with
  fine-grained control over the lighting during image generation, we augment the
  text-prompt with detailed lighting information in the form of radiance hints,
  i.e., visualizations of the scene geometry with a homogeneous canonical
  material under the target lighting. However, the scene geometry needed to
  produce the radiance hints is unknown. Our key observation is that we only need
  to guide the diffusion process, hence exact radiance hints are not necessary;
  we only need to point the diffusion model in the right direction. Based on this
  observation, we introduce a three stage method for controlling the lighting
  during image generation. In the first stage, we leverage a standard pretrained
  diffusion model to generate a provisional image under uncontrolled lighting.
  Next, in the second stage, we resynthesize and refine the foreground object in
  the generated image by passing the target lighting to a refined diffusion
  model, named DiLightNet, using radiance hints computed on a coarse shape of the
  foreground object inferred from the provisional image. To retain the texture
  details, we multiply the radiance hints with a neural encoding of the
  provisional synthesized image before passing it to DiLightNet. Finally, in the
  third stage, we resynthesize the background to be consistent with the lighting
  on the foreground object. We demonstrate and validate our lighting controlled
  diffusion model on a variety of text prompts and lighting conditions.
paper_ai_summary: A novel three-stage method uses radiance hints to enable fine-grained lighting control during text-driven diffusion-based image generation, ensuring consistency between foreground and background lighting.
paper_ai_keywords: diffusion-based image generation radiance hints scene geometry diffusion process DiLightNet neural encoding