title: SpotLight: Shadow-Guided Object Relighting via Diffusion
summary: Recent work has shown that diffusion models can be used as powerful neural
  rendering engines that can be leveraged for inserting virtual objects into
  images. Unlike typical physics-based renderers, however, neural rendering
  engines are limited by the lack of manual control over the lighting setup,
  which is often essential for improving or personalizing the desired image
  outcome. In this paper, we show that precise lighting control can be achieved
  for object relighting simply by specifying the desired shadows of the object.
  Rather surprisingly, we show that injecting only the shadow of the object into
  a pre-trained diffusion-based neural renderer enables it to accurately shade
  the object according to the desired light position, while properly harmonizing
  the object (and its shadow) within the target background image. Our method,
  SpotLight, leverages existing neural rendering approaches and achieves
  controllable relighting results with no additional training. Specifically, we
  demonstrate its use with two neural renderers from the recent literature. We
  show that SpotLight achieves superior object compositing results, both
  quantitatively and perceptually, as confirmed by a user study, outperforming
  existing diffusion-based models specifically designed for relighting.
paper_authors._id: 674e0997e7861d5a0277b79f
paper_authors.name: Frédéric Fortier-Chouinard
paper_authors._id: 674e0997e7861d5a0277b7a0
paper_authors.name: Zitian Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-02T22:07:18.112Z
paper_authors._id: 655dbda6668b64adf1495528
paper_authors.avatarUrl: /avatars/dbf6072dcf0905873289d78209a92f87.svg
paper_authors.fullname: Zitian Zhang
paper_authors.type: user
paper_authors.user: zzt76
paper_authors._id: 674e0997e7861d5a0277b7a1
paper_authors.name: Louis-Etienne Messier
paper_authors._id: 674e0997e7861d5a0277b7a2
paper_authors.name: Mathieu Garon
paper_authors._id: 674e0997e7861d5a0277b7a3
paper_authors.name: Anand Bhattad
paper_authors._id: 674e0997e7861d5a0277b7a4
paper_authors.name: Jean-François Lalonde
paper_title: SpotLight: Shadow-Guided Object Relighting via Diffusion
paper_summary: Recent work has shown that diffusion models can be used as powerful neural
  rendering engines that can be leveraged for inserting virtual objects into
  images. Unlike typical physics-based renderers, however, neural rendering
  engines are limited by the lack of manual control over the lighting setup,
  which is often essential for improving or personalizing the desired image
  outcome. In this paper, we show that precise lighting control can be achieved
  for object relighting simply by specifying the desired shadows of the object.
  Rather surprisingly, we show that injecting only the shadow of the object into
  a pre-trained diffusion-based neural renderer enables it to accurately shade
  the object according to the desired light position, while properly harmonizing
  the object (and its shadow) within the target background image. Our method,
  SpotLight, leverages existing neural rendering approaches and achieves
  controllable relighting results with no additional training. Specifically, we
  demonstrate its use with two neural renderers from the recent literature. We
  show that SpotLight achieves superior object compositing results, both
  quantitatively and perceptually, as confirmed by a user study, outperforming
  existing diffusion-based models specifically designed for relighting.
paper_ai_summary: SpotLight enables controllable relighting in neural rendering by specifying object shadows, outperforming existing models without additional training.
paper_ai_keywords: diffusion models neural renderers object relighting shadows SpotLight object compositing user study