title: A Diffusion Approach to Radiance Field Relighting using
    Multi-Illumination Synthesis
summary: Relighting radiance fields is severely underconstrained for multi-view data,
  which is most often captured under a single illumination condition; It is
  especially hard for full scenes containing multiple objects. We introduce a
  method to create relightable radiance fields using such single-illumination
  data by exploiting priors extracted from 2D image diffusion models. We first
  fine-tune a 2D diffusion model on a multi-illumination dataset conditioned by
  light direction, allowing us to augment a single-illumination capture into a
  realistic -- but possibly inconsistent -- multi-illumination dataset from
  directly defined light directions. We use this augmented data to create a
  relightable radiance field represented by 3D Gaussian splats. To allow direct
  control of light direction for low-frequency lighting, we represent appearance
  with a multi-layer perceptron parameterized on light direction. To enforce
  multi-view consistency and overcome inaccuracies we optimize a per-image
  auxiliary feature vector. We show results on synthetic and real multi-view data
  under single illumination, demonstrating that our method successfully exploits
  2D diffusion model priors to allow realistic 3D relighting for complete scenes.
  Project site
  https://repo-sam.inria.fr/fungraph/generative-radiance-field-relighting/
paper_authors._id: 66e8381119f4837cdfe210c4
paper_authors.name: Yohan Poirier-Ginter
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-17T07:19:01.667Z
paper_authors._id: 637b48da56db0404b7c64b20
paper_authors.avatarUrl: /avatars/059b869881aefb230df72d3282eeab1b.svg
paper_authors.fullname: Yohan Poirier-Ginter
paper_authors.type: user
paper_authors.user: yohanpg
paper_authors._id: 66e8381119f4837cdfe210c5
paper_authors.name: Alban Gauthier
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-17T07:19:07.774Z
paper_authors._id: 634474ca6e988c6a792cfead
paper_authors.avatarUrl: /avatars/87fea91613c0854cc0f134e067eb47cd.svg
paper_authors.fullname: Alban Gauthier
paper_authors.type: user
paper_authors.user: albangwut
paper_authors._id: 66e8381119f4837cdfe210c6
paper_authors.name: Julien Phillip
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-17T15:08:18.255Z
paper_authors._id: 63e0dbb35ba41def8794dab8
paper_authors.avatarUrl: /avatars/38a28103b15d93c7d9243bc5418e623e.svg
paper_authors.fullname: Julien Philip
paper_authors.type: user
paper_authors.user: Jovphi
paper_authors._id: 66e8381119f4837cdfe210c7
paper_authors.name: Jean-Francois Lalonde
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-17T15:08:16.450Z
paper_authors._id: 66e9905f00d5cd69e36e0e1a
paper_authors.avatarUrl: /avatars/082eb98a634ae2f46c3664c2e6d331a1.svg
paper_authors.fullname: Jean-Francois Lalonde
paper_authors.type: user
paper_authors.user: jflalonde
paper_authors._id: 66e8381119f4837cdfe210c8
paper_authors.name: George Drettakis
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-18T11:34:43.792Z
paper_authors._id: 66eaa4ba1654494d77f28d42
paper_authors.avatarUrl: /avatars/42e6d78bc6c32ea43dad0684b210c815.svg
paper_authors.fullname: George Drettakis
paper_authors.type: user
paper_authors.user: gdrett
paper_title: A Diffusion Approach to Radiance Field Relighting using
    Multi-Illumination Synthesis
paper_summary: Relighting radiance fields is severely underconstrained for multi-view data,
  which is most often captured under a single illumination condition; It is
  especially hard for full scenes containing multiple objects. We introduce a
  method to create relightable radiance fields using such single-illumination
  data by exploiting priors extracted from 2D image diffusion models. We first
  fine-tune a 2D diffusion model on a multi-illumination dataset conditioned by
  light direction, allowing us to augment a single-illumination capture into a
  realistic -- but possibly inconsistent -- multi-illumination dataset from
  directly defined light directions. We use this augmented data to create a
  relightable radiance field represented by 3D Gaussian splats. To allow direct
  control of light direction for low-frequency lighting, we represent appearance
  with a multi-layer perceptron parameterized on light direction. To enforce
  multi-view consistency and overcome inaccuracies we optimize a per-image
  auxiliary feature vector. We show results on synthetic and real multi-view data
  under single illumination, demonstrating that our method successfully exploits
  2D diffusion model priors to allow realistic 3D relighting for complete scenes.
  Project site
  https://repo-sam.inria.fr/fungraph/generative-radiance-field-relighting/
paper_ai_summary: The method creates relightable 3D radiance fields using single-illumination data by leveraging 2D diffusion model priors, achieving realistic relighting in full scenes.
paper_ai_keywords: radiance fields 2D diffusion models multi-illumination Gaussian splats multi-layer perceptron per-image auxiliary feature vector multi-view consistency