title: Neural Gaffer: Relighting Any Object via Diffusion
summary: Single-image relighting is a challenging task that involves reasoning about
  the complex interplay between geometry, materials, and lighting. Many prior
  methods either support only specific categories of images, such as portraits,
  or require special capture conditions, like using a flashlight. Alternatively,
  some methods explicitly decompose a scene into intrinsic components, such as
  normals and BRDFs, which can be inaccurate or under-expressive. In this work,
  we propose a novel end-to-end 2D relighting diffusion model, called Neural
  Gaffer, that takes a single image of any object and can synthesize an accurate,
  high-quality relit image under any novel environmental lighting condition,
  simply by conditioning an image generator on a target environment map, without
  an explicit scene decomposition. Our method builds on a pre-trained diffusion
  model, and fine-tunes it on a synthetic relighting dataset, revealing and
  harnessing the inherent understanding of lighting present in the diffusion
  model. We evaluate our model on both synthetic and in-the-wild Internet imagery
  and demonstrate its advantages in terms of generalization and accuracy.
  Moreover, by combining with other generative methods, our model enables many
  downstream 2D tasks, such as text-based relighting and object insertion. Our
  model can also operate as a strong relighting prior for 3D tasks, such as
  relighting a radiance field.
paper_authors._id: 6669d80f1537c1bb0f8eb068
paper_authors.name: Haian Jin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T09:35:04.055Z
paper_authors._id: 64a04ad150251e408af12e01
paper_authors.avatarUrl: /avatars/e4f630616a16a7fc691b5275bb62f2aa.svg
paper_authors.fullname: Haian Jin
paper_authors.type: user
paper_authors.user: coast01
paper_authors._id: 6669d80f1537c1bb0f8eb069
paper_authors.name: Yuan Li
paper_authors._id: 6669d80f1537c1bb0f8eb06a
paper_authors.name: Fujun Luan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T09:35:26.325Z
paper_authors._id: 662ae0aa4f711ee4e1e17ffc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/662ae0aa4f711ee4e1e17ffc/nEi6tv8f6DIhTauwHpw_C.jpeg
paper_authors.fullname: Fujun Luan
paper_authors.type: user
paper_authors.user: luanfujun
paper_authors._id: 6669d80f1537c1bb0f8eb06b
paper_authors.name: Yuanbo Xiangli
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T09:35:32.164Z
paper_authors._id: 651edd0faa888068acb5d680
paper_authors.avatarUrl: /avatars/d5e03e8bc1d22bc67a3e684b52dc5598.svg
paper_authors.fullname: Yuanbo Xiangli
paper_authors.type: user
paper_authors.user: amberkk
paper_authors._id: 6669d80f1537c1bb0f8eb06c
paper_authors.name: Sai Bi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T09:35:38.403Z
paper_authors._id: 66303560597dc7c535168f07
paper_authors.avatarUrl: /avatars/a69d52a4a198c6c8451c1d137031bf07.svg
paper_authors.fullname: Sai Bi
paper_authors.type: user
paper_authors.user: saibi
paper_authors._id: 6669d80f1537c1bb0f8eb06d
paper_authors.name: Kai Zhang
paper_authors._id: 6669d80f1537c1bb0f8eb06e
paper_authors.name: Zexiang Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T09:35:56.113Z
paper_authors._id: 65565b1b8616d8d740abb976
paper_authors.avatarUrl: /avatars/c245b19ed05292bf1bf4a9d8cf2a409c.svg
paper_authors.fullname: Zexiang Xu
paper_authors.type: user
paper_authors.user: zexiangxu
paper_authors._id: 6669d80f1537c1bb0f8eb06f
paper_authors.name: Jin Sun
paper_authors._id: 6669d80f1537c1bb0f8eb070
paper_authors.name: Noah Snavely
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T09:36:17.215Z
paper_authors._id: 632b79ca0a9dc02985af9e0b
paper_authors.avatarUrl: /avatars/e2ca585398a3212b2956a14804d0ba67.svg
paper_authors.fullname: Noah Snavely
paper_authors.type: user
paper_authors.user: jimantha
paper_title: Neural Gaffer: Relighting Any Object via Diffusion
paper_summary: Single-image relighting is a challenging task that involves reasoning about
  the complex interplay between geometry, materials, and lighting. Many prior
  methods either support only specific categories of images, such as portraits,
  or require special capture conditions, like using a flashlight. Alternatively,
  some methods explicitly decompose a scene into intrinsic components, such as
  normals and BRDFs, which can be inaccurate or under-expressive. In this work,
  we propose a novel end-to-end 2D relighting diffusion model, called Neural
  Gaffer, that takes a single image of any object and can synthesize an accurate,
  high-quality relit image under any novel environmental lighting condition,
  simply by conditioning an image generator on a target environment map, without
  an explicit scene decomposition. Our method builds on a pre-trained diffusion
  model, and fine-tunes it on a synthetic relighting dataset, revealing and
  harnessing the inherent understanding of lighting present in the diffusion
  model. We evaluate our model on both synthetic and in-the-wild Internet imagery
  and demonstrate its advantages in terms of generalization and accuracy.
  Moreover, by combining with other generative methods, our model enables many
  downstream 2D tasks, such as text-based relighting and object insertion. Our
  model can also operate as a strong relighting prior for 3D tasks, such as
  relighting a radiance field.
paper_ai_summary: A novel end-to-end 2D relighting diffusion model, Neural Gaffer, synthesizes high-quality relit images under various lighting conditions without scene decomposition by fine-tuning a pre-trained diffusion model.
paper_ai_keywords: end-to-end relighting diffusion model Neural Gaffer pre-trained diffusion model synthetic relighting dataset environment map text-based relighting object insertion relighting prior radiance field