title: LumiNet: Latent Intrinsics Meets Diffusion Models for Indoor Scene
    Relighting
summary: We introduce LumiNet, a novel architecture that leverages generative models
  and latent intrinsic representations for effective lighting transfer. Given a
  source image and a target lighting image, LumiNet synthesizes a relit version
  of the source scene that captures the target's lighting. Our approach makes two
  key contributions: a data curation strategy from the StyleGAN-based relighting
  model for our training, and a modified diffusion-based ControlNet that
  processes both latent intrinsic properties from the source image and latent
  extrinsic properties from the target image. We further improve lighting
  transfer through a learned adaptor (MLP) that injects the target's latent
  extrinsic properties via cross-attention and fine-tuning.
    Unlike traditional ControlNet, which generates images with conditional maps
  from a single scene, LumiNet processes latent representations from two
  different images - preserving geometry and albedo from the source while
  transferring lighting characteristics from the target. Experiments demonstrate
  that our method successfully transfers complex lighting phenomena including
  specular highlights and indirect illumination across scenes with varying
  spatial layouts and materials, outperforming existing approaches on challenging
  indoor scenes using only images as input.
paper_authors._id: 6751af55cee25c032a044d06
paper_authors.name: Xiaoyan Xing
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-05T14:29:05.624Z
paper_authors._id: 654b92bebe11400417aecff9
paper_authors.avatarUrl: /avatars/b3e442dedb29544256b352e059cc829b.svg
paper_authors.fullname: Xiaoyan
paper_authors.type: user
paper_authors.user: xyxingx
paper_authors._id: 6751af55cee25c032a044d07
paper_authors.name: Konrad Groh
paper_authors._id: 6751af55cee25c032a044d08
paper_authors.name: Sezer Karaoglu
paper_authors._id: 6751af55cee25c032a044d09
paper_authors.name: Theo Gevers
paper_authors._id: 6751af55cee25c032a044d0a
paper_authors.name: Anand Bhattad
paper_title: LumiNet: Latent Intrinsics Meets Diffusion Models for Indoor Scene
    Relighting
paper_summary: We introduce LumiNet, a novel architecture that leverages generative models
  and latent intrinsic representations for effective lighting transfer. Given a
  source image and a target lighting image, LumiNet synthesizes a relit version
  of the source scene that captures the target's lighting. Our approach makes two
  key contributions: a data curation strategy from the StyleGAN-based relighting
  model for our training, and a modified diffusion-based ControlNet that
  processes both latent intrinsic properties from the source image and latent
  extrinsic properties from the target image. We further improve lighting
  transfer through a learned adaptor (MLP) that injects the target's latent
  extrinsic properties via cross-attention and fine-tuning.
    Unlike traditional ControlNet, which generates images with conditional maps
  from a single scene, LumiNet processes latent representations from two
  different images - preserving geometry and albedo from the source while
  transferring lighting characteristics from the target. Experiments demonstrate
  that our method successfully transfers complex lighting phenomena including
  specular highlights and indirect illumination across scenes with varying
  spatial layouts and materials, outperforming existing approaches on challenging
  indoor scenes using only images as input.
paper_ai_summary: LumiNet uses a generative model with latent representations to transfer lighting from a target image to a source image, preserving geometry and albedo while injecting target lighting properties via a learned adaptor.
paper_ai_keywords: generative models latent intrinsic representations lighting transfer StyleGAN diffusion-based ControlNet latent extrinsic properties cross-attention learned adaptor MLP