LumiNet: Latent Intrinsics Meets Diffusion Models for Indoor Scene
  Relighting
We introduce LumiNet, a novel architecture that leverages generative models
and latent intrinsic representations for effective lighting transfer. Given a
source image and a target lighting image, LumiNet synthesizes a relit version
of the source scene that captures the target's lighting. Our approach makes two
key contributions: a data curation strategy from the StyleGAN-based relighting
model for our training, and a modified diffusion-based ControlNet that
processes both latent intrinsic properties from the source image and latent
extrinsic properties from the target image. We further improve lighting
transfer through a learned adaptor (MLP) that injects the target's latent
extrinsic properties via cross-attention and fine-tuning.
  Unlike traditional ControlNet, which generates images with conditional maps
from a single scene, LumiNet processes latent representations from two
different images - preserving geometry and albedo from the source while
transferring lighting characteristics from the target. Experiments demonstrate
that our method successfully transfers complex lighting phenomena including
specular highlights and indirect illumination across scenes with varying
spatial layouts and materials, outperforming existing approaches on challenging
indoor scenes using only images as input.
6751af55cee25c032a044d06
Xiaoyan Xing
claimed_verified
2024-12-05T14:29:05.624Z
654b92bebe11400417aecff9
/avatars/b3e442dedb29544256b352e059cc829b.svg
Xiaoyan
user
xyxingx
6751af55cee25c032a044d07
Konrad Groh
6751af55cee25c032a044d08
Sezer Karaoglu
6751af55cee25c032a044d09
Theo Gevers
6751af55cee25c032a044d0a
Anand Bhattad
claimed_verified
2025-06-29T10:15:00.613Z
6303d5924ec2dfa82a5627de
/avatars/1e433b7fb9d30b9f42dec487d658b1e3.svg
Anand Bhattad
user
anandbhattad
LumiNet: Latent Intrinsics Meets Diffusion Models for Indoor Scene
  Relighting
We introduce LumiNet, a novel architecture that leverages generative models
and latent intrinsic representations for effective lighting transfer. Given a
source image and a target lighting image, LumiNet synthesizes a relit version
of the source scene that captures the target's lighting. Our approach makes two
key contributions: a data curation strategy from the StyleGAN-based relighting
model for our training, and a modified diffusion-based ControlNet that
processes both latent intrinsic properties from the source image and latent
extrinsic properties from the target image. We further improve lighting
transfer through a learned adaptor (MLP) that injects the target's latent
extrinsic properties via cross-attention and fine-tuning.
  Unlike traditional ControlNet, which generates images with conditional maps
from a single scene, LumiNet processes latent representations from two
different images - preserving geometry and albedo from the source while
transferring lighting characteristics from the target. Experiments demonstrate
that our method successfully transfers complex lighting phenomena including
specular highlights and indirect illumination across scenes with varying
spatial layouts and materials, outperforming existing approaches on challenging
indoor scenes using only images as input.
LumiNet uses a generative model with latent representations to transfer lighting from a target image to a source image, preserving geometry and albedo while injecting target lighting properties via a learned adaptor.
generative models latent intrinsic representations lighting transfer StyleGAN diffusion-based ControlNet latent extrinsic properties cross-attention learned adaptor MLP