SynthLight: Portrait Relighting with Diffusion Model by Learning to
  Re-render Synthetic Faces
We introduce SynthLight, a diffusion model for portrait relighting. Our
approach frames image relighting as a re-rendering problem, where pixels are
transformed in response to changes in environmental lighting conditions. Using
a physically-based rendering engine, we synthesize a dataset to simulate this
lighting-conditioned transformation with 3D head assets under varying lighting.
We propose two training and inference strategies to bridge the gap between the
synthetic and real image domains: (1) multi-task training that takes advantage
of real human portraits without lighting labels; (2) an inference time
diffusion sampling procedure based on classifier-free guidance that leverages
the input portrait to better preserve details. Our method generalizes to
diverse real photographs and produces realistic illumination effects, including
specular highlights and cast shadows, while preserving the subject's identity.
Our quantitative experiments on Light Stage data demonstrate results comparable
to state-of-the-art relighting methods. Our qualitative results on in-the-wild
images showcase rich and unprecedented illumination effects. Project Page:
https://vrroom.github.io/synthlight/
6789e9b78a27185f5084533a
Sumit Chaturvedi
6789e9b78a27185f5084533b
Mengwei Ren
admin_assigned
2025-01-17T10:35:43.879Z
63b48ed5a50cfcefda9dbe67
/avatars/98a2f07ea6a7ce3792f250cf9fecf402.svg
Mengwei Ren
user
mengweir
6789e9b78a27185f5084533c
Yannick Hold-Geoffroy
6789e9b78a27185f5084533d
Jingyuan Liu
6789e9b78a27185f5084533e
Julie Dorsey
6789e9b78a27185f5084533f
Zhixin Shu
admin_assigned
2025-01-17T10:36:05.684Z
62a8efd508a7ea93ff18785a
/avatars/ff259233d437833a304329bb973a5a04.svg
Zhixin Shu
user
zhixinshu
SynthLight: Portrait Relighting with Diffusion Model by Learning to
  Re-render Synthetic Faces
We introduce SynthLight, a diffusion model for portrait relighting. Our
approach frames image relighting as a re-rendering problem, where pixels are
transformed in response to changes in environmental lighting conditions. Using
a physically-based rendering engine, we synthesize a dataset to simulate this
lighting-conditioned transformation with 3D head assets under varying lighting.
We propose two training and inference strategies to bridge the gap between the
synthetic and real image domains: (1) multi-task training that takes advantage
of real human portraits without lighting labels; (2) an inference time
diffusion sampling procedure based on classifier-free guidance that leverages
the input portrait to better preserve details. Our method generalizes to
diverse real photographs and produces realistic illumination effects, including
specular highlights and cast shadows, while preserving the subject's identity.
Our quantitative experiments on Light Stage data demonstrate results comparable
to state-of-the-art relighting methods. Our qualitative results on in-the-wild
images showcase rich and unprecedented illumination effects. Project Page:
https://vrroom.github.io/synthlight/
SynthLight, a diffusion model, achieves realistic portrait relighting by combining synthetic data generation, multi-task training, and classifier-free guidance, producing photo-realistic illumination effects while preserving subject identity.
diffusion model image relighting re-rendering problem physically-based rendering engine 3D head assets multi-task training classifier-free guidance Light Stage data