title: SynthLight: Portrait Relighting with Diffusion Model by Learning to
    Re-render Synthetic Faces
summary: We introduce SynthLight, a diffusion model for portrait relighting. Our
  approach frames image relighting as a re-rendering problem, where pixels are
  transformed in response to changes in environmental lighting conditions. Using
  a physically-based rendering engine, we synthesize a dataset to simulate this
  lighting-conditioned transformation with 3D head assets under varying lighting.
  We propose two training and inference strategies to bridge the gap between the
  synthetic and real image domains: (1) multi-task training that takes advantage
  of real human portraits without lighting labels; (2) an inference time
  diffusion sampling procedure based on classifier-free guidance that leverages
  the input portrait to better preserve details. Our method generalizes to
  diverse real photographs and produces realistic illumination effects, including
  specular highlights and cast shadows, while preserving the subject's identity.
  Our quantitative experiments on Light Stage data demonstrate results comparable
  to state-of-the-art relighting methods. Our qualitative results on in-the-wild
  images showcase rich and unprecedented illumination effects. Project Page:
  https://vrroom.github.io/synthlight/
paper_authors._id: 6789e9b78a27185f5084533a
paper_authors.name: Sumit Chaturvedi
paper_authors._id: 6789e9b78a27185f5084533b
paper_authors.name: Mengwei Ren
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-17T10:35:43.879Z
paper_authors._id: 63b48ed5a50cfcefda9dbe67
paper_authors.avatarUrl: /avatars/98a2f07ea6a7ce3792f250cf9fecf402.svg
paper_authors.fullname: Mengwei Ren
paper_authors.type: user
paper_authors.user: mengweir
paper_authors._id: 6789e9b78a27185f5084533c
paper_authors.name: Yannick Hold-Geoffroy
paper_authors._id: 6789e9b78a27185f5084533d
paper_authors.name: Jingyuan Liu
paper_authors._id: 6789e9b78a27185f5084533e
paper_authors.name: Julie Dorsey
paper_authors._id: 6789e9b78a27185f5084533f
paper_authors.name: Zhixin Shu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-17T10:36:05.684Z
paper_authors._id: 62a8efd508a7ea93ff18785a
paper_authors.avatarUrl: /avatars/ff259233d437833a304329bb973a5a04.svg
paper_authors.fullname: Zhixin Shu
paper_authors.type: user
paper_authors.user: zhixinshu
paper_title: SynthLight: Portrait Relighting with Diffusion Model by Learning to
    Re-render Synthetic Faces
paper_summary: We introduce SynthLight, a diffusion model for portrait relighting. Our
  approach frames image relighting as a re-rendering problem, where pixels are
  transformed in response to changes in environmental lighting conditions. Using
  a physically-based rendering engine, we synthesize a dataset to simulate this
  lighting-conditioned transformation with 3D head assets under varying lighting.
  We propose two training and inference strategies to bridge the gap between the
  synthetic and real image domains: (1) multi-task training that takes advantage
  of real human portraits without lighting labels; (2) an inference time
  diffusion sampling procedure based on classifier-free guidance that leverages
  the input portrait to better preserve details. Our method generalizes to
  diverse real photographs and produces realistic illumination effects, including
  specular highlights and cast shadows, while preserving the subject's identity.
  Our quantitative experiments on Light Stage data demonstrate results comparable
  to state-of-the-art relighting methods. Our qualitative results on in-the-wild
  images showcase rich and unprecedented illumination effects. Project Page:
  https://vrroom.github.io/synthlight/
paper_ai_summary: SynthLight, a diffusion model, achieves realistic portrait relighting by combining synthetic data generation, multi-task training, and classifier-free guidance, producing photo-realistic illumination effects while preserving subject identity.
paper_ai_keywords: diffusion model image relighting re-rendering problem physically-based rendering engine 3D head assets multi-task training classifier-free guidance Light Stage data