title: Comprehensive Relighting: Generalizable and Consistent Monocular Human
    Relighting and Harmonization
summary: This paper introduces Comprehensive Relighting, the first all-in-one approach
  that can both control and harmonize the lighting from an image or video of
  humans with arbitrary body parts from any scene. Building such a generalizable
  model is extremely challenging due to the lack of dataset, restricting existing
  image-based relighting models to a specific scenario (e.g., face or static
  human). To address this challenge, we repurpose a pre-trained diffusion model
  as a general image prior and jointly model the human relighting and background
  harmonization in the coarse-to-fine framework. To further enhance the temporal
  coherence of the relighting, we introduce an unsupervised temporal lighting
  model that learns the lighting cycle consistency from many real-world videos
  without any ground truth. In inference time, our temporal lighting module is
  combined with the diffusion models through the spatio-temporal feature blending
  algorithms without extra training; and we apply a new guided refinement as a
  post-processing to preserve the high-frequency details from the input image. In
  the experiments, Comprehensive Relighting shows a strong generalizability and
  lighting temporal coherence, outperforming existing image-based human
  relighting and harmonization methods.
paper_authors._id: 67f35d8bdb1a843e1ceff38f
paper_authors.name: Junying Wang
paper_authors._id: 67f35d8bdb1a843e1ceff390
paper_authors.name: Jingyuan Liu
paper_authors._id: 67f35d8bdb1a843e1ceff391
paper_authors.name: Xin Sun
paper_authors._id: 67f35d8bdb1a843e1ceff392
paper_authors.name: Krishna Kumar Singh
paper_authors._id: 67f35d8bdb1a843e1ceff393
paper_authors.name: Zhixin Shu
paper_authors._id: 67f35d8bdb1a843e1ceff394
paper_authors.name: He Zhang
paper_authors._id: 67f35d8bdb1a843e1ceff395
paper_authors.name: Jimei Yang
paper_authors._id: 67f35d8bdb1a843e1ceff396
paper_authors.name: Nanxuan Zhao
paper_authors._id: 67f35d8bdb1a843e1ceff397
paper_authors.name: Tuanfeng Y. Wang
paper_authors._id: 67f35d8bdb1a843e1ceff398
paper_authors.name: Simon S. Chen
paper_authors._id: 67f35d8bdb1a843e1ceff399
paper_authors.name: Ulrich Neumann
paper_authors._id: 67f35d8bdb1a843e1ceff39a
paper_authors.name: Jae Shin Yoon
paper_title: Comprehensive Relighting: Generalizable and Consistent Monocular Human
    Relighting and Harmonization
paper_summary: This paper introduces Comprehensive Relighting, the first all-in-one approach
  that can both control and harmonize the lighting from an image or video of
  humans with arbitrary body parts from any scene. Building such a generalizable
  model is extremely challenging due to the lack of dataset, restricting existing
  image-based relighting models to a specific scenario (e.g., face or static
  human). To address this challenge, we repurpose a pre-trained diffusion model
  as a general image prior and jointly model the human relighting and background
  harmonization in the coarse-to-fine framework. To further enhance the temporal
  coherence of the relighting, we introduce an unsupervised temporal lighting
  model that learns the lighting cycle consistency from many real-world videos
  without any ground truth. In inference time, our temporal lighting module is
  combined with the diffusion models through the spatio-temporal feature blending
  algorithms without extra training; and we apply a new guided refinement as a
  post-processing to preserve the high-frequency details from the input image. In
  the experiments, Comprehensive Relighting shows a strong generalizability and
  lighting temporal coherence, outperforming existing image-based human
  relighting and harmonization methods.
paper_ai_summary: Comprehensive Relighting uses diffusion models and temporal lighting modules to control and harmonize lighting in human images and videos with arbitrary body parts.
paper_ai_keywords: diffusion model image prior coarse-to-fine temporal lighting model lighting cycle consistency spatio-temporal feature blending guided refinement generalizability lighting temporal coherence