title: A Variational Perspective on Solving Inverse Problems with Diffusion
    Models
summary: Diffusion models have emerged as a key pillar of foundation models in visual
  domains. One of their critical applications is to universally solve different
  downstream inverse tasks via a single diffusion prior without re-training for
  each task. Most inverse tasks can be formulated as inferring a posterior
  distribution over data (e.g., a full image) given a measurement (e.g., a masked
  image). This is however challenging in diffusion models since the nonlinear and
  iterative nature of the diffusion process renders the posterior intractable. To
  cope with this challenge, we propose a variational approach that by design
  seeks to approximate the true posterior distribution. We show that our approach
  naturally leads to regularization by denoising diffusion process (RED-Diff)
  where denoisers at different timesteps concurrently impose different structural
  constraints over the image. To gauge the contribution of denoisers from
  different timesteps, we propose a weighting mechanism based on
  signal-to-noise-ratio (SNR). Our approach provides a new variational
  perspective for solving inverse problems with diffusion models, allowing us to
  formulate sampling as stochastic optimization, where one can simply apply
  off-the-shelf solvers with lightweight iterates. Our experiments for image
  restoration tasks such as inpainting and superresolution demonstrate the
  strengths of our method compared with state-of-the-art sampling-based diffusion
  models.
paper_authors._id: 64599a2e130a95cb18fd20ba
paper_authors.name: Morteza Mardani
paper_authors._id: 64599a2e130a95cb18fd20bb
paper_authors.name: Jiaming Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T14:21:44.854Z
paper_authors._id: 6312c2f4bbaa385279d2da1b
paper_authors.avatarUrl: /avatars/2bf162889fa726ed18cc205b3f28609e.svg
paper_authors.fullname: Jiaming Song
paper_authors.type: user
paper_authors.user: jiamings
paper_authors._id: 64599a2e130a95cb18fd20bc
paper_authors.name: Jan Kautz
paper_authors._id: 64599a2e130a95cb18fd20bd
paper_authors.name: Arash Vahdat
paper_title: A Variational Perspective on Solving Inverse Problems with Diffusion
    Models
paper_summary: Diffusion models have emerged as a key pillar of foundation models in visual
  domains. One of their critical applications is to universally solve different
  downstream inverse tasks via a single diffusion prior without re-training for
  each task. Most inverse tasks can be formulated as inferring a posterior
  distribution over data (e.g., a full image) given a measurement (e.g., a masked
  image). This is however challenging in diffusion models since the nonlinear and
  iterative nature of the diffusion process renders the posterior intractable. To
  cope with this challenge, we propose a variational approach that by design
  seeks to approximate the true posterior distribution. We show that our approach
  naturally leads to regularization by denoising diffusion process (RED-Diff)
  where denoisers at different timesteps concurrently impose different structural
  constraints over the image. To gauge the contribution of denoisers from
  different timesteps, we propose a weighting mechanism based on
  signal-to-noise-ratio (SNR). Our approach provides a new variational
  perspective for solving inverse problems with diffusion models, allowing us to
  formulate sampling as stochastic optimization, where one can simply apply
  off-the-shelf solvers with lightweight iterates. Our experiments for image
  restoration tasks such as inpainting and superresolution demonstrate the
  strengths of our method compared with state-of-the-art sampling-based diffusion
  models.
paper_ai_summary: The approach uses a variational method called RED-Diff to approximate the posterior distribution for solving inverse problems with diffusion models by leveraging denoisers at different timesteps and a signal-to-noise-ratio weighting mechanism, demonstrating superior performance in image restoration tasks.
paper_ai_keywords: diffusion models variational approach posterior distribution RED-Diff denoising diffusion process signal-to-noise-ratio inverse problems image restoration inpainting superresolution stochastic optimization