title: Diffusion with Forward Models: Solving Stochastic Inverse Problems
    Without Direct Supervision
summary: Denoising diffusion models are a powerful type of generative models used to
  capture complex distributions of real-world signals. However, their
  applicability is limited to scenarios where training samples are readily
  available, which is not always the case in real-world applications. For
  example, in inverse graphics, the goal is to generate samples from a
  distribution of 3D scenes that align with a given image, but ground-truth 3D
  scenes are unavailable and only 2D images are accessible. To address this
  limitation, we propose a novel class of denoising diffusion probabilistic
  models that learn to sample from distributions of signals that are never
  directly observed. Instead, these signals are measured indirectly through a
  known differentiable forward model, which produces partial observations of the
  unknown signal. Our approach involves integrating the forward model directly
  into the denoising process. This integration effectively connects the
  generative modeling of observations with the generative modeling of the
  underlying signals, allowing for end-to-end training of a conditional
  generative model over signals. During inference, our approach enables sampling
  from the distribution of underlying signals that are consistent with a given
  partial observation. We demonstrate the effectiveness of our method on three
  challenging computer vision tasks. For instance, in the context of inverse
  graphics, our model enables direct sampling from the distribution of 3D scenes
  that align with a single 2D input image.
paper_authors._id: 649281a961c2f48ab3319458
paper_authors.name: Ayush Tewari
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-21T20:58:02.933Z
paper_authors._id: 6386e304b3ef0523c6419a05
paper_authors.avatarUrl: /avatars/3e6d292962dd27c8b299e3c96697061b.svg
paper_authors.fullname: Ayush Tewari
paper_authors.type: user
paper_authors.user: ayusht
paper_authors._id: 649281a961c2f48ab3319459
paper_authors.name: Tianwei Yin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-21T21:28:28.552Z
paper_authors._id: 63363b864067f020756275b7
paper_authors.avatarUrl: /avatars/bf18f9f76183f81589a80b3d0215f1a5.svg
paper_authors.fullname: Tianwei Yin
paper_authors.type: user
paper_authors.user: tianweiy
paper_authors._id: 649281a961c2f48ab331945a
paper_authors.name: George Cazenavette
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-21T21:28:51.896Z
paper_authors._id: 62f68717c3372328414b4403
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62f68717c3372328414b4403/3GjzdZQqb4-tIAkdrh4GN.png
paper_authors.fullname: George Cazenavette
paper_authors.type: user
paper_authors.user: georgecazenavette
paper_authors._id: 649281a961c2f48ab331945b
paper_authors.name: Semon Rezchikov
paper_authors._id: 649281a961c2f48ab331945c
paper_authors.name: Joshua B. Tenenbaum
paper_authors._id: 649281a961c2f48ab331945d
paper_authors.name: Frédo Durand
paper_authors._id: 649281a961c2f48ab331945e
paper_authors.name: William T. Freeman
paper_authors._id: 649281a961c2f48ab331945f
paper_authors.name: Vincent Sitzmann
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-23T07:32:15.573Z
paper_authors._id: 6494740dcd40d639d5f309cf
paper_authors.avatarUrl: /avatars/65eb911f1063dc10791359531f842122.svg
paper_authors.fullname: Vincent Sitzmann
paper_authors.type: user
paper_authors.user: vsitzmann
paper_title: Diffusion with Forward Models: Solving Stochastic Inverse Problems
    Without Direct Supervision
paper_summary: Denoising diffusion models are a powerful type of generative models used to
  capture complex distributions of real-world signals. However, their
  applicability is limited to scenarios where training samples are readily
  available, which is not always the case in real-world applications. For
  example, in inverse graphics, the goal is to generate samples from a
  distribution of 3D scenes that align with a given image, but ground-truth 3D
  scenes are unavailable and only 2D images are accessible. To address this
  limitation, we propose a novel class of denoising diffusion probabilistic
  models that learn to sample from distributions of signals that are never
  directly observed. Instead, these signals are measured indirectly through a
  known differentiable forward model, which produces partial observations of the
  unknown signal. Our approach involves integrating the forward model directly
  into the denoising process. This integration effectively connects the
  generative modeling of observations with the generative modeling of the
  underlying signals, allowing for end-to-end training of a conditional
  generative model over signals. During inference, our approach enables sampling
  from the distribution of underlying signals that are consistent with a given
  partial observation. We demonstrate the effectiveness of our method on three
  challenging computer vision tasks. For instance, in the context of inverse
  graphics, our model enables direct sampling from the distribution of 3D scenes
  that align with a single 2D input image.
paper_ai_summary: A novel denoising diffusion model integrates a forward model to sample indirect signals, enabling end-to-end training for challenging vision tasks like inverse graphics.
paper_ai_keywords: denoising diffusion models generative models distribution of real-world signals inverse graphics 3D scenes 2D images ground-truth differentiable forward model partial observations conditional generative model sampling from underlying signals