title: The Surprising Effectiveness of Diffusion Models for Optical Flow and
    Monocular Depth Estimation
summary: Denoising diffusion probabilistic models have transformed image generation
  with their impressive fidelity and diversity. We show that they also excel in
  estimating optical flow and monocular depth, surprisingly, without
  task-specific architectures and loss functions that are predominant for these
  tasks. Compared to the point estimates of conventional regression-based
  methods, diffusion models also enable Monte Carlo inference, e.g., capturing
  uncertainty and ambiguity in flow and depth. With self-supervised pre-training,
  the combined use of synthetic and real data for supervised training, and
  technical innovations (infilling and step-unrolled denoising diffusion
  training) to handle noisy-incomplete training data, and a simple form of
  coarse-to-fine refinement, one can train state-of-the-art diffusion models for
  depth and optical flow estimation. Extensive experiments focus on quantitative
  performance against benchmarks, ablations, and the model's ability to capture
  uncertainty and multimodality, and impute missing values. Our model, DDVM
  (Denoising Diffusion Vision Model), obtains a state-of-the-art relative depth
  error of 0.074 on the indoor NYU benchmark and an Fl-all outlier rate of 3.26\%
  on the KITTI optical flow benchmark, about 25\% better than the best published
  method. For an overview see https://diffusion-vision.github.io.
paper_authors._id: 647eacb20ed7d0c8760f36ff
paper_authors.name: Saurabh Saxena
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-07T19:39:18.577Z
paper_authors._id: 6480af48de559d48dbb20d95
paper_authors.avatarUrl: /avatars/69f1e19921ba3ecdcd40bf55a6196aca.svg
paper_authors.fullname: Saurabh Saxena
paper_authors.type: user
paper_authors.user: saurabhsaxena
paper_authors._id: 647eacb20ed7d0c8760f3700
paper_authors.name: Charles Herrmann
paper_authors._id: 647eacb20ed7d0c8760f3701
paper_authors.name: Junhwa Hur
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-06T17:23:17.244Z
paper_authors._id: 647f3046b0e96764589e107a
paper_authors.avatarUrl: /avatars/1074ee5ec1b2d5d297406f9d455213c4.svg
paper_authors.fullname: Junhwa Hur
paper_authors.type: user
paper_authors.user: hurjunhwa
paper_authors._id: 647eacb20ed7d0c8760f3702
paper_authors.name: Abhishek Kar
paper_authors._id: 647eacb20ed7d0c8760f3703
paper_authors.name: Mohammad Norouzi
paper_authors._id: 647eacb20ed7d0c8760f3704
paper_authors.name: Deqing Sun
paper_authors._id: 647eacb20ed7d0c8760f3705
paper_authors.name: David J. Fleet
paper_title: The Surprising Effectiveness of Diffusion Models for Optical Flow and
    Monocular Depth Estimation
paper_summary: Denoising diffusion probabilistic models have transformed image generation
  with their impressive fidelity and diversity. We show that they also excel in
  estimating optical flow and monocular depth, surprisingly, without
  task-specific architectures and loss functions that are predominant for these
  tasks. Compared to the point estimates of conventional regression-based
  methods, diffusion models also enable Monte Carlo inference, e.g., capturing
  uncertainty and ambiguity in flow and depth. With self-supervised pre-training,
  the combined use of synthetic and real data for supervised training, and
  technical innovations (infilling and step-unrolled denoising diffusion
  training) to handle noisy-incomplete training data, and a simple form of
  coarse-to-fine refinement, one can train state-of-the-art diffusion models for
  depth and optical flow estimation. Extensive experiments focus on quantitative
  performance against benchmarks, ablations, and the model's ability to capture
  uncertainty and multimodality, and impute missing values. Our model, DDVM
  (Denoising Diffusion Vision Model), obtains a state-of-the-art relative depth
  error of 0.074 on the indoor NYU benchmark and an Fl-all outlier rate of 3.26\%
  on the KITTI optical flow benchmark, about 25\% better than the best published
  method. For an overview see https://diffusion-vision.github.io.
paper_ai_keywords: denoising diffusion probabilistic models image generation optical flow monocular depth task-specific architectures Monte Carlo inference self-supervised pre-training synthetic data real data infilling step-unrolled denoising diffusion training coarse-fine refinement relative depth error NYU benchmark Fl-all outlier rate KITTI optical flow benchmark DDVM (Denoising Diffusion Vision Model)