The Surprising Effectiveness of Diffusion Models for Optical Flow and
  Monocular Depth Estimation
Denoising diffusion probabilistic models have transformed image generation
with their impressive fidelity and diversity. We show that they also excel in
estimating optical flow and monocular depth, surprisingly, without
task-specific architectures and loss functions that are predominant for these
tasks. Compared to the point estimates of conventional regression-based
methods, diffusion models also enable Monte Carlo inference, e.g., capturing
uncertainty and ambiguity in flow and depth. With self-supervised pre-training,
the combined use of synthetic and real data for supervised training, and
technical innovations (infilling and step-unrolled denoising diffusion
training) to handle noisy-incomplete training data, and a simple form of
coarse-to-fine refinement, one can train state-of-the-art diffusion models for
depth and optical flow estimation. Extensive experiments focus on quantitative
performance against benchmarks, ablations, and the model's ability to capture
uncertainty and multimodality, and impute missing values. Our model, DDVM
(Denoising Diffusion Vision Model), obtains a state-of-the-art relative depth
error of 0.074 on the indoor NYU benchmark and an Fl-all outlier rate of 3.26\%
on the KITTI optical flow benchmark, about 25\% better than the best published
method. For an overview see https://diffusion-vision.github.io.
647eacb20ed7d0c8760f36ff
Saurabh Saxena
admin_assigned
2023-06-07T19:39:18.577Z
6480af48de559d48dbb20d95
/avatars/69f1e19921ba3ecdcd40bf55a6196aca.svg
Saurabh Saxena
user
saurabhsaxena
647eacb20ed7d0c8760f3700
Charles Herrmann
647eacb20ed7d0c8760f3701
Junhwa Hur
claimed_verified
2023-06-06T17:23:17.244Z
647f3046b0e96764589e107a
/avatars/1074ee5ec1b2d5d297406f9d455213c4.svg
Junhwa Hur
user
hurjunhwa
647eacb20ed7d0c8760f3702
Abhishek Kar
647eacb20ed7d0c8760f3703
Mohammad Norouzi
647eacb20ed7d0c8760f3704
Deqing Sun
647eacb20ed7d0c8760f3705
David J. Fleet
The Surprising Effectiveness of Diffusion Models for Optical Flow and
  Monocular Depth Estimation
Denoising diffusion probabilistic models have transformed image generation
with their impressive fidelity and diversity. We show that they also excel in
estimating optical flow and monocular depth, surprisingly, without
task-specific architectures and loss functions that are predominant for these
tasks. Compared to the point estimates of conventional regression-based
methods, diffusion models also enable Monte Carlo inference, e.g., capturing
uncertainty and ambiguity in flow and depth. With self-supervised pre-training,
the combined use of synthetic and real data for supervised training, and
technical innovations (infilling and step-unrolled denoising diffusion
training) to handle noisy-incomplete training data, and a simple form of
coarse-to-fine refinement, one can train state-of-the-art diffusion models for
depth and optical flow estimation. Extensive experiments focus on quantitative
performance against benchmarks, ablations, and the model's ability to capture
uncertainty and multimodality, and impute missing values. Our model, DDVM
(Denoising Diffusion Vision Model), obtains a state-of-the-art relative depth
error of 0.074 on the indoor NYU benchmark and an Fl-all outlier rate of 3.26\%
on the KITTI optical flow benchmark, about 25\% better than the best published
method. For an overview see https://diffusion-vision.github.io.
Denoising diffusion probabilistic models achieve state-of-the-art performance in optical flow and monocular depth estimation through self-supervised pre-training, handling of noisy data, and Monte Carlo inference.
denoising diffusion probabilistic models optical flow monocular depth Monte Carlo inference self-supervised pre-training synthetic data real data infilling step-unrolled denoising diffusion training coarse-to-fine refinement relative depth error Fl-all outlier rate