title: Video Depth without Video Models
summary: Video depth estimation lifts monocular video clips to 3D by inferring dense
  depth at every frame. Recent advances in single-image depth estimation, brought
  about by the rise of large foundation models and the use of synthetic training
  data, have fueled a renewed interest in video depth. However, naively applying
  a single-image depth estimator to every frame of a video disregards temporal
  continuity, which not only leads to flickering but may also break when camera
  motion causes sudden changes in depth range. An obvious and principled solution
  would be to build on top of video foundation models, but these come with their
  own limitations; including expensive training and inference, imperfect 3D
  consistency, and stitching routines for the fixed-length (short) outputs. We
  take a step back and demonstrate how to turn a single-image latent diffusion
  model (LDM) into a state-of-the-art video depth estimator. Our model, which we
  call RollingDepth, has two main ingredients: (i) a multi-frame depth estimator
  that is derived from a single-image LDM and maps very short video snippets
  (typically frame triplets) to depth snippets. (ii) a robust, optimization-based
  registration algorithm that optimally assembles depth snippets sampled at
  various different frame rates back into a consistent video. RollingDepth is
  able to efficiently handle long videos with hundreds of frames and delivers
  more accurate depth videos than both dedicated video depth estimators and
  high-performing single-frame models. Project page: rollingdepth.github.io.
paper_authors._id: 674d64114e69219448bcf70d
paper_authors.name: Bingxin Ke
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T09:44:14.028Z
paper_authors._id: 63d90391da4f72339244c2a8
paper_authors.avatarUrl: /avatars/eb0e0259c391d59739c1a205c36bb539.svg
paper_authors.fullname: Bingxin Ke
paper_authors.type: user
paper_authors.user: Bingxin
paper_authors._id: 674d64114e69219448bcf70e
paper_authors.name: Dominik Narnhofer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T09:44:18.859Z
paper_authors._id: 674995c2d523f2743e6c3859
paper_authors.avatarUrl: /avatars/41b60f776a41d85da9ea0b1ed7d7615b.svg
paper_authors.fullname: Dominik Narnhofer
paper_authors.type: user
paper_authors.user: dnarnhofer
paper_authors._id: 674d64114e69219448bcf70f
paper_authors.name: Shengyu Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T09:44:25.115Z
paper_authors._id: 65b277cd47674826d1a08fd6
paper_authors.avatarUrl: /avatars/795fb6f321aafbc56535571bc4a85dee.svg
paper_authors.fullname: Sheng-Yu Huang
paper_authors.type: user
paper_authors.user: peterjohnson
paper_authors._id: 674d64114e69219448bcf710
paper_authors.name: Lei Ke
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-02T17:31:56.753Z
paper_authors._id: 6258a6455ea3a0a9b6de3f22
paper_authors.avatarUrl: /avatars/6eeed72a97fb24465e5e65583fbe50cf.svg
paper_authors.fullname: Lei Ke
paper_authors.type: user
paper_authors.user: lkeab
paper_authors._id: 674d64114e69219448bcf711
paper_authors.name: Torben Peters
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T09:44:39.755Z
paper_authors._id: 656d886ad848a6683a0f71c3
paper_authors.avatarUrl: /avatars/494f5fd5d5fa7849850cfe21f574f12c.svg
paper_authors.fullname: Torben Peters
paper_authors.type: user
paper_authors.user: PeterTor
paper_authors._id: 674d64114e69219448bcf712
paper_authors.name: Katerina Fragkiadaki
paper_authors._id: 674d64114e69219448bcf713
paper_authors.name: Anton Obukhov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T09:44:55.130Z
paper_authors._id: 62f93abbc4817cfc0756b6f8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62f93abbc4817cfc0756b6f8/rGYLaq-rmoJJYotkC1VXk.jpeg
paper_authors.fullname: Anton Obukhov
paper_authors.type: user
paper_authors.user: toshas
paper_authors._id: 674d64114e69219448bcf714
paper_authors.name: Konrad Schindler
paper_title: Video Depth without Video Models
paper_summary: Video depth estimation lifts monocular video clips to 3D by inferring dense
  depth at every frame. Recent advances in single-image depth estimation, brought
  about by the rise of large foundation models and the use of synthetic training
  data, have fueled a renewed interest in video depth. However, naively applying
  a single-image depth estimator to every frame of a video disregards temporal
  continuity, which not only leads to flickering but may also break when camera
  motion causes sudden changes in depth range. An obvious and principled solution
  would be to build on top of video foundation models, but these come with their
  own limitations; including expensive training and inference, imperfect 3D
  consistency, and stitching routines for the fixed-length (short) outputs. We
  take a step back and demonstrate how to turn a single-image latent diffusion
  model (LDM) into a state-of-the-art video depth estimator. Our model, which we
  call RollingDepth, has two main ingredients: (i) a multi-frame depth estimator
  that is derived from a single-image LDM and maps very short video snippets
  (typically frame triplets) to depth snippets. (ii) a robust, optimization-based
  registration algorithm that optimally assembles depth snippets sampled at
  various different frame rates back into a consistent video. RollingDepth is
  able to efficiently handle long videos with hundreds of frames and delivers
  more accurate depth videos than both dedicated video depth estimators and
  high-performing single-frame models. Project page: rollingdepth.github.io.
paper_ai_keywords: depth estimation monocular video clips dense depth single-image depth estimation large foundation models synthetic training data video depth temporal continuity flickering video foundation models expensive training inference imperfect 3D consistency stitching routines latent diffusion model LDM multi-frame depth estimator optimization-based registration algorithm RollingDepth depth snippets frame triplets depth videos dedicated video depth estimators high-performing single-frame models