title: Learning Temporally Consistent Video Depth from Video Diffusion Priors
summary: This work addresses the challenge of video depth estimation, which expects
  not only per-frame accuracy but, more importantly, cross-frame consistency.
  Instead of directly developing a depth estimator from scratch, we reformulate
  the prediction task into a conditional generation problem. This allows us to
  leverage the prior knowledge embedded in existing video generation models,
  thereby reducing learn- ing difficulty and enhancing generalizability.
  Concretely, we study how to tame the public Stable Video Diffusion (SVD) to
  predict reliable depth from input videos using a mixture of image depth and
  video depth datasets. We empirically confirm that a procedural training
  strategy - first optimizing the spatial layers of SVD and then optimizing the
  temporal layers while keeping the spatial layers frozen - yields the best
  results in terms of both spatial accuracy and temporal consistency. We further
  examine the sliding window strategy for inference on arbitrarily long videos.
  Our observations indicate a trade-off between efficiency and performance, with
  a one-frame overlap already producing favorable results. Extensive experimental
  results demonstrate the superiority of our approach, termed ChronoDepth, over
  existing alternatives, particularly in terms of the temporal consistency of the
  estimated depth. Additionally, we highlight the benefits of more consistent
  video depth in two practical applications: depth-conditioned video generation
  and novel view synthesis. Our project page is available at
  https://jhaoshao.github.io/ChronoDepth/{this http URL}.
paper_authors._id: 665e9487f4f7557684cbaaa9
paper_authors.name: Jiahao Shao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-04T08:08:01.319Z
paper_authors._id: 644a16eded295eb43e6815e1
paper_authors.avatarUrl: /avatars/0ee0f8c649247c1c618111899f0a577d.svg
paper_authors.fullname: Jiahao Shao
paper_authors.type: user
paper_authors.user: jhshao
paper_authors._id: 665e9487f4f7557684cbaaaa
paper_authors.name: Yuanbo Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-04T08:08:14.007Z
paper_authors._id: 65d5f8aa933e3da6e7d5d627
paper_authors.avatarUrl: /avatars/b82ea295bf07f9060e781f07deebba31.svg
paper_authors.fullname: yangyuanbo
paper_authors.type: user
paper_authors.user: yangyuanbo
paper_authors._id: 665e9487f4f7557684cbaaab
paper_authors.name: Hongyu Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-03T11:17:15.837Z
paper_authors._id: 664d9d1eafe6e8c3a986ef67
paper_authors.avatarUrl: /avatars/88535076724e97f412ecd362a38904db.svg
paper_authors.fullname: hyzhou
paper_authors.type: user
paper_authors.user: hyzhou404
paper_authors._id: 665e9487f4f7557684cbaaac
paper_authors.name: Youmin Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-04T08:09:06.777Z
paper_authors._id: 65e054827409351736166d73
paper_authors.avatarUrl: /avatars/d56b8af16b47cdac7a47d7bc0563687a.svg
paper_authors.fullname: Youmin Zhang
paper_authors.type: user
paper_authors.user: Rock-youmi
paper_authors._id: 665e9487f4f7557684cbaaad
paper_authors.name: Yujun Shen
paper_authors._id: 665e9487f4f7557684cbaaae
paper_authors.name: Matteo Poggi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-04T08:09:20.630Z
paper_authors._id: 6613b33d1dc41eabb76b8d82
paper_authors.avatarUrl: /avatars/5c31d071c53dee4c19fcceb5b437c130.svg
paper_authors.fullname: Matteo Poggi
paper_authors.type: user
paper_authors.user: mpoggi
paper_authors._id: 665e9487f4f7557684cbaaaf
paper_authors.name: Yiyi Liao
paper_title: Learning Temporally Consistent Video Depth from Video Diffusion Priors
paper_summary: This work addresses the challenge of video depth estimation, which expects
  not only per-frame accuracy but, more importantly, cross-frame consistency.
  Instead of directly developing a depth estimator from scratch, we reformulate
  the prediction task into a conditional generation problem. This allows us to
  leverage the prior knowledge embedded in existing video generation models,
  thereby reducing learn- ing difficulty and enhancing generalizability.
  Concretely, we study how to tame the public Stable Video Diffusion (SVD) to
  predict reliable depth from input videos using a mixture of image depth and
  video depth datasets. We empirically confirm that a procedural training
  strategy - first optimizing the spatial layers of SVD and then optimizing the
  temporal layers while keeping the spatial layers frozen - yields the best
  results in terms of both spatial accuracy and temporal consistency. We further
  examine the sliding window strategy for inference on arbitrarily long videos.
  Our observations indicate a trade-off between efficiency and performance, with
  a one-frame overlap already producing favorable results. Extensive experimental
  results demonstrate the superiority of our approach, termed ChronoDepth, over
  existing alternatives, particularly in terms of the temporal consistency of the
  estimated depth. Additionally, we highlight the benefits of more consistent
  video depth in two practical applications: depth-conditioned video generation
  and novel view synthesis. Our project page is available at
  https://jhaoshao.github.io/ChronoDepth/{this http URL}.
paper_ai_keywords: video depth estimation cross-frame consistency conditional generation video generation models Stable Video Diffusion (SVD) image depth datasets video depth datasets procedural training strategy spatial layers temporal layers sliding window strategy depth-conditioned video generation novel view synthesis temporal consistency