Learning Temporally Consistent Video Depth from Video Diffusion Priors
This work addresses the challenge of video depth estimation, which expects
not only per-frame accuracy but, more importantly, cross-frame consistency.
Instead of directly developing a depth estimator from scratch, we reformulate
the prediction task into a conditional generation problem. This allows us to
leverage the prior knowledge embedded in existing video generation models,
thereby reducing learn- ing difficulty and enhancing generalizability.
Concretely, we study how to tame the public Stable Video Diffusion (SVD) to
predict reliable depth from input videos using a mixture of image depth and
video depth datasets. We empirically confirm that a procedural training
strategy - first optimizing the spatial layers of SVD and then optimizing the
temporal layers while keeping the spatial layers frozen - yields the best
results in terms of both spatial accuracy and temporal consistency. We further
examine the sliding window strategy for inference on arbitrarily long videos.
Our observations indicate a trade-off between efficiency and performance, with
a one-frame overlap already producing favorable results. Extensive experimental
results demonstrate the superiority of our approach, termed ChronoDepth, over
existing alternatives, particularly in terms of the temporal consistency of the
estimated depth. Additionally, we highlight the benefits of more consistent
video depth in two practical applications: depth-conditioned video generation
and novel view synthesis. Our project page is available at
https://jhaoshao.github.io/ChronoDepth/{this http URL}.
665e9487f4f7557684cbaaa9
Jiahao Shao
admin_assigned
2024-06-04T08:08:01.319Z
644a16eded295eb43e6815e1
/avatars/0ee0f8c649247c1c618111899f0a577d.svg
Jiahao Shao
user
jhshao
665e9487f4f7557684cbaaaa
Yuanbo Yang
admin_assigned
2024-06-04T08:08:14.007Z
65d5f8aa933e3da6e7d5d627
/avatars/b82ea295bf07f9060e781f07deebba31.svg
yangyuanbo
user
yangyuanbo
665e9487f4f7557684cbaaab
Hongyu Zhou
claimed_verified
2024-12-03T11:17:15.837Z
664d9d1eafe6e8c3a986ef67
/avatars/88535076724e97f412ecd362a38904db.svg
hyzhou
user
hyzhou404
665e9487f4f7557684cbaaac
Youmin Zhang
admin_assigned
2024-06-04T08:09:06.777Z
65e054827409351736166d73
/avatars/d56b8af16b47cdac7a47d7bc0563687a.svg
Youmin Zhang
user
Rock-youmi
665e9487f4f7557684cbaaad
Yujun Shen
665e9487f4f7557684cbaaae
Matteo Poggi
admin_assigned
2024-06-04T08:09:20.630Z
6613b33d1dc41eabb76b8d82
/avatars/5c31d071c53dee4c19fcceb5b437c130.svg
Matteo Poggi
user
mpoggi
665e9487f4f7557684cbaaaf
Yiyi Liao
Learning Temporally Consistent Video Depth from Video Diffusion Priors
This work addresses the challenge of video depth estimation, which expects
not only per-frame accuracy but, more importantly, cross-frame consistency.
Instead of directly developing a depth estimator from scratch, we reformulate
the prediction task into a conditional generation problem. This allows us to
leverage the prior knowledge embedded in existing video generation models,
thereby reducing learn- ing difficulty and enhancing generalizability.
Concretely, we study how to tame the public Stable Video Diffusion (SVD) to
predict reliable depth from input videos using a mixture of image depth and
video depth datasets. We empirically confirm that a procedural training
strategy - first optimizing the spatial layers of SVD and then optimizing the
temporal layers while keeping the spatial layers frozen - yields the best
results in terms of both spatial accuracy and temporal consistency. We further
examine the sliding window strategy for inference on arbitrarily long videos.
Our observations indicate a trade-off between efficiency and performance, with
a one-frame overlap already producing favorable results. Extensive experimental
results demonstrate the superiority of our approach, termed ChronoDepth, over
existing alternatives, particularly in terms of the temporal consistency of the
estimated depth. Additionally, we highlight the benefits of more consistent
video depth in two practical applications: depth-conditioned video generation
and novel view synthesis. Our project page is available at
https://jhaoshao.github.io/ChronoDepth/{this http URL}.
A conditional generation approach using a video diffusion model optimizes spatial and temporal layers for improved depth estimation consistency across frames.
video depth estimation per-frame accuracy cross-frame consistency conditional generation Stable Video Diffusion (SVD) image depth video depth datasets procedural training strategy spatial layers temporal layers sliding window strategy depth-conditioned video generation novel view synthesis ChronoDepth