title: DepthCrafter: Generating Consistent Long Depth Sequences for Open-world
    Videos
summary: Despite significant advancements in monocular depth estimation for static
  images, estimating video depth in the open world remains challenging, since
  open-world videos are extremely diverse in content, motion, camera movement,
  and length. We present DepthCrafter, an innovative method for generating
  temporally consistent long depth sequences with intricate details for
  open-world videos, without requiring any supplementary information such as
  camera poses or optical flow. DepthCrafter achieves generalization ability to
  open-world videos by training a video-to-depth model from a pre-trained
  image-to-video diffusion model, through our meticulously designed three-stage
  training strategy with the compiled paired video-depth datasets. Our training
  approach enables the model to generate depth sequences with variable lengths at
  one time, up to 110 frames, and harvest both precise depth details and rich
  content diversity from realistic and synthetic datasets. We also propose an
  inference strategy that processes extremely long videos through segment-wise
  estimation and seamless stitching. Comprehensive evaluations on multiple
  datasets reveal that DepthCrafter achieves state-of-the-art performance in
  open-world video depth estimation under zero-shot settings. Furthermore,
  DepthCrafter facilitates various downstream applications, including depth-based
  visual effects and conditional video generation.
paper_authors._id: 66d7f2770b84833536d80d7d
paper_authors.name: Wenbo Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-04T12:00:48.078Z
paper_authors._id: 657a7458afbb0117ba15c59f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/657a7458afbb0117ba15c59f/8_iwTS1UG_mKnfylFbLsY.jpeg
paper_authors.fullname: Wenbo Hu
paper_authors.type: user
paper_authors.user: wbhu-tc
paper_authors._id: 66d7f2770b84833536d80d7e
paper_authors.name: Xiangjun Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T08:45:38.421Z
paper_authors._id: 64c0953a8137192a1e2474dc
paper_authors.avatarUrl: /avatars/546405a7eaf2f60ad108ceaa0dda7d08.svg
paper_authors.fullname: xiangjun gao
paper_authors.type: user
paper_authors.user: xiangjun0211
paper_authors._id: 66d7f2770b84833536d80d7f
paper_authors.name: Xiaoyu Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-04T09:30:49.559Z
paper_authors._id: 66d8284b6bddfb32e77ddafb
paper_authors.avatarUrl: /avatars/edd371ecef6e7d58b945a30bbc6095ee.svg
paper_authors.fullname: Xiaoyu Li
paper_authors.type: user
paper_authors.user: Xiaoyu521
paper_authors._id: 66d7f2770b84833536d80d80
paper_authors.name: Sijie Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T08:45:13.134Z
paper_authors._id: 652cf6bad78452c474434fa6
paper_authors.avatarUrl: /avatars/106bd7a6f8af3ae3b079a0533f6fb619.svg
paper_authors.fullname: Sijie Zhao
paper_authors.type: user
paper_authors.user: walkingshadow
paper_authors._id: 66d7f2770b84833536d80d81
paper_authors.name: Xiaodong Cun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T08:43:56.736Z
paper_authors._id: 63184c517ca1b876d99b7e0e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63184c517ca1b876d99b7e0e/b-qDExoeJuDXK0cJBZKnz.jpeg
paper_authors.fullname: Xiaodong Cun
paper_authors.type: user
paper_authors.user: vinthony
paper_authors._id: 66d7f2770b84833536d80d82
paper_authors.name: Yong Zhang
paper_authors._id: 66d7f2770b84833536d80d83
paper_authors.name: Long Quan
paper_authors._id: 66d7f2770b84833536d80d84
paper_authors.name: Ying Shan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T08:43:36.253Z
paper_authors._id: 63ca3ddc04c979828310bfcb
paper_authors.avatarUrl: /avatars/615e0d8622950b4408b40d550f02a894.svg
paper_authors.fullname: Ying Shan
paper_authors.type: user
paper_authors.user: yshan2u
paper_title: DepthCrafter: Generating Consistent Long Depth Sequences for Open-world
    Videos
paper_summary: Despite significant advancements in monocular depth estimation for static
  images, estimating video depth in the open world remains challenging, since
  open-world videos are extremely diverse in content, motion, camera movement,
  and length. We present DepthCrafter, an innovative method for generating
  temporally consistent long depth sequences with intricate details for
  open-world videos, without requiring any supplementary information such as
  camera poses or optical flow. DepthCrafter achieves generalization ability to
  open-world videos by training a video-to-depth model from a pre-trained
  image-to-video diffusion model, through our meticulously designed three-stage
  training strategy with the compiled paired video-depth datasets. Our training
  approach enables the model to generate depth sequences with variable lengths at
  one time, up to 110 frames, and harvest both precise depth details and rich
  content diversity from realistic and synthetic datasets. We also propose an
  inference strategy that processes extremely long videos through segment-wise
  estimation and seamless stitching. Comprehensive evaluations on multiple
  datasets reveal that DepthCrafter achieves state-of-the-art performance in
  open-world video depth estimation under zero-shot settings. Furthermore,
  DepthCrafter facilitates various downstream applications, including depth-based
  visual effects and conditional video generation.
paper_ai_summary: DepthCrafter generates temporally consistent depth sequences for open-world videos using a video-to-depth model derived from a pre-trained image-to-video diffusion model.
paper_ai_keywords: depth estimation open-world videos video-to-depth model image-to-video diffusion model three-stage training strategy paired video-depth datasets segment-wise estimation seamless stitching depth-based visual effects conditional video generation