title: Shape of Motion: 4D Reconstruction from a Single Video
summary: Monocular dynamic reconstruction is a challenging and long-standing vision
  problem due to the highly ill-posed nature of the task. Existing approaches are
  limited in that they either depend on templates, are effective only in
  quasi-static scenes, or fail to model 3D motion explicitly. In this work, we
  introduce a method capable of reconstructing generic dynamic scenes, featuring
  explicit, full-sequence-long 3D motion, from casually captured monocular
  videos. We tackle the under-constrained nature of the problem with two key
  insights: First, we exploit the low-dimensional structure of 3D motion by
  representing scene motion with a compact set of SE3 motion bases. Each point's
  motion is expressed as a linear combination of these bases, facilitating soft
  decomposition of the scene into multiple rigidly-moving groups. Second, we
  utilize a comprehensive set of data-driven priors, including monocular depth
  maps and long-range 2D tracks, and devise a method to effectively consolidate
  these noisy supervisory signals, resulting in a globally consistent
  representation of the dynamic scene. Experiments show that our method achieves
  state-of-the-art performance for both long-range 3D/2D motion estimation and
  novel view synthesis on dynamic scenes. Project Page:
  https://shape-of-motion.github.io/
paper_authors._id: 6699ebc9f1356d2d964799b5
paper_authors.name: Qianqian Wang
paper_authors._id: 6699ebc9f1356d2d964799b6
paper_authors.name: Vickie Ye
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-12T15:55:00.251Z
paper_authors._id: 66e2f933fa7c50759b060f2f
paper_authors.avatarUrl: /avatars/9394f37459ab2da1c2f865f1a3cdd739.svg
paper_authors.fullname: Vickie Ye
paper_authors.type: user
paper_authors.user: vickieye
paper_authors._id: 6699ebc9f1356d2d964799b7
paper_authors.name: Hang Gao
paper_authors._id: 6699ebc9f1356d2d964799b8
paper_authors.name: Jake Austin
paper_authors._id: 6699ebc9f1356d2d964799b9
paper_authors.name: Zhengqi Li
paper_authors._id: 6699ebc9f1356d2d964799ba
paper_authors.name: Angjoo Kanazawa
paper_title: Shape of Motion: 4D Reconstruction from a Single Video
paper_summary: Monocular dynamic reconstruction is a challenging and long-standing vision
  problem due to the highly ill-posed nature of the task. Existing approaches are
  limited in that they either depend on templates, are effective only in
  quasi-static scenes, or fail to model 3D motion explicitly. In this work, we
  introduce a method capable of reconstructing generic dynamic scenes, featuring
  explicit, full-sequence-long 3D motion, from casually captured monocular
  videos. We tackle the under-constrained nature of the problem with two key
  insights: First, we exploit the low-dimensional structure of 3D motion by
  representing scene motion with a compact set of SE3 motion bases. Each point's
  motion is expressed as a linear combination of these bases, facilitating soft
  decomposition of the scene into multiple rigidly-moving groups. Second, we
  utilize a comprehensive set of data-driven priors, including monocular depth
  maps and long-range 2D tracks, and devise a method to effectively consolidate
  these noisy supervisory signals, resulting in a globally consistent
  representation of the dynamic scene. Experiments show that our method achieves
  state-of-the-art performance for both long-range 3D/2D motion estimation and
  novel view synthesis on dynamic scenes. Project Page:
  https://shape-of-motion.github.io/
paper_ai_keywords: SE3 motion bases rigidly-moving groups monocular depth maps long-range 2D tracks globally consistent representation novel view synthesis