title: Fast Encoder-Based 3D from Casual Videos via Point Track Processing
summary: This paper addresses the long-standing challenge of reconstructing 3D
  structures from videos with dynamic content. Current approaches to this problem
  were not designed to operate on casual videos recorded by standard cameras or
  require a long optimization time.
    Aiming to significantly improve the efficiency of previous approaches, we
  present TracksTo4D, a learning-based approach that enables inferring 3D
  structure and camera positions from dynamic content originating from casual
  videos using a single efficient feed-forward pass. To achieve this, we propose
  operating directly over 2D point tracks as input and designing an architecture
  tailored for processing 2D point tracks. Our proposed architecture is designed
  with two key principles in mind: (1) it takes into account the inherent
  symmetries present in the input point tracks data, and (2) it assumes that the
  movement patterns can be effectively represented using a low-rank
  approximation. TracksTo4D is trained in an unsupervised way on a dataset of
  casual videos utilizing only the 2D point tracks extracted from the videos,
  without any 3D supervision. Our experiments show that TracksTo4D can
  reconstruct a temporal point cloud and camera positions of the underlying video
  with accuracy comparable to state-of-the-art methods, while drastically
  reducing runtime by up to 95\%. We further show that TracksTo4D generalizes
  well to unseen videos of unseen semantic categories at inference time.
paper_authors._id: 67a07a4b605a6c919dea84ec
paper_authors.name: Yoni Kasten
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-03T12:59:44.973Z
paper_authors._id: 642294c112b4b51aae368d30
paper_authors.avatarUrl: /avatars/017c19910ba01d6bd9cd864132652448.svg
paper_authors.fullname: Yoni Kasten
paper_authors.type: user
paper_authors.user: ykasten
paper_authors._id: 67a07a4b605a6c919dea84ed
paper_authors.name: Wuyue Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-03T12:59:51.676Z
paper_authors._id: 65fbd280b0068def429d426f
paper_authors.avatarUrl: /avatars/1c9caccbb08ce3c9fa3bd60fecab10b5.svg
paper_authors.fullname: Wuyue Lu
paper_authors.type: user
paper_authors.user: Woo-wy
paper_authors._id: 67a07a4b605a6c919dea84ee
paper_authors.name: Haggai Maron
paper_title: Fast Encoder-Based 3D from Casual Videos via Point Track Processing
paper_summary: This paper addresses the long-standing challenge of reconstructing 3D
  structures from videos with dynamic content. Current approaches to this problem
  were not designed to operate on casual videos recorded by standard cameras or
  require a long optimization time.
    Aiming to significantly improve the efficiency of previous approaches, we
  present TracksTo4D, a learning-based approach that enables inferring 3D
  structure and camera positions from dynamic content originating from casual
  videos using a single efficient feed-forward pass. To achieve this, we propose
  operating directly over 2D point tracks as input and designing an architecture
  tailored for processing 2D point tracks. Our proposed architecture is designed
  with two key principles in mind: (1) it takes into account the inherent
  symmetries present in the input point tracks data, and (2) it assumes that the
  movement patterns can be effectively represented using a low-rank
  approximation. TracksTo4D is trained in an unsupervised way on a dataset of
  casual videos utilizing only the 2D point tracks extracted from the videos,
  without any 3D supervision. Our experiments show that TracksTo4D can
  reconstruct a temporal point cloud and camera positions of the underlying video
  with accuracy comparable to state-of-the-art methods, while drastically
  reducing runtime by up to 95\%. We further show that TracksTo4D generalizes
  well to unseen videos of unseen semantic categories at inference time.
paper_ai_keywords: TracksTo4D learning-based approach feed-forward pass 2D point tracks novel architecture low-rank approximation unsupervised training temporal cloud camera positions state-of-the-art methods