Fast Encoder-Based 3D from Casual Videos via Point Track Processing
This paper addresses the long-standing challenge of reconstructing 3D
structures from videos with dynamic content. Current approaches to this problem
were not designed to operate on casual videos recorded by standard cameras or
require a long optimization time.
  Aiming to significantly improve the efficiency of previous approaches, we
present TracksTo4D, a learning-based approach that enables inferring 3D
structure and camera positions from dynamic content originating from casual
videos using a single efficient feed-forward pass. To achieve this, we propose
operating directly over 2D point tracks as input and designing an architecture
tailored for processing 2D point tracks. Our proposed architecture is designed
with two key principles in mind: (1) it takes into account the inherent
symmetries present in the input point tracks data, and (2) it assumes that the
movement patterns can be effectively represented using a low-rank
approximation. TracksTo4D is trained in an unsupervised way on a dataset of
casual videos utilizing only the 2D point tracks extracted from the videos,
without any 3D supervision. Our experiments show that TracksTo4D can
reconstruct a temporal point cloud and camera positions of the underlying video
with accuracy comparable to state-of-the-art methods, while drastically
reducing runtime by up to 95\%. We further show that TracksTo4D generalizes
well to unseen videos of unseen semantic categories at inference time.
67a07a4b605a6c919dea84ec
Yoni Kasten
admin_assigned
2025-02-03T12:59:44.973Z
642294c112b4b51aae368d30
/avatars/017c19910ba01d6bd9cd864132652448.svg
Yoni Kasten
user
ykasten
67a07a4b605a6c919dea84ed
Wuyue Lu
admin_assigned
2025-02-03T12:59:51.676Z
65fbd280b0068def429d426f
/avatars/1c9caccbb08ce3c9fa3bd60fecab10b5.svg
Wuyue Lu
user
Woo-wy
67a07a4b605a6c919dea84ee
Haggai Maron
Fast Encoder-Based 3D from Casual Videos via Point Track Processing
This paper addresses the long-standing challenge of reconstructing 3D
structures from videos with dynamic content. Current approaches to this problem
were not designed to operate on casual videos recorded by standard cameras or
require a long optimization time.
  Aiming to significantly improve the efficiency of previous approaches, we
present TracksTo4D, a learning-based approach that enables inferring 3D
structure and camera positions from dynamic content originating from casual
videos using a single efficient feed-forward pass. To achieve this, we propose
operating directly over 2D point tracks as input and designing an architecture
tailored for processing 2D point tracks. Our proposed architecture is designed
with two key principles in mind: (1) it takes into account the inherent
symmetries present in the input point tracks data, and (2) it assumes that the
movement patterns can be effectively represented using a low-rank
approximation. TracksTo4D is trained in an unsupervised way on a dataset of
casual videos utilizing only the 2D point tracks extracted from the videos,
without any 3D supervision. Our experiments show that TracksTo4D can
reconstruct a temporal point cloud and camera positions of the underlying video
with accuracy comparable to state-of-the-art methods, while drastically
reducing runtime by up to 95\%. We further show that TracksTo4D generalizes
well to unseen videos of unseen semantic categories at inference time.
TracksTo4D is a learning-based method that efficiently reconstructs 3D structures and camera positions from casual videos using 2D point tracks, reducing runtime significantly compared to existing approaches.
TracksTo4D 3D reconstruction dynamic content casual videos feed-forward pass 2D point tracks low-rank approximation unsupervised learning temporal point cloud semantic categories