title: SpatialTracker: Tracking Any 2D Pixels in 3D Space
summary: Recovering dense and long-range pixel motion in videos is a challenging
  problem. Part of the difficulty arises from the 3D-to-2D projection process,
  leading to occlusions and discontinuities in the 2D motion domain. While 2D
  motion can be intricate, we posit that the underlying 3D motion can often be
  simple and low-dimensional. In this work, we propose to estimate point
  trajectories in 3D space to mitigate the issues caused by image projection. Our
  method, named SpatialTracker, lifts 2D pixels to 3D using monocular depth
  estimators, represents the 3D content of each frame efficiently using a
  triplane representation, and performs iterative updates using a transformer to
  estimate 3D trajectories. Tracking in 3D allows us to leverage
  as-rigid-as-possible (ARAP) constraints while simultaneously learning a
  rigidity embedding that clusters pixels into different rigid parts. Extensive
  evaluation shows that our approach achieves state-of-the-art tracking
  performance both qualitatively and quantitatively, particularly in challenging
  scenarios such as out-of-plane rotation.
paper_authors._id: 6614cb62a025df49c6a03b6f
paper_authors.name: Yuxi Xiao
paper_authors._id: 6614cb62a025df49c6a03b70
paper_authors.name: Qianqian Wang
paper_authors._id: 6614cb62a025df49c6a03b71
paper_authors.name: Shangzhan Zhang
paper_authors._id: 6614cb62a025df49c6a03b72
paper_authors.name: Nan Xue
paper_authors._id: 6614cb62a025df49c6a03b73
paper_authors.name: Sida Peng
paper_authors._id: 6614cb62a025df49c6a03b74
paper_authors.name: Yujun Shen
paper_authors._id: 6614cb62a025df49c6a03b75
paper_authors.name: Xiaowei Zhou
paper_title: SpatialTracker: Tracking Any 2D Pixels in 3D Space
paper_summary: Recovering dense and long-range pixel motion in videos is a challenging
  problem. Part of the difficulty arises from the 3D-to-2D projection process,
  leading to occlusions and discontinuities in the 2D motion domain. While 2D
  motion can be intricate, we posit that the underlying 3D motion can often be
  simple and low-dimensional. In this work, we propose to estimate point
  trajectories in 3D space to mitigate the issues caused by image projection. Our
  method, named SpatialTracker, lifts 2D pixels to 3D using monocular depth
  estimators, represents the 3D content of each frame efficiently using a
  triplane representation, and performs iterative updates using a transformer to
  estimate 3D trajectories. Tracking in 3D allows us to leverage
  as-rigid-as-possible (ARAP) constraints while simultaneously learning a
  rigidity embedding that clusters pixels into different rigid parts. Extensive
  evaluation shows that our approach achieves state-of-the-art tracking
  performance both qualitatively and quantitatively, particularly in challenging
  scenarios such as out-of-plane rotation.
paper_ai_summary: SpatialTracker estimates 3D point trajectories using monocular depth estimation and transformer updates, achieving superior video tracking performance with ARAP constraints and rigidity embeddings.
paper_ai_keywords: monocular depth estimators triplane representation transformer as-rigid-as-possible (ARAP) constraints rigidity embedding