title: TAPIP3D: Tracking Any Point in Persistent 3D Geometry
summary: We introduce TAPIP3D, a novel approach for long-term 3D point tracking in
  monocular RGB and RGB-D videos. TAPIP3D represents videos as camera-stabilized
  spatio-temporal feature clouds, leveraging depth and camera motion information
  to lift 2D video features into a 3D world space where camera motion is
  effectively canceled. TAPIP3D iteratively refines multi-frame 3D motion
  estimates within this stabilized representation, enabling robust tracking over
  extended periods. To manage the inherent irregularities of 3D point
  distributions, we propose a Local Pair Attention mechanism. This 3D
  contextualization strategy effectively exploits spatial relationships in 3D,
  forming informative feature neighborhoods for precise 3D trajectory estimation.
  Our 3D-centric approach significantly outperforms existing 3D point tracking
  methods and even enhances 2D tracking accuracy compared to conventional 2D
  pixel trackers when accurate depth is available. It supports inference in both
  camera coordinates (i.e., unstabilized) and world coordinates, and our results
  demonstrate that compensating for camera motion improves tracking performance.
  Our approach replaces the conventional 2D square correlation neighborhoods used
  in prior 2D and 3D trackers, leading to more robust and accurate results across
  various 3D point tracking benchmarks. Project Page: https://tapip3d.github.io
paper_authors._id: 680720e066b60d551b653f1b
paper_authors.name: Bowei Zhang
paper_authors._id: 680720e066b60d551b653f1c
paper_authors.name: Lei Ke
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-22T11:05:00.815Z
paper_authors._id: 6258a6455ea3a0a9b6de3f22
paper_authors.avatarUrl: /avatars/6eeed72a97fb24465e5e65583fbe50cf.svg
paper_authors.fullname: Lei Ke
paper_authors.type: user
paper_authors.user: lkeab
paper_authors._id: 680720e066b60d551b653f1d
paper_authors.name: Adam W. Harley
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-22T11:04:46.261Z
paper_authors._id: 6314ff478be84e1ab412155a
paper_authors.avatarUrl: /avatars/5a0c0aedc99aac6731e0652faf98b2e6.svg
paper_authors.fullname: Adam W Harley
paper_authors.type: user
paper_authors.user: aharley
paper_authors._id: 680720e066b60d551b653f1e
paper_authors.name: Katerina Fragkiadaki
paper_title: TAPIP3D: Tracking Any Point in Persistent 3D Geometry
paper_summary: We introduce TAPIP3D, a novel approach for long-term 3D point tracking in
  monocular RGB and RGB-D videos. TAPIP3D represents videos as camera-stabilized
  spatio-temporal feature clouds, leveraging depth and camera motion information
  to lift 2D video features into a 3D world space where camera motion is
  effectively canceled. TAPIP3D iteratively refines multi-frame 3D motion
  estimates within this stabilized representation, enabling robust tracking over
  extended periods. To manage the inherent irregularities of 3D point
  distributions, we propose a Local Pair Attention mechanism. This 3D
  contextualization strategy effectively exploits spatial relationships in 3D,
  forming informative feature neighborhoods for precise 3D trajectory estimation.
  Our 3D-centric approach significantly outperforms existing 3D point tracking
  methods and even enhances 2D tracking accuracy compared to conventional 2D
  pixel trackers when accurate depth is available. It supports inference in both
  camera coordinates (i.e., unstabilized) and world coordinates, and our results
  demonstrate that compensating for camera motion improves tracking performance.
  Our approach replaces the conventional 2D square correlation neighborhoods used
  in prior 2D and 3D trackers, leading to more robust and accurate results across
  various 3D point tracking benchmarks. Project Page: https://tapip3d.github.io
paper_ai_summary: A novel 3D point tracking method called TAPIP3D stabilizes video features into a 3D space using depth and motion data, refining 3D trajectories and surpassing 2D and conventional 3D tracking methods.
paper_ai_keywords: camera-stabilized spatio-temporal feature clouds depth information 3D world space camera motion multi-frame 3D motion estimates Local Pair Attention 3D contextualization 3D trajectory estimation 3D-centric approach 2D pixel trackers correlation neighborhoods