title: TAPTRv3: Spatial and Temporal Context Foster Robust Tracking of Any
    Point in Long Video
summary: In this paper, we present TAPTRv3, which is built upon TAPTRv2 to improve its
  point tracking robustness in long videos. TAPTRv2 is a simple DETR-like
  framework that can accurately track any point in real-world videos without
  requiring cost-volume. TAPTRv3 improves TAPTRv2 by addressing its shortage in
  querying high quality features from long videos, where the target tracking
  points normally undergo increasing variation over time. In TAPTRv3, we propose
  to utilize both spatial and temporal context to bring better feature querying
  along the spatial and temporal dimensions for more robust tracking in long
  videos. For better spatial feature querying, we present Context-aware
  Cross-Attention (CCA), which leverages surrounding spatial context to enhance
  the quality of attention scores when querying image features. For better
  temporal feature querying, we introduce Visibility-aware Long-Temporal
  Attention (VLTA) to conduct temporal attention to all past frames while
  considering their corresponding visibilities, which effectively addresses the
  feature drifting problem in TAPTRv2 brought by its RNN-like long-temporal
  modeling. TAPTRv3 surpasses TAPTRv2 by a large margin on most of the
  challenging datasets and obtains state-of-the-art performance. Even when
  compared with methods trained with large-scale extra internal data, TAPTRv3 is
  still competitive.
paper_authors._id: 674e74e2b3b82188d2e57535
paper_authors.name: Jinyuan Qu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T14:32:00.661Z
paper_authors._id: 64c23a7639f458b91a27d0c4
paper_authors.avatarUrl: /avatars/ab9059be0aa475ff45e0cbdd8a2e7e51.svg
paper_authors.fullname: QuJinyuan
paper_authors.type: user
paper_authors.user: JinyuanQu
paper_authors._id: 674e74e2b3b82188d2e57536
paper_authors.name: Hongyang Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T14:32:23.009Z
paper_authors._id: 6499b0184936457997180c90
paper_authors.avatarUrl: /avatars/b8be7bfabf746639e30330f5f623f560.svg
paper_authors.fullname: Hongyang Li
paper_authors.type: user
paper_authors.user: compileme
paper_authors._id: 674e74e2b3b82188d2e57537
paper_authors.name: Shilong Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T14:32:29.516Z
paper_authors._id: 638b13c0c1d591879698f4e2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/638b13c0c1d591879698f4e2/X8X4EWMXuzhBpG62wO2xS.jpeg
paper_authors.fullname: Shilong Liu
paper_authors.type: user
paper_authors.user: ShilongLiu
paper_authors._id: 674e74e2b3b82188d2e57538
paper_authors.name: Tianhe Ren
paper_authors._id: 674e74e2b3b82188d2e57539
paper_authors.name: Zhaoyang Zeng
paper_authors._id: 674e74e2b3b82188d2e5753a
paper_authors.name: Lei Zhang
paper_title: TAPTRv3: Spatial and Temporal Context Foster Robust Tracking of Any
    Point in Long Video
paper_summary: In this paper, we present TAPTRv3, which is built upon TAPTRv2 to improve its
  point tracking robustness in long videos. TAPTRv2 is a simple DETR-like
  framework that can accurately track any point in real-world videos without
  requiring cost-volume. TAPTRv3 improves TAPTRv2 by addressing its shortage in
  querying high quality features from long videos, where the target tracking
  points normally undergo increasing variation over time. In TAPTRv3, we propose
  to utilize both spatial and temporal context to bring better feature querying
  along the spatial and temporal dimensions for more robust tracking in long
  videos. For better spatial feature querying, we present Context-aware
  Cross-Attention (CCA), which leverages surrounding spatial context to enhance
  the quality of attention scores when querying image features. For better
  temporal feature querying, we introduce Visibility-aware Long-Temporal
  Attention (VLTA) to conduct temporal attention to all past frames while
  considering their corresponding visibilities, which effectively addresses the
  feature drifting problem in TAPTRv2 brought by its RNN-like long-temporal
  modeling. TAPTRv3 surpasses TAPTRv2 by a large margin on most of the
  challenging datasets and obtains state-of-the-art performance. Even when
  compared with methods trained with large-scale extra internal data, TAPTRv3 is
  still competitive.
paper_ai_keywords: DETR-like point tracking feature querying Context-aware Cross-Attention (CCA) Visibility-aware Long-Temporal Attention (VLTA) spatial feature querying temporal feature querying attention scores feature drifting RNN-like long-temporal modeling