TAPTRv3: Spatial and Temporal Context Foster Robust Tracking of Any
  Point in Long Video
In this paper, we present TAPTRv3, which is built upon TAPTRv2 to improve its
point tracking robustness in long videos. TAPTRv2 is a simple DETR-like
framework that can accurately track any point in real-world videos without
requiring cost-volume. TAPTRv3 improves TAPTRv2 by addressing its shortage in
querying high quality features from long videos, where the target tracking
points normally undergo increasing variation over time. In TAPTRv3, we propose
to utilize both spatial and temporal context to bring better feature querying
along the spatial and temporal dimensions for more robust tracking in long
videos. For better spatial feature querying, we present Context-aware
Cross-Attention (CCA), which leverages surrounding spatial context to enhance
the quality of attention scores when querying image features. For better
temporal feature querying, we introduce Visibility-aware Long-Temporal
Attention (VLTA) to conduct temporal attention to all past frames while
considering their corresponding visibilities, which effectively addresses the
feature drifting problem in TAPTRv2 brought by its RNN-like long-temporal
modeling. TAPTRv3 surpasses TAPTRv2 by a large margin on most of the
challenging datasets and obtains state-of-the-art performance. Even when
compared with methods trained with large-scale extra internal data, TAPTRv3 is
still competitive.
674e74e2b3b82188d2e57535
Jinyuan Qu
admin_assigned
2024-12-03T14:32:00.661Z
64c23a7639f458b91a27d0c4
/avatars/ab9059be0aa475ff45e0cbdd8a2e7e51.svg
QuJinyuan
user
JinyuanQu
674e74e2b3b82188d2e57536
Hongyang Li
admin_assigned
2024-12-03T14:32:23.009Z
6499b0184936457997180c90
/avatars/b8be7bfabf746639e30330f5f623f560.svg
Hongyang Li
user
compileme
674e74e2b3b82188d2e57537
Shilong Liu
admin_assigned
2024-12-03T14:32:29.516Z
638b13c0c1d591879698f4e2
https://cdn-avatars.huggingface.co/v1/production/uploads/638b13c0c1d591879698f4e2/X8X4EWMXuzhBpG62wO2xS.jpeg
Shilong Liu
user
ShilongLiu
674e74e2b3b82188d2e57538
Tianhe Ren
674e74e2b3b82188d2e57539
Zhaoyang Zeng
674e74e2b3b82188d2e5753a
Lei Zhang
TAPTRv3: Spatial and Temporal Context Foster Robust Tracking of Any
  Point in Long Video
In this paper, we present TAPTRv3, which is built upon TAPTRv2 to improve its
point tracking robustness in long videos. TAPTRv2 is a simple DETR-like
framework that can accurately track any point in real-world videos without
requiring cost-volume. TAPTRv3 improves TAPTRv2 by addressing its shortage in
querying high quality features from long videos, where the target tracking
points normally undergo increasing variation over time. In TAPTRv3, we propose
to utilize both spatial and temporal context to bring better feature querying
along the spatial and temporal dimensions for more robust tracking in long
videos. For better spatial feature querying, we present Context-aware
Cross-Attention (CCA), which leverages surrounding spatial context to enhance
the quality of attention scores when querying image features. For better
temporal feature querying, we introduce Visibility-aware Long-Temporal
Attention (VLTA) to conduct temporal attention to all past frames while
considering their corresponding visibilities, which effectively addresses the
feature drifting problem in TAPTRv2 brought by its RNN-like long-temporal
modeling. TAPTRv3 surpasses TAPTRv2 by a large margin on most of the
challenging datasets and obtains state-of-the-art performance. Even when
compared with methods trained with large-scale extra internal data, TAPTRv3 is
still competitive.
TAPTRv3 enhances point tracking robustness in long videos by integrating spatial and temporal context through Context-aware Cross-Attention and Visibility-aware Long-Temporal Attention, achieving superior performance compared to existing methods.
DETR CCA Context-aware Cross-Attention VLTA Visibility-aware Long-Temporal Attention RNN feature querying spatial context temporal attention feature drifting