TAPTRv2: Attention-based Position Update Improves Tracking Any Point
In this paper, we present TAPTRv2, a Transformer-based approach built upon
TAPTR for solving the Tracking Any Point (TAP) task. TAPTR borrows designs from
DEtection TRansformer (DETR) and formulates each tracking point as a point
query, making it possible to leverage well-studied operations in DETR-like
algorithms. TAPTRv2 improves TAPTR by addressing a critical issue regarding its
reliance on cost-volume,which contaminates the point query\'s content feature
and negatively impacts both visibility prediction and cost-volume computation.
In TAPTRv2, we propose a novel attention-based position update (APU) operation
and use key-aware deformable attention to realize. For each query, this
operation uses key-aware attention weights to combine their corresponding
deformable sampling positions to predict a new query position. This design is
based on the observation that local attention is essentially the same as
cost-volume, both of which are computed by dot-production between a query and
its surrounding features. By introducing this new operation, TAPTRv2 not only
removes the extra burden of cost-volume computation, but also leads to a
substantial performance improvement. TAPTRv2 surpasses TAPTR and achieves
state-of-the-art performance on many challenging datasets, demonstrating the
superiority
66a85b0d95054e8a31b46b7c
Hongyang Li
claimed_verified
2024-07-30T07:25:46.596Z
6543de2aee7bbb595295c179
/avatars/8d8a410900562fe9ce0b5bbd4ae6c55a.svg
HYeung Lee
user
HYeungLee
66a85b0d95054e8a31b46b7d
Hao Zhang
66a85b0d95054e8a31b46b7e
Shilong Liu
claimed_verified
2025-05-29T07:47:43.646Z
638b13c0c1d591879698f4e2
https://cdn-avatars.huggingface.co/v1/production/uploads/638b13c0c1d591879698f4e2/X8X4EWMXuzhBpG62wO2xS.jpeg
Shilong Liu
user
ShilongLiu
66a85b0d95054e8a31b46b7f
Zhaoyang Zeng
66a85b0d95054e8a31b46b80
Feng Li
66a85b0d95054e8a31b46b81
Tianhe Ren
66a85b0d95054e8a31b46b82
Bohan Li
66a85b0d95054e8a31b46b83
Lei Zhang
TAPTRv2: Attention-based Position Update Improves Tracking Any Point
In this paper, we present TAPTRv2, a Transformer-based approach built upon
TAPTR for solving the Tracking Any Point (TAP) task. TAPTR borrows designs from
DEtection TRansformer (DETR) and formulates each tracking point as a point
query, making it possible to leverage well-studied operations in DETR-like
algorithms. TAPTRv2 improves TAPTR by addressing a critical issue regarding its
reliance on cost-volume,which contaminates the point query\'s content feature
and negatively impacts both visibility prediction and cost-volume computation.
In TAPTRv2, we propose a novel attention-based position update (APU) operation
and use key-aware deformable attention to realize. For each query, this
operation uses key-aware attention weights to combine their corresponding
deformable sampling positions to predict a new query position. This design is
based on the observation that local attention is essentially the same as
cost-volume, both of which are computed by dot-production between a query and
its surrounding features. By introducing this new operation, TAPTRv2 not only
removes the extra burden of cost-volume computation, but also leads to a
substantial performance improvement. TAPTRv2 surpasses TAPTR and achieves
state-of-the-art performance on many challenging datasets, demonstrating the
superiority
TAPTRv2 introduces a novel attention-based position update operation using key-aware deformable attention, eliminating the need for cost-volume computation and achieving state-of-the-art performance in point tracking.
Transformer-based approach DEtection TRansformer (DETR) point query attention-based position update (APU) key-aware deformable attention cost-volume dot-production