TAPTRv2: Attention-based Position Update Improves Tracking Any Point
<|reference_start|>TAPTRv2: Attention-based Position Update Improves Tracking Any Point: In this paper, we present TAPTRv2, a Transformer-based approach built upon TAPTR for solving the Tracking Any Point (TAP) task. TAPTR borrows designs from DEtection TRansformer (DETR) and formulates each tracking point as a point query, making it possible to leverage well-studied operations in DETR-like algorithms. TAPTRv2 improves TAPTR by addressing a critical issue regarding its reliance on cost-volume,which contaminates the point query\'s content feature and negatively impacts both visibility prediction and cost-volume computation. In TAPTRv2, we propose a novel attention-based position update (APU) operation and use key-aware deformable attention to realize. For each query, this operation uses key-aware attention weights to combine their corresponding deformable sampling positions to predict a new query position. This design is based on the observation that local attention is essentially the same as cost-volume, both of which are computed by dot-production between a query and its surrounding features. By introducing this new operation, TAPTRv2 not only removes the extra burden of cost-volume computation, but also leads to a substantial performance improvement. TAPTRv2 surpasses TAPTR and achieves state-of-the-art performance on many challenging datasets, demonstrating the superiority<|reference_end|>
@article{li2024taptrv2:,
    title={TAPTRv2: Attention-based Position Update Improves Tracking Any Point},
    author={Hongyang Li, Hao Zhang, Shilong Liu, Zhaoyang Zeng, Feng Li, Tianhe
  Ren, Bohan Li, Lei Zhang},
    journal={arXiv preprint arXiv:2407.16291},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.16291},
    primaryClass={cs.CV cs.RO}
}