title: TAPTRv2: Attention-based Position Update Improves Tracking Any Point
summary: In this paper, we present TAPTRv2, a Transformer-based approach built upon
  TAPTR for solving the Tracking Any Point (TAP) task. TAPTR borrows designs from
  DEtection TRansformer (DETR) and formulates each tracking point as a point
  query, making it possible to leverage well-studied operations in DETR-like
  algorithms. TAPTRv2 improves TAPTR by addressing a critical issue regarding its
  reliance on cost-volume,which contaminates the point query\'s content feature
  and negatively impacts both visibility prediction and cost-volume computation.
  In TAPTRv2, we propose a novel attention-based position update (APU) operation
  and use key-aware deformable attention to realize. For each query, this
  operation uses key-aware attention weights to combine their corresponding
  deformable sampling positions to predict a new query position. This design is
  based on the observation that local attention is essentially the same as
  cost-volume, both of which are computed by dot-production between a query and
  its surrounding features. By introducing this new operation, TAPTRv2 not only
  removes the extra burden of cost-volume computation, but also leads to a
  substantial performance improvement. TAPTRv2 surpasses TAPTR and achieves
  state-of-the-art performance on many challenging datasets, demonstrating the
  superiority
paper_authors._id: 66a85b0d95054e8a31b46b7c
paper_authors.name: Hongyang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-30T07:25:46.596Z
paper_authors._id: 6543de2aee7bbb595295c179
paper_authors.avatarUrl: /avatars/8d8a410900562fe9ce0b5bbd4ae6c55a.svg
paper_authors.fullname: HYeung Lee
paper_authors.type: user
paper_authors.user: HYeungLee
paper_authors._id: 66a85b0d95054e8a31b46b7d
paper_authors.name: Hao Zhang
paper_authors._id: 66a85b0d95054e8a31b46b7e
paper_authors.name: Shilong Liu
paper_authors._id: 66a85b0d95054e8a31b46b7f
paper_authors.name: Zhaoyang Zeng
paper_authors._id: 66a85b0d95054e8a31b46b80
paper_authors.name: Feng Li
paper_authors._id: 66a85b0d95054e8a31b46b81
paper_authors.name: Tianhe Ren
paper_authors._id: 66a85b0d95054e8a31b46b82
paper_authors.name: Bohan Li
paper_authors._id: 66a85b0d95054e8a31b46b83
paper_authors.name: Lei Zhang
paper_title: TAPTRv2: Attention-based Position Update Improves Tracking Any Point
paper_summary: In this paper, we present TAPTRv2, a Transformer-based approach built upon
  TAPTR for solving the Tracking Any Point (TAP) task. TAPTR borrows designs from
  DEtection TRansformer (DETR) and formulates each tracking point as a point
  query, making it possible to leverage well-studied operations in DETR-like
  algorithms. TAPTRv2 improves TAPTR by addressing a critical issue regarding its
  reliance on cost-volume,which contaminates the point query\'s content feature
  and negatively impacts both visibility prediction and cost-volume computation.
  In TAPTRv2, we propose a novel attention-based position update (APU) operation
  and use key-aware deformable attention to realize. For each query, this
  operation uses key-aware attention weights to combine their corresponding
  deformable sampling positions to predict a new query position. This design is
  based on the observation that local attention is essentially the same as
  cost-volume, both of which are computed by dot-production between a query and
  its surrounding features. By introducing this new operation, TAPTRv2 not only
  removes the extra burden of cost-volume computation, but also leads to a
  substantial performance improvement. TAPTRv2 surpasses TAPTR and achieves
  state-of-the-art performance on many challenging datasets, demonstrating the
  superiority
paper_ai_keywords: Transformer-based approach TAPTR TAPTRv2 DEtection TRansformer (DETR) point query cost-volume ground attention-based position update (APU) key-aware attention deformable sampling dot-production cost-volume computation state-of-the-art performance