title: TAPNext: Tracking Any Point (TAP) as Next Token Prediction
summary: Tracking Any Point (TAP) in a video is a challenging computer vision problem
  with many demonstrated applications in robotics, video editing, and 3D
  reconstruction. Existing methods for TAP rely heavily on complex
  tracking-specific inductive biases and heuristics, limiting their generality
  and potential for scaling. To address these challenges, we present TAPNext, a
  new approach that casts TAP as sequential masked token decoding. Our model is
  causal, tracks in a purely online fashion, and removes tracking-specific
  inductive biases. This enables TAPNext to run with minimal latency, and removes
  the temporal windowing required by many existing state of art trackers. Despite
  its simplicity, TAPNext achieves a new state-of-the-art tracking performance
  among both online and offline trackers. Finally, we present evidence that many
  widely used tracking heuristics emerge naturally in TAPNext through end-to-end
  training.
paper_authors._id: 67f8fccffe86522817d0fb03
paper_authors.name: Artem Zholus
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-11T12:05:00.728Z
paper_authors._id: 6267cdd3f91d1c1633c08bbf
paper_authors.avatarUrl: /avatars/51d7961098f52f39ee406009a12982b8.svg
paper_authors.fullname: Artem Zholus
paper_authors.type: user
paper_authors.user: artemZholus
paper_authors._id: 67f8fccffe86522817d0fb04
paper_authors.name: Carl Doersch
paper_authors._id: 67f8fccffe86522817d0fb05
paper_authors.name: Yi Yang
paper_authors._id: 67f8fccffe86522817d0fb06
paper_authors.name: Skanda Koppula
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-11T12:07:20.239Z
paper_authors._id: 65451016321f0393f453cd7b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/XBmF81Uyj0N7lPclU1pUG.jpeg
paper_authors.fullname: Skanda Koppula
paper_authors.type: user
paper_authors.user: skoppula
paper_authors._id: 67f8fccffe86522817d0fb07
paper_authors.name: Viorica Patraucean
paper_authors._id: 67f8fccffe86522817d0fb08
paper_authors.name: Xu Owen He
paper_authors._id: 67f8fccffe86522817d0fb09
paper_authors.name: Ignacio Rocco
paper_authors._id: 67f8fccffe86522817d0fb0a
paper_authors.name: Mehdi S. M. Sajjadi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-11T12:06:47.857Z
paper_authors._id: 6733e8c6d7e101ba376800a7
paper_authors.avatarUrl: /avatars/5ed957b5cb8e0d6be0f270ed53683622.svg
paper_authors.fullname: Mehdi Sajjadi
paper_authors.type: user
paper_authors.user: msajjadi
paper_authors._id: 67f8fccffe86522817d0fb0b
paper_authors.name: Sarath Chandar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-11T12:06:39.497Z
paper_authors._id: 66fabb66c2bf89d75e8cdd4d
paper_authors.avatarUrl: /avatars/c40f55a77e2fb34ba38a79f04df82893.svg
paper_authors.fullname: Sarath Chandar
paper_authors.type: user
paper_authors.user: apsarath
paper_authors._id: 67f8fccffe86522817d0fb0c
paper_authors.name: Ross Goroshin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-11T12:06:30.044Z
paper_authors._id: 6491a50cfa736a2ad5fa91ee
paper_authors.avatarUrl: /avatars/41f7c0bf41e45309d5a86899fa72454f.svg
paper_authors.fullname: Ross Goroshin
paper_authors.type: user
paper_authors.user: rgoroshin
paper_title: TAPNext: Tracking Any Point (TAP) as Next Token Prediction
paper_summary: Tracking Any Point (TAP) in a video is a challenging computer vision problem
  with many demonstrated applications in robotics, video editing, and 3D
  reconstruction. Existing methods for TAP rely heavily on complex
  tracking-specific inductive biases and heuristics, limiting their generality
  and potential for scaling. To address these challenges, we present TAPNext, a
  new approach that casts TAP as sequential masked token decoding. Our model is
  causal, tracks in a purely online fashion, and removes tracking-specific
  inductive biases. This enables TAPNext to run with minimal latency, and removes
  the temporal windowing required by many existing state of art trackers. Despite
  its simplicity, TAPNext achieves a new state-of-the-art tracking performance
  among both online and offline trackers. Finally, we present evidence that many
  widely used tracking heuristics emerge naturally in TAPNext through end-to-end
  training.
paper_ai_summary: TAPNext addresses video tracking by framing it as sequential masked token decoding, achieving state-of-the-art performance while eliminating tracking-specific biases.
paper_ai_keywords: sequential masked token decoding causal tracking end-to-end training