TAPNext: Tracking Any Point (TAP) as Next Token Prediction
Tracking Any Point (TAP) in a video is a challenging computer vision problem
with many demonstrated applications in robotics, video editing, and 3D
reconstruction. Existing methods for TAP rely heavily on complex
tracking-specific inductive biases and heuristics, limiting their generality
and potential for scaling. To address these challenges, we present TAPNext, a
new approach that casts TAP as sequential masked token decoding. Our model is
causal, tracks in a purely online fashion, and removes tracking-specific
inductive biases. This enables TAPNext to run with minimal latency, and removes
the temporal windowing required by many existing state of art trackers. Despite
its simplicity, TAPNext achieves a new state-of-the-art tracking performance
among both online and offline trackers. Finally, we present evidence that many
widely used tracking heuristics emerge naturally in TAPNext through end-to-end
training.
67f8fccffe86522817d0fb03
Artem Zholus
admin_assigned
2025-04-11T12:05:00.728Z
6267cdd3f91d1c1633c08bbf
/avatars/51d7961098f52f39ee406009a12982b8.svg
Artem Zholus
user
artemZholus
67f8fccffe86522817d0fb04
Carl Doersch
67f8fccffe86522817d0fb05
Yi Yang
67f8fccffe86522817d0fb06
Skanda Koppula
admin_assigned
2025-04-11T12:07:20.239Z
65451016321f0393f453cd7b
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/XBmF81Uyj0N7lPclU1pUG.jpeg
Skanda Koppula
user
skoppula
67f8fccffe86522817d0fb07
Viorica Patraucean
67f8fccffe86522817d0fb08
Xu Owen He
67f8fccffe86522817d0fb09
Ignacio Rocco
67f8fccffe86522817d0fb0a
Mehdi S. M. Sajjadi
admin_assigned
2025-04-11T12:06:47.857Z
6733e8c6d7e101ba376800a7
/avatars/5ed957b5cb8e0d6be0f270ed53683622.svg
Mehdi Sajjadi
user
msajjadi
67f8fccffe86522817d0fb0b
Sarath Chandar
admin_assigned
2025-04-11T12:06:39.497Z
66fabb66c2bf89d75e8cdd4d
/avatars/c40f55a77e2fb34ba38a79f04df82893.svg
Sarath Chandar
user
apsarath
67f8fccffe86522817d0fb0c
Ross Goroshin
admin_assigned
2025-04-11T12:06:30.044Z
6491a50cfa736a2ad5fa91ee
/avatars/41f7c0bf41e45309d5a86899fa72454f.svg
Ross Goroshin
user
rgoroshin
TAPNext: Tracking Any Point (TAP) as Next Token Prediction
Tracking Any Point (TAP) in a video is a challenging computer vision problem
with many demonstrated applications in robotics, video editing, and 3D
reconstruction. Existing methods for TAP rely heavily on complex
tracking-specific inductive biases and heuristics, limiting their generality
and potential for scaling. To address these challenges, we present TAPNext, a
new approach that casts TAP as sequential masked token decoding. Our model is
causal, tracks in a purely online fashion, and removes tracking-specific
inductive biases. This enables TAPNext to run with minimal latency, and removes
the temporal windowing required by many existing state of art trackers. Despite
its simplicity, TAPNext achieves a new state-of-the-art tracking performance
among both online and offline trackers. Finally, we present evidence that many
widely used tracking heuristics emerge naturally in TAPNext through end-to-end
training.
TAPNext addresses video tracking by framing it as sequential masked token decoding, achieving state-of-the-art performance while eliminating tracking-specific biases.
sequential masked token decoding causal tracking end-to-end training