TAPVid-3D: A Benchmark for Tracking Any Point in 3D
<|reference_start|>TAPVid-3D: A Benchmark for Tracking Any Point in 3D: We introduce a new benchmark, TAPVid-3D, for evaluating the task of long-range Tracking Any Point in 3D (TAP-3D). While point tracking in two dimensions (TAP) has many benchmarks measuring performance on real-world videos, such as TAPVid-DAVIS, three-dimensional point tracking has none. To this end, leveraging existing footage, we build a new benchmark for 3D point tracking featuring 4,000+ real-world videos, composed of three different data sources spanning a variety of object types, motion patterns, and indoor and outdoor environments. To measure performance on the TAP-3D task, we formulate a collection of metrics that extend the Jaccard-based metric used in TAP to handle the complexities of ambiguous depth scales across models, occlusions, and multi-track spatio-temporal smoothness. We manually verify a large sample of trajectories to ensure correct video annotations, and assess the current state of the TAP-3D task by constructing competitive baselines using existing tracking models. We anticipate this benchmark will serve as a guidepost to improve our ability to understand precise 3D motion and surface deformation from monocular video. Code for dataset download, generation, and model evaluation is available at https://tapvid3d.github.io<|reference_end|>
@article{koppula2024tapvid-3d:,
    title={TAPVid-3D: A Benchmark for Tracking Any Point in 3D},
    author={Skanda Koppula, Ignacio Rocco, Yi Yang, Joe Heyward, Jo~ao Carreira,
  Andrew Zisserman, Gabriel Brostow, Carl Doersch},
    journal={arXiv preprint arXiv:2407.05921},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.05921},
    primaryClass={cs.CV cs.AI cs.LG}
}