title: Self-Supervised Any-Point Tracking by Contrastive Random Walks
summary: We present a simple, self-supervised approach to the Tracking Any Point (TAP)
  problem. We train a global matching transformer to find cycle consistent tracks
  through video via contrastive random walks, using the transformer's
  attention-based global matching to define the transition matrices for a random
  walk on a space-time graph. The ability to perform "all pairs" comparisons
  between points allows the model to obtain high spatial precision and to obtain
  a strong contrastive learning signal, while avoiding many of the complexities
  of recent approaches (such as coarse-to-fine matching). To do this, we propose
  a number of design decisions that allow global matching architectures to be
  trained through self-supervision using cycle consistency. For example, we
  identify that transformer-based methods are sensitive to shortcut solutions,
  and propose a data augmentation scheme to address them. Our method achieves
  strong performance on the TapVid benchmarks, outperforming previous
  self-supervised tracking methods, such as DIFT, and is competitive with several
  supervised methods.
paper_authors._id: 66f4cf4c8fa8e9549199f8ae
paper_authors.name: Ayush Shrivastava
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-26T08:23:02.260Z
paper_authors._id: 66f41c37d8de0f55059d3317
paper_authors.avatarUrl: /avatars/a730ff35ea1da9a2da79aa56617e82d8.svg
paper_authors.fullname: Ayush Shrivastava
paper_authors.type: user
paper_authors.user: ayshrv
paper_authors._id: 66f4cf4c8fa8e9549199f8af
paper_authors.name: Andrew Owens
paper_title: Self-Supervised Any-Point Tracking by Contrastive Random Walks
paper_summary: We present a simple, self-supervised approach to the Tracking Any Point (TAP)
  problem. We train a global matching transformer to find cycle consistent tracks
  through video via contrastive random walks, using the transformer's
  attention-based global matching to define the transition matrices for a random
  walk on a space-time graph. The ability to perform "all pairs" comparisons
  between points allows the model to obtain high spatial precision and to obtain
  a strong contrastive learning signal, while avoiding many of the complexities
  of recent approaches (such as coarse-to-fine matching). To do this, we propose
  a number of design decisions that allow global matching architectures to be
  trained through self-supervision using cycle consistency. For example, we
  identify that transformer-based methods are sensitive to shortcut solutions,
  and propose a data augmentation scheme to address them. Our method achieves
  strong performance on the TapVid benchmarks, outperforming previous
  self-supervised tracking methods, such as DIFT, and is competitive with several
  supervised methods.
paper_ai_keywords: global matching transformer cycle consistent tracks contrastive random walks space-time graph all pairs comparisons spatial precision contrastive learning signal self-supervision cycle consistency data augmentation scheme TapVid benchmarks DIFT