title: CoTracker: It is Better to Track Together
summary: Methods for video motion prediction either estimate jointly the instantaneous
  motion of all points in a given video frame using optical flow or independently
  track the motion of individual points throughout the video. The latter is true
  even for powerful deep-learning methods that can track points through
  occlusions. Tracking points individually ignores the strong correlation that
  can exist between the points, for instance, because they belong to the same
  physical object, potentially harming performance. In this paper, we thus
  propose CoTracker, an architecture that jointly tracks multiple points
  throughout an entire video. This architecture combines several ideas from the
  optical flow and tracking literature in a new, flexible and powerful design. It
  is based on a transformer network that models the correlation of different
  points in time via specialised attention layers. The transformer iteratively
  updates an estimate of several trajectories. It can be applied in a
  sliding-window manner to very long videos, for which we engineer an unrolled
  training loop. It can track from one to several points jointly and supports
  adding new points to track at any time. The result is a flexible and powerful
  tracking algorithm that outperforms state-of-the-art methods in almost all
  benchmarks.
paper_authors._id: 64b5feb57d9d8f9970fa4961
paper_authors.name: Nikita Karaev
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-18T08:28:46.099Z
paper_authors._id: 6393a73584c565d2c3416cb9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6393a73584c565d2c3416cb9/OGtX-i-_MLg5--qA054ti.jpeg
paper_authors.fullname: Nikita Karaev
paper_authors.type: user
paper_authors.user: nikkar
paper_authors._id: 64b5feb57d9d8f9970fa4962
paper_authors.name: Ignacio Rocco
paper_authors._id: 64b5feb57d9d8f9970fa4963
paper_authors.name: Benjamin Graham
paper_authors._id: 64b5feb57d9d8f9970fa4964
paper_authors.name: Natalia Neverova
paper_authors._id: 64b5feb57d9d8f9970fa4965
paper_authors.name: Andrea Vedaldi
paper_authors._id: 64b5feb57d9d8f9970fa4966
paper_authors.name: Christian Rupprecht
paper_title: CoTracker: It is Better to Track Together
paper_summary: Methods for video motion prediction either estimate jointly the instantaneous
  motion of all points in a given video frame using optical flow or independently
  track the motion of individual points throughout the video. The latter is true
  even for powerful deep-learning methods that can track points through
  occlusions. Tracking points individually ignores the strong correlation that
  can exist between the points, for instance, because they belong to the same
  physical object, potentially harming performance. In this paper, we thus
  propose CoTracker, an architecture that jointly tracks multiple points
  throughout an entire video. This architecture combines several ideas from the
  optical flow and tracking literature in a new, flexible and powerful design. It
  is based on a transformer network that models the correlation of different
  points in time via specialised attention layers. The transformer iteratively
  updates an estimate of several trajectories. It can be applied in a
  sliding-window manner to very long videos, for which we engineer an unrolled
  training loop. It can track from one to several points jointly and supports
  adding new points to track at any time. The result is a flexible and powerful
  tracking algorithm that outperforms state-of-the-art methods in almost all
  benchmarks.
paper_ai_summary: CoTracker, a deep learning architecture combining transformer networks and specialized attention layers, jointly tracks multiple video points across frames, offering superior performance compared to existing methods.
paper_ai_keywords: optical flow tracking CoTracker transformer network specialized attention layers trajectories sliding-window unrolled training loop