title: Tracking Everything Everywhere All at Once
summary: We present a new test-time optimization method for estimating dense and
  long-range motion from a video sequence. Prior optical flow or particle video
  tracking algorithms typically operate within limited temporal windows,
  struggling to track through occlusions and maintain global consistency of
  estimated motion trajectories. We propose a complete and globally consistent
  motion representation, dubbed OmniMotion, that allows for accurate, full-length
  motion estimation of every pixel in a video. OmniMotion represents a video
  using a quasi-3D canonical volume and performs pixel-wise tracking via
  bijections between local and canonical space. This representation allows us to
  ensure global consistency, track through occlusions, and model any combination
  of camera and object motion. Extensive evaluations on the TAP-Vid benchmark and
  real-world footage show that our approach outperforms prior state-of-the-art
  methods by a large margin both quantitatively and qualitatively. See our
  project page for more results: http://omnimotion.github.io/
paper_authors._id: 6482c29c99110097e5490fe1
paper_authors.name: Qianqian Wang
paper_authors._id: 6482c29c99110097e5490fe2
paper_authors.name: Yen-Yu Chang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T13:04:56.683Z
paper_authors._id: 638b94ffdaa4cd1d6b40d719
paper_authors.avatarUrl: /avatars/1cb7f8d4062838710365b79b2f1d595a.svg
paper_authors.fullname: Yen-Yu Chang
paper_authors.type: user
paper_authors.user: KAXY
paper_authors._id: 6482c29c99110097e5490fe3
paper_authors.name: Ruojin Cai
paper_authors._id: 6482c29c99110097e5490fe4
paper_authors.name: Zhengqi Li
paper_authors._id: 6482c29c99110097e5490fe5
paper_authors.name: Bharath Hariharan
paper_authors._id: 6482c29c99110097e5490fe6
paper_authors.name: Aleksander Holynski
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T13:05:46.990Z
paper_authors._id: 62f6e2792e53c2efd33faa92
paper_authors.avatarUrl: /avatars/85d94b4022577747b8d2d10a82c2f3c7.svg
paper_authors.fullname: Aleksander Holynski
paper_authors.type: user
paper_authors.user: holynski
paper_authors._id: 6482c29c99110097e5490fe7
paper_authors.name: Noah Snavely
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T13:06:03.310Z
paper_authors._id: 632b79ca0a9dc02985af9e0b
paper_authors.avatarUrl: /avatars/e2ca585398a3212b2956a14804d0ba67.svg
paper_authors.fullname: Noah Snavely
paper_authors.type: user
paper_authors.user: jimantha
paper_title: Tracking Everything Everywhere All at Once
paper_summary: We present a new test-time optimization method for estimating dense and
  long-range motion from a video sequence. Prior optical flow or particle video
  tracking algorithms typically operate within limited temporal windows,
  struggling to track through occlusions and maintain global consistency of
  estimated motion trajectories. We propose a complete and globally consistent
  motion representation, dubbed OmniMotion, that allows for accurate, full-length
  motion estimation of every pixel in a video. OmniMotion represents a video
  using a quasi-3D canonical volume and performs pixel-wise tracking via
  bijections between local and canonical space. This representation allows us to
  ensure global consistency, track through occlusions, and model any combination
  of camera and object motion. Extensive evaluations on the TAP-Vid benchmark and
  real-world footage show that our approach outperforms prior state-of-the-art
  methods by a large margin both quantitatively and qualitatively. See our
  project page for more results: http://omnimotion.github.io/
paper_ai_keywords: OmniMotion quasi-3D canonical volume bijections fully-length motion estimation hierarchical motion representation global consistency camera and object motion TAP-Vid benchmark state-of-the-art methods