title: Self-Supervised Learning of Motion Concepts by Optimizing
    Counterfactuals
summary: Estimating motion in videos is an essential computer vision problem with many
  downstream applications, including controllable video generation and robotics.
  Current solutions are primarily trained using synthetic data or require tuning
  of situation-specific heuristics, which inherently limits these models'
  capabilities in real-world contexts. Despite recent developments in large-scale
  self-supervised learning from videos, leveraging such representations for
  motion estimation remains relatively underexplored. In this work, we develop
  Opt-CWM, a self-supervised technique for flow and occlusion estimation from a
  pre-trained next-frame prediction model. Opt-CWM works by learning to optimize
  counterfactual probes that extract motion information from a base video model,
  avoiding the need for fixed heuristics while training on unrestricted video
  inputs. We achieve state-of-the-art performance for motion estimation on
  real-world videos while requiring no labeled data.
paper_authors._id: 67e4b46cc90e5edf25f581f8
paper_authors.name: Stefan Stojanov
paper_authors._id: 67e4b46cc90e5edf25f581f9
paper_authors.name: David Wendt
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-27T10:14:24.479Z
paper_authors._id: 6303e4bb0547362a22a62274
paper_authors.avatarUrl: /avatars/91c9922913e45410ceed77f2e4971588.svg
paper_authors.fullname: David Wendt
paper_authors.type: user
paper_authors.user: kmeisthax
paper_authors._id: 67e4b46cc90e5edf25f581fa
paper_authors.name: Seungwoo Kim
paper_authors._id: 67e4b46cc90e5edf25f581fb
paper_authors.name: Rahul Venkatesh
paper_authors._id: 67e4b46cc90e5edf25f581fc
paper_authors.name: Kevin Feigelis
paper_authors._id: 67e4b46cc90e5edf25f581fd
paper_authors.name: Jiajun Wu
paper_authors._id: 67e4b46cc90e5edf25f581fe
paper_authors.name: Daniel LK Yamins
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-09T09:48:22.969Z
paper_authors._id: 67f625c5f7a398c2a285d1b8
paper_authors.avatarUrl: /avatars/1e4b708d68caee4a2d9ae6fe6e1c7b91.svg
paper_authors.fullname: Dan Yamins
paper_authors.type: user
paper_authors.user: dyamins
paper_title: Self-Supervised Learning of Motion Concepts by Optimizing
    Counterfactuals
paper_summary: Estimating motion in videos is an essential computer vision problem with many
  downstream applications, including controllable video generation and robotics.
  Current solutions are primarily trained using synthetic data or require tuning
  of situation-specific heuristics, which inherently limits these models'
  capabilities in real-world contexts. Despite recent developments in large-scale
  self-supervised learning from videos, leveraging such representations for
  motion estimation remains relatively underexplored. In this work, we develop
  Opt-CWM, a self-supervised technique for flow and occlusion estimation from a
  pre-trained next-frame prediction model. Opt-CWM works by learning to optimize
  counterfactual probes that extract motion information from a base video model,
  avoiding the need for fixed heuristics while training on unrestricted video
  inputs. We achieve state-of-the-art performance for motion estimation on
  real-world videos while requiring no labeled data.
paper_ai_summary: A self-supervised technique, Opt-CWM, extracts motion and occlusion information from pre-trained models without labeled data, achieving state-of-the-art performance on real-world videos.
paper_ai_keywords: self-supervised learning flow estimation occlusion estimation next-frame prediction counterfactual probes